Skip to content

Stress/perf automation#1077

Open
CIPop wants to merge 11 commits into
Azure:mainfrom
CIPop:perfAuto
Open

Stress/perf automation#1077
CIPop wants to merge 11 commits into
Azure:mainfrom
CIPop:perfAuto

Conversation

@CIPop

@CIPop CIPop commented Oct 1, 2019

Copy link
Copy Markdown
Contributor

DevOps perf/stress automation.

Checklist

  • I have read the contribution guidelines.
  • I added or modified the existing tests to cover the change (we do not allow our test coverage to go down).
  • This pull-request is submitted against the master branch.

Description of the changes

  • Adding devops yaml file
  • Fault injection using Windows Firewall
  • KPI measurement against set parameters
  • Exception handling to simplify pass/fail decision
  • Nuget package testing
  • Handling import/export function internally
  • Better logging

@CIPop

CIPop commented Oct 1, 2019

Copy link
Copy Markdown
Contributor Author

Note: includes #1066

@CIPop

CIPop commented Oct 1, 2019

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

@CIPop

CIPop commented Oct 1, 2019

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

@CIPop

CIPop commented Oct 2, 2019

Copy link
Copy Markdown
Contributor Author

File upload tests likely fixed by #1065

@CIPop

CIPop commented Oct 2, 2019

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

@CIPop

CIPop commented Oct 2, 2019

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

@CIPop

CIPop commented Oct 4, 2019

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

@CIPop

CIPop commented Oct 7, 2019

Copy link
Copy Markdown
Contributor Author

Unrelated multi-connection test failures in Canary.
Known horton failure likely due to pytest (under investigation).

@CIPop CIPop mentioned this pull request Oct 16, 2019
@CIPop

CIPop commented Mar 24, 2020

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 1077 in repo Azure/azure-iot-sdk-csharp

@CIPop

CIPop commented Mar 24, 2020

Copy link
Copy Markdown
Contributor Author

@drwill-ms @prmathur-microsoft @abhipsaMisra what's the status on this one?
Please let me know by EOW if you would still want to take it or if I should close and keep it in my CS branch. I'll lock the branch on my side either way to ensure work isn't deleted by mistake.

@abhipsaMisra

Copy link
Copy Markdown
Member

@CIPop , I am not able to update your branch to pull changes from master. Could you update your branch; I'll then run the E2E tests and that'll give us an idea if we need to update this PR to incorporate the changes we've made to the SDK since then.

@CIPop

CIPop commented May 13, 2020

Copy link
Copy Markdown
Contributor Author

I am not able to update your branch

Ah, sorry about that - I've protected my branch to avoid deleting it by mistake. I have updated it locally, will unlock then push new changes.

@abhipsaMisra

Copy link
Copy Markdown
Member

/azp run csharp-westcentralus

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@abhipsaMisra

Copy link
Copy Markdown
Member

@CIPop Hey Chris, do you see vsts/vstsPerf.yaml being used to set up a different pipeline; one that will run these perf tests as a part of our CI/ nightly run?

@CIPop

CIPop commented May 21, 2020

Copy link
Copy Markdown
Contributor Author

@CIPop Hey Chris, do you see vsts/vstsPerf.yaml being used to set up a different pipeline; one that will run these perf tests as a part of our CI/ nightly run?

My recommendation would be to run them nightly and monitor the results or tune the KPI parameters to avoid perf regressions (i.e. with each perf enhancement).
At a minimum you could run the tests prior to every release and compare results with the prior runs. This though won't give you proper failure analysis for intermittent issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants