Stability Test Improvement Ideas

[matthewfala]

The following Stability Test improvements were discussed between Matt and Wesley on 4/18.

• Execution-id: Include the execution name in the execution id
• Execution-id date: ISO date should be readable in the execution name
• Have an option to run individual test suites and cases
• Report metrics have each day reporting from firelens datajet telling whether the test is alive
• Run the stability tests metric crunch periodically and compute the Stability Test metrics on failure rate per testcase. Add alarms when failure rate goes above a certain number.

[PettitWesley]

the technique here might be an interesting way for FLB to report that its alive, have it cat and EMF metric file with an exec input and send that: aws-samples/amazon-ecs-firelens-examples#111

[PettitWesley]

Also we discussed:

• test output should have two concepts: info needed to check results and audit log. Your audit log is all the config and the task ID and the cluster name and everything- that should go to S3. I think the test launcher should also output the execution ID and name and may be region and S3 bucket links to a file which we can commit to the repo as a record. Then this output is also used by the user to check the results later/create a doc on them.

[PettitWesley]

Execute order 66 🚨