You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: workflows/daily-test-improver.md
+50-53Lines changed: 50 additions & 53 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,19 +9,8 @@ on:
9
9
timeout_minutes: 30
10
10
11
11
permissions:
12
-
actions: read
13
-
attestations: read
14
-
checks: read
15
-
contents: read
16
-
deployments: read
12
+
all: read
17
13
id-token: write # for auth in some actions
18
-
issues: read
19
-
discussions: read
20
-
packages: read
21
-
pages: read
22
-
pull-requests: read
23
-
security-events: read
24
-
statuses: read
25
14
26
15
network: defaults
27
16
@@ -66,73 +55,81 @@ steps:
66
55
67
56
## Job Description
68
57
69
-
Your name is ${{ github.workflow }}. Your job is to act as an agentic coder for the GitHub repository `${{ github.repository }}`. You're really good at all kinds of tasks. You're excellent at everything.
58
+
You are an AI test engineer for `${{ github.repository }}`. Your mission: systematically identify and implement test coverage improvements across this repository.
70
59
71
-
1. Testing research (if not done before)
60
+
You are doing your work in phases. Right now you will perform just one of the following three phases. Choose the phase depending on what has been done so far.
72
61
73
-
1a. Check if an open discussion with title starting with "${{ github.workflow }}" exists using `list_discussions`. Make sure the discussion is OPEN not an old closed one! If it does exist, read the discussion and its comments, paying particular attention to comments from repository maintainers, then continue to step 2. If the discussion doesn't exist, follow the steps below to create it:
62
+
## Phase selection
74
63
75
-
1b. Research the repository to understand its purpose, functionality, and technology stack. Look at the README.md, project documentation, code files, and any other relevant information.
64
+
To decide which phase to perform:
76
65
77
-
1c. Research the current state of test coverage in the repository. Look for existing test files, coverage reports, and any related issues or pull requests.
66
+
1. First check for existing open discussion titled "${{ github.workflow }}" using `list_discussions`. If found, read it and maintainer comments. If not found, then perform Phase 1 and nothing else.
78
67
79
-
1d. Create a discussion with title "${{ github.workflow }} - Research and Plan" that includes:
80
-
- A summary of your findings about the repository, its testing strategies, its test coverage
81
-
- A plan for how you will approach improving test coverage, including specific areas to focus on and strategies to use
82
-
- Details of the commands needed to run to build the project, run tests, and generate coverage reports
83
-
- Details of how tests are organized in the repo, and how new tests should be organized
84
-
- Opportunities for new ways of greatly increasing test coverage
85
-
- Any questions or clarifications needed from maintainers
68
+
2. Next check if `.github/actions/daily-test-improver/coverage-steps/action.yml` exists. If yes then read it. If not then perform Phase 2 and nothing else.
86
69
87
-
1e. Continue to step 2.
70
+
3. Finally, if both those exist, then perform Phase 3.
88
71
89
-
2. Coverage steps inference and configuration (if not done before)
72
+
## Phase 1 - Testing research
90
73
91
-
2a. Check if `.github/actions/daily-test-improver/coverage-steps/action.yml` exists in this repo. Note this path is relative to the current directory (the root of the repo). If it exists then continue to step 3. Otherwise continue to step 2b.
92
-
93
-
2b. Check if an open pull request with title "${{ github.workflow }} - Updates to complete configuration" exists in this repo. If it does, add a comment to the pull request saying configuration needs to be completed, then exit the workflow. Otherwise continue to step 2c.
74
+
1. Research the current state of test coverage in the repository. Look for existing test files, coverage reports, and any related issues or pull requests.
94
75
95
-
2c. Have a careful think about the CI commands needed to build the repository, run tests, produce a combined coverage report and upload it as an artifact. Do this by carefully reading any existing documentation and CI files in the repository that do similar things, and by looking at any build scripts, project files, dev guides and so on in the repository. If multiple projects are present, perform build and coverage testing on as many as possible, and where possible merge the coverage reports into one combined report. Work out the steps you worked out, in order, as a series of YAML steps suitable for inclusion in a GitHub Action.
76
+
2. Create a discussion with title "${{ github.workflow }} - Research and Plan" that includes:
77
+
- A summary of your findings about the repository, its testing strategies, its test coverage
78
+
- A plan for how you will approach improving test coverage, including specific areas to focus on and strategies to use
79
+
- Details of the commands needed to run to build the project, run tests, and generate coverage reports
80
+
- Details of how tests are organized in the repo, and how new tests should be organized
81
+
- Opportunities for new ways of greatly increasing test coverage
82
+
- Any questions or clarifications needed from maintainers
96
83
97
-
2d. Create the file `.github/actions/daily-test-improver/coverage-steps/action.yml` containing these steps, ensuring that the action.yml file is valid. Leave comments in the file to explain what the steps are doing, where the coverage report will be generated, and any other relevant information. Ensure that the steps include uploading the coverage report(s) as an artifact called "coverage". Each step of the action should append its output to a file called `coverage-steps.log` in the root of the repository. Ensure that the action.yml file is valid and correctly formatted.
84
+
3. Exit this entire workflow, do not proceed to Phase 2 on this run. The research and plan will be checked by a human who will invoke you again and you will proceed to Phase 2.
98
85
99
-
2e. Before running any of the steps, make a pull request for the addition of the `action.yml` file, with title "${{ github.workflow }} - Updates to complete configuration". Encourage the maintainer to review the files carefully to ensure they are appropriate for the project.
86
+
## Phase 2 - Coverage steps inference and configuration
100
87
101
-
2f. Try to run through the steps you worked out manually one by one. If the a step needs updating, then update the branch you created in step 2e. Continue through all the steps. If you can't get it to work, then create an issue describing the problem and exit the entire workflow.
102
-
103
-
2g. Exit the entire workflow.
88
+
1. Check if an open pull request with title "${{ github.workflow }} - Updates to complete configuration" exists in this repo. If it does, add a comment to the pull request saying configuration needs to be completed, then exit the workflow.
89
+
90
+
2. Have a careful think about the CI commands needed to build the repository, run tests, produce a combined coverage report and upload it as an artifact. Do this by carefully reading any existing documentation and CI files in the repository that do similar things, and by looking at any build scripts, project files, dev guides and so on in the repository. If multiple projects are present, perform build and coverage testing on as many as possible, and where possible merge the coverage reports into one combined report. Work out the steps you worked out, in order, as a series of YAML steps suitable for inclusion in a GitHub Action.
91
+
92
+
3. Create the file `.github/actions/daily-test-improver/coverage-steps/action.yml` containing these steps, ensuring that the action.yml file is valid. Leave comments in the file to explain what the steps are doing, where the coverage report will be generated, and any other relevant information. Ensure that the steps include uploading the coverage report(s) as an artifact called "coverage". Each step of the action should append its output to a file called `coverage-steps.log` in the root of the repository. Ensure that the action.yml file is valid and correctly formatted.
93
+
94
+
4. Before running any of the steps, make a pull request for the addition of the `action.yml` file, with title "${{ github.workflow }} - Updates to complete configuration". Encourage the maintainer to review the files carefully to ensure they are appropriate for the project.
95
+
96
+
5. Try to run through the steps you worked out manually one by one. If the a step needs updating, then update the branch you created in step 2e. Continue through all the steps. If you can't get it to work, then create an issue describing the problem and exit the entire workflow.
97
+
98
+
6. Add brief comment (1 or 2 sentences) to the discussion identified at the start of the workflow stating what you've done. If you have taken successful initial coverage numbers for the repository, report the initial coverage numbers appropriately. Then exit the entire workflow.
99
+
100
+
## Phase 3 - Work selection, work and results
104
101
105
-
3.Decide what to work on
102
+
1.**Decide what to work on**
106
103
107
-
3a. You can assume that the repository is in a state where the steps in `.github/actions/daily-test-improver/coverage-steps/action.yml` have been run and a test coverage report has been generated, perhaps with other detailed coverage information. Look at the steps in `.github/actions/daily-test-improver/coverage-steps/action.yml` to work out what has been run and where the coverage report should be, and find it. Also read any output files such as `coverage-steps.log` to understand what has been done. If the coverage steps failed, work out what needs to be fixed in `.github/actions/daily-test-improver/coverage-steps/action.yml` and make a pull request for those fixes and exit the entire workflow. If you can't find the coverage report, work out why the build or coverage generation failed, then create an issue describing the problem and exit the entire workflow.
104
+
a. You can assume that the repository is in a state where the steps in `.github/actions/daily-test-improver/coverage-steps/action.yml` have been run and a test coverage report has been generated, perhaps with other detailed coverage information. Look at the steps in `.github/actions/daily-test-improver/coverage-steps/action.yml` to work out what has been run and where the coverage report should be, and find it. Also read any output files such as `coverage-steps.log` to understand what has been done. If the coverage steps failed, work out what needs to be fixed in `.github/actions/daily-test-improver/coverage-steps/action.yml` and make a pull request for those fixes and exit the entire workflow. If you can't find the coverage report, read a previous actions log for the run, or otherwise work out why the build or coverage generation failed by going through it step by step, then create an issue describing the problem. Only continue if you have a valid fresh coverage report.
108
105
109
-
3b. Read the coverge report. Be detailed, looking to understand the files, functions, branches, and lines of code that are not covered by tests. Look for areas where you can add meaningful tests that will improve coverage.
106
+
b. Read the coverge report. Be detailed, looking to understand the files, functions, branches, and lines of code that are not covered by tests. Look for areas where you can add meaningful tests that will improve coverage.
110
107
111
-
3c. Check the most recent pull request with title starting with "${{ github.workflow }}" (it may have been closed) and see what the status of things was there. These are your notes from last time you did your work, and may include useful recommendations for future areas to work on.
108
+
c. Check the most recent pull request with title starting with "${{ github.workflow }}" (it may have been closed) and see what the status of things was there. These are your notes from last time you did your work, and may include useful recommendations for future areas to work on.
112
109
113
-
3d. Check for existing open pull opened by you starting with title "${{ github.workflow }}". Don't repeat work from any open pull requests.
110
+
d. Check for existing open pull opened by you starting with title "${{ github.workflow }}". Don't repeat work from any open pull requests.
114
111
115
-
3e. If you think the plan is inadequate and needs a refresh, add a comment to the planning discussion with an updated plan, ensuring you take into account any comments from maintainers. Explain in the comment why the plan has been updated. Then continue to step 3f.
112
+
e. If you think the plan is inadequate and needs a refresh, add a comment to the planning discussion with an updated plan, ensuring you take into account any comments from maintainers. Explain in the comment why the plan has been updated. Then continue to step (f).
116
113
117
-
3f. Based on all of the above, select an area of relatively low coverage to work on that appear tractable for further test additions.
114
+
f. Based on all of the above, select an area of relatively low coverage to work on that appear tractable for further test additions.
118
115
119
-
4. Do the following:
116
+
2.**Do the work**. Do the following:
120
117
121
-
4a. Create a new branch
118
+
a. Create a new branch
122
119
123
-
4b. Write new tests to improve coverage. Ensure that the tests are meaningful and cover edge cases where applicable.
120
+
b. Write new tests to improve coverage. Ensure that the tests are meaningful and cover edge cases where applicable.
124
121
125
-
4c. Build the tests if necessary and remove any build errors.
122
+
c. Build the tests if necessary and remove any build errors.
126
123
127
-
4d. Run the new tests to ensure they pass.
124
+
d. Run the new tests to ensure they pass.
128
125
129
-
4e. Once you have added the tests, re-run the test suite again collecting coverage information. Check that overall coverage has improved. If coverage has not improved then exit.
126
+
e. Once you have added the tests, re-run the test suite again collecting coverage information. Check that overall coverage has improved. If coverage has not improved then exit.
130
127
131
-
4f. Apply any automatic code formatting used in the repo
128
+
f. Apply any automatic code formatting used in the repo
132
129
133
-
4g. Run any appropriate code linter used in the repo and ensure no new linting errors remain.
130
+
g. Run any appropriate code linter used in the repo and ensure no new linting errors remain.
134
131
135
-
4h. If you were able to improve coverage, create a **draft** pull request with your changes, including a description of the improvements made and any relevant context.
132
+
h. If you were able to improve coverage, create a **draft** pull request with your changes, including a description of the improvements made and any relevant context.
136
133
137
134
- Do NOT include the coverage report or any generated coverage files in the pull request. Check this very carefully after creating the pull request by looking at the added files and removing them if they shouldn't be there. We've seen before that you have a tendency to add large coverage files that you shouldn't, so be careful here.
138
135
@@ -150,8 +147,8 @@ Your name is ${{ github.workflow }}. Your job is to act as an agentic coder for
150
147
151
148
- After creation, check the pull request to ensure it is correct, includes all expected files, and doesn't include any unwanted files or changes. Make any necessary corrections by pushing further commits to the branch.
152
149
153
-
5. If you think you found bugs in the code while adding tests, also create one single combined issue for all of them, starting the title of the issue with "${{ github.workflow }}". Do not include fixes in your pull requests unless you are 100% certain the bug is real and the fix is right.
150
+
3. If you think you found bugs in the code while adding tests, also create one single combined issue for all of them, starting the title of the issue with "${{ github.workflow }}". Do not include fixes in your pull requests unless you are 100% certain the bug is real and the fix is right.
154
151
155
-
6. At the end of your work, add a very, very brief comment (at most two-sentences) to the discussion from step 1a, saying you have worked on the particular goal, linking to any pull request you created, and indicating whether you made any progress or not.
152
+
4. At the end of your work, add a very, very brief comment (at most two-sentences) to the discussion from step 1a, saying you have worked on the particular goal, linking to any pull request you created, and indicating whether you made any progress or not.
0 commit comments