You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/daily-test-improver.md
+19-6Lines changed: 19 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,11 +22,21 @@ You can start a run of this workflow immediately by running:
22
22
gh aw run daily-test-improver
23
23
```
24
24
25
+
## How It Works
26
+
27
+
The workflow operates in two phases:
28
+
29
+
**Phase 1 - Testing Research:** On the first run, the workflow researches the repository's testing landscape, identifies build and coverage commands, and creates a discussion with the plan. Memory notes are stored persistently in a `memory/daily-test-improver` branch for future runs.
30
+
31
+
**Phase 2 - Goal Selection, Work and Results:** On subsequent runs, the workflow consults its memory notes, selects a coverage area to improve, writes new tests, and submits draft PRs with coverage improvements.
32
+
25
33
## Configuration
26
34
27
-
1. The first run of the workflow will produce a pull request with inferred action pre-steps that need approval.
35
+
1. The first run of the workflow will create a **discussion** in the repository with a research summary and plan for improving test coverage. Review the plan and provide feedback via comments.
36
+
37
+
2. The workflow uses **repo-memory** to persist notes about build commands, coverage steps, and testing strategies across runs. These are stored in the `memory/daily-test-improver` branch.
28
38
29
-
2. The first run of the workflow will also create an issue in the repository with a plan for improving test coverage. You can comment on this issue to provide feedback or adjustments to the plan. Comments will not be picked up until the next run.
39
+
3. Subsequent runs will implement test improvements based on the plan and create draft pull requests.
30
40
31
41
After editing run `gh aw compile` to update the workflow and commit all changes to the default branch.
32
42
@@ -35,17 +45,20 @@ After editing run `gh aw compile` to update the workflow and commit all changes
35
45
- Repository contents and source code for coverage analysis
36
46
- Existing test files and test coverage reports
37
47
- Build scripts and testing configuration files
38
-
- Previous issues and pull requests related to testing
48
+
- Previous issues, pull requests, and discussions related to testing
49
+
- Its own memory notes from previous runs (stored in `memory/daily-test-improver` branch)
39
50
40
51
## What it creates
41
52
42
-
- Creates new branches with additional test cases
53
+
- Creates discussions with testing research and coverage improvement plans
54
+
- Creates new branches (prefixed with `test/`) with additional test cases
43
55
- Creates draft pull requests with improved test coverage
44
-
- Creates issues documenting coverage analysis and improvements
45
-
-Makes file changes to add meaningful tests for edge cases and uncovered code
56
+
- Creates issues if bugs are discovered while adding tests
57
+
-Stores persistent memory notes for build commands, coverage steps, and testing strategies
46
58
47
59
## Human in the loop
48
60
61
+
- Review the initial research discussion and provide feedback on the coverage plan
49
62
- Review test coverage improvement pull requests for test quality
50
63
- Validate that new tests properly cover edge cases and uncovered code
51
64
- Ensure tests are meaningful and not just coverage-padding
@@ -72,95 +56,80 @@ To decide which phase to perform:
72
56
73
57
1. First check for existing open discussion titled "${{ github.workflow }}" using `list_discussions`. Double check the discussion is actually still open - if it's closed you need to ignore it. If found, and open, read it and maintainer comments. If not found, then perform Phase 1 and nothing else.
74
58
75
-
2. Next check if `.github/actions/daily-test-improver/coverage-steps/action.yml` exists. If yes then read it. If not then perform Phase 2 and nothing else.
76
-
77
-
3. Finally, if both those exist, then perform Phase 3.
59
+
2. If that exists, then perform Phase 2.
78
60
79
61
## Phase 1 - Testing research
80
62
81
63
1. Research the current state of test coverage in the repository. Look for existing test files, coverage reports, and any related issues or pull requests.
82
64
83
-
2. Create a discussion with title "${{ github.workflow }} - Research and Plan" that includes:
84
-
- A summary of your findings about the repository, its testing strategies, its test coverage
85
-
- A plan for how you will approach improving test coverage, including specific areas to focus on and strategies to use
86
-
- Details of the commands needed to run to build the project, run tests, and generate coverage reports
87
-
- Details of how tests are organized in the repo, and how new tests should be organized
88
-
- Opportunities for new ways of greatly increasing test coverage
89
-
- Any questions or clarifications needed from maintainers
65
+
2. Have a careful think about the CI commands needed to build the repository, run tests, produce a combined coverage report and upload it as an artifact. Do this by carefully reading any existing documentation and CI files in the repository that do similar things, and by looking at any build scripts, project files, dev guides and so on in the repository. If multiple projects are present, perform build and coverage testing on as many as possible, and where possible merge the coverage reports into one combined report. Organize the steps in order as a series of YAML steps suitable for inclusion in a GitHub Action.
66
+
67
+
3. Try to run through the steps you worked out manually one by one. If the a step needs updating, then update the branch you created in step 4. Continue through all the steps. If you can't get it to work, then create an issue describing the problem and exit the entire workflow.
68
+
69
+
4. Keep memory notes in `/tmp/gh-aw/repo-memory-daily-test-improver/` about how to do this, and what the commands are, so you can refer back to them in future runs. Store notes in structured files:
-`coverage-notes.md` - Coverage generation commands and report locations
72
+
-`testing-notes.md` - Test organization, frameworks used, and strategies
73
+
74
+
You will need these notes for Phase 2.
75
+
76
+
5. Create a discussion with title "${{ github.workflow }} - Research and Plan" that includes:
77
+
78
+
- A summary of your findings about the repository, its testing strategies, its test coverage
79
+
- A plan for how you will approach improving test coverage, including specific areas to focus on and strategies to use
80
+
- Details of the commands needed to run to build the project, run tests, and generate coverage reports
81
+
- Details of how tests are organized in the repo, and how new tests should be organized
82
+
- Opportunities for new ways of greatly increasing test coverage
83
+
- Any questions or clarifications needed from maintainers
90
84
91
85
**Include a "How to Control this Workflow" section at the end of the discussion that explains:**
92
-
- The user can add comments to the discussion to provide feedback or adjustments to the plan
93
-
- The user can use these commands:
86
+
87
+
- The user can add comments to the discussion to provide feedback or adjustments to the plan
88
+
- The user can use these commands:
94
89
95
90
gh aw disable daily-test-improver --repo ${{ github.repository }}
96
91
gh aw enable daily-test-improver --repo ${{ github.repository }}
97
92
gh aw run daily-test-improver --repo ${{ github.repository }} --repeat <number-of-repeats>
98
93
gh aw logs daily-test-improver --repo ${{ github.repository }}
99
94
100
95
**Include a "What Happens Next" section at the end of the discussion that explains:**
101
-
- The next time this workflow runs, Phase 2 will be performed, which will analyze the codebase to create coverage steps configuration
102
-
- After Phase 2 completes, Phase 3 will begin on subsequent runs to implement actual test coverage improvements
103
-
- If running in "repeat" mode, the workflow will automatically run again to proceed to the next phase
104
-
- Humans can review this research and add comments before the workflow continues
105
-
106
-
3. Exit this entire workflow, do not proceed to Phase 2 on this run. The research and plan will be checked by a human who will invoke you again and you will proceed to Phase 2.
107
-
108
-
## Phase 2 - Coverage steps inference and configuration
109
-
110
-
1. Check if an open pull request with title "${{ github.workflow }} - Updates to complete configuration" exists in this repo. If it does, add a comment to the pull request saying configuration needs to be completed, then exit the workflow.
111
-
112
-
2. Have a careful think about the CI commands needed to build the repository, run tests, produce a combined coverage report and upload it as an artifact. Do this by carefully reading any existing documentation and CI files in the repository that do similar things, and by looking at any build scripts, project files, dev guides and so on in the repository. If multiple projects are present, perform build and coverage testing on as many as possible, and where possible merge the coverage reports into one combined report. Organize the steps in order as a series of YAML steps suitable for inclusion in a GitHub Action.
113
-
114
-
3. Create the file `.github/actions/daily-test-improver/coverage-steps/action.yml` containing these steps, ensuring that the action.yml file is valid. Leave comments in the file to explain what the steps are doing, where the coverage report will be generated, and any other relevant information. Ensure that the steps include uploading the coverage report(s) as an artifact called "coverage". Each step of the action should append its output to a file called `coverage-steps.log` in the root of the repository. Ensure that the action.yml file is valid and correctly formatted.
115
-
116
-
4. Before running any of the steps, make a pull request for the addition of the `action.yml` file, with title "${{ github.workflow }} - Updates to complete configuration". Encourage the maintainer to review the files carefully to ensure they are appropriate for the project.
117
96
118
-
**Include a "What Happens Next" section in the PR description that explains:**
119
-
- Once this PR is merged, the next workflow run will proceed to Phase 3, where actual test coverage improvements will be implemented
120
-
- Phase 3 will use the coverage steps to systematically improve test coverage
121
-
- If running in "repeat" mode, the workflow will automatically run again to proceed to Phase 3
122
-
- Humans can review and merge this configuration before continuing
97
+
6. Exit this entire workflow, do not proceed to Phase 2 on this run. The coverage steps will now be checked by a human who will invoke you again and you will proceed to Phase 2.
123
98
124
-
5. Try to run through the steps you worked out manually one by one. If the a step needs updating, then update the branch you created in step 4. Continue through all the steps. If you can't get it to work, then create an issue describing the problem and exit the entire workflow.
125
-
126
-
6. Add brief comment (1 or 2 sentences) to the discussion identified at the start of the workflow stating what you've done and giving links to the PR created. If you have taken successful initial coverage numbers for the repository, report the initial coverage numbers appropriately.
127
-
128
-
7. Exit this entire workflow, do not proceed to Phase 3 on this run. The coverage steps will now be checked by a human who will invoke you again and you will proceed to Phase 3.
129
-
130
-
## Phase 3 - Goal selection, work and results
99
+
## Phase 2 - Goal selection, work and results (repeat this phase on multiple runs)
131
100
132
101
1.**Goal selection**. Build an understanding of what to work on and select an area of the test coverage plan to pursue
133
102
134
-
a. Repository is now test-ready. Review `coverage-steps/action.yml` and `coverage-steps.log` to understand setup. If coverage steps failed then create fix PR and exit.
103
+
a. Consult your memory notes in `/tmp/gh-aw/repo-memory-daily-test-improver/` (especially `build-notes.md`, `coverage-notes.md`, and `testing-notes.md`), and build and test the repository taking coverage. If coverage steps failed then create fix PR, update memory notes and exit.
135
104
136
105
b. Locate and read the coverage report. Be detailed, looking to understand the files, functions, branches, and lines of code that are not covered by tests. Look for areas where you can add meaningful tests that will improve coverage.
137
-
106
+
138
107
c. Read the plan in the discussion mentioned earlier, along with comments.
139
-
108
+
140
109
d. Check the most recent pull request with title starting with "${{ github.workflow }}" (it may have been closed) and see what the status of things was there. These are your notes from last time you did your work, and may include useful recommendations for future areas to work on.
141
110
142
111
e. Check for existing open pull requests (especially yours with "${{ github.workflow }}" prefix). Avoid duplicate work.
143
-
112
+
144
113
f. If plan needs updating then comment on planning discussion with revised plan and rationale. Consider maintainer feedback.
145
114
146
115
g. Based on all of the above, select an area of relatively low coverage to work on that appears tractable for further test additions. Ensure that you have a good understanding of the code and the testing requirements before proceeding.
147
116
148
117
2.**Work towards your selected goal**. For the test coverage improvement goal you selected, do the following:
149
118
150
119
a. Create a new branch starting with "test/".
151
-
120
+
152
121
b. Write new tests to improve coverage. Ensure that the tests are meaningful and cover edge cases where applicable.
153
122
154
123
c. Build the tests if necessary and remove any build errors.
155
-
124
+
156
125
d. Run the new tests to ensure they pass.
157
126
158
127
e. Re-run the test suite collecting coverage information. Check that overall coverage has improved. Document measurement attempts even if unsuccessful. If no improvement then iterate, revert, or try different approach.
159
128
160
129
3.**Finalizing changes**
161
130
162
131
a. Apply any automatic code formatting used in the repo. If necessary check CI files to understand what code formatting is used.
163
-
132
+
164
133
b. Run any appropriate code linter used in the repo and ensure no new linting errors remain. If necessary check CI files to understand what code linting is used.
165
134
166
135
4.**Results and learnings**
@@ -170,7 +139,7 @@ To decide which phase to perform:
170
139
**Critical:** Exclude coverage reports and tool-generated files from PR. Double-check added files and remove any that don't belong.
171
140
172
141
Include a description of the improvements with evidence of impact. In the description, explain:
173
-
142
+
174
143
-**Goal and rationale:** Coverage area chosen and why it matters
175
144
-**Approach:** Testing strategy, methodology, and implementation steps
176
145
-**Impact measurement:** How coverage was tested and results achieved
@@ -188,4 +157,4 @@ To decide which phase to perform:
188
157
189
158
b. If you think you found bugs in the code while adding tests, also create one single combined issue for all of them, starting the title of the issue with "${{ github.workflow }}". Do not include fixes in your pull requests unless you are 100% certain the bug is real and the fix is right.
190
159
191
-
5.**Final update**: Add brief comment (1 or 2 sentences) to the discussion identified at the start of the workflow stating goal worked on, PR links, and progress made, reporting the coverage improvement numbers achieved and current overall coverage numbers.
160
+
5.**Final update**: Add brief comment (1 or 2 sentences) to the discussion identified at the start of the workflow stating goal worked on, PR links, and progress made, reporting the coverage improvement numbers achieved and current overall coverage numbers.
0 commit comments