Skip to content

Commit d5e1e69

Browse files
bee0511darko-marinov
bee0511
authored and
darko-marinov
committed
improve the git bisect script
1 parent fd0947c commit d5e1e69

3 files changed

Lines changed: 282 additions & 53 deletions

File tree

‎auto-check-fix-commit/README.md‎

Lines changed: 117 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,7 @@ There are many flaky tests marked as ```DeveloperFixed```. Once you find the fir
99

1010
## Find the commit which fixed the flaky test using Git Bisect
1111

12-
Git Bisect command uses a binary search algorithm to find which commit in your project’s history introduced a bug. You use it by first telling it a "bad" commit that is known to contain the bug, and a "good" commit that is known to be before the bug was introduced.
13-
14-
The git-bisect-script-alluxio.sh script handles the case where the flaky test is fixed in the latest commit.
12+
Git Bisect uses a binary search algorithm to find which commit fixed a flaky test. This tool uses custom terminology: "flaky" for the old broken state and "non-flaky" for the new fixed state, making the process more intuitive.
1513

1614
Reference - https://git-scm.com/docs/git-bisect/
1715

@@ -21,17 +19,125 @@ Reference - https://git-scm.com/docs/git-bisect/
2119

2220
2. Copy scripts `git-bisect-runner.sh` and `git-bisect-script.sh` to the directory of the project containing the fixed flaky test.
2321

24-
3. Within the project containing the DeveloperFixed test, run the script using the command below. Modify arguments to specific flaky commit, fixed commit, module path, test case of your project, mvn install options, and NonDex version. This command will also ensure standard error and output messages go to `git_bisect_output.log`, while running command in the background (since using nohup).
22+
3. Within the project containing the DeveloperFixed test, run the script using the command below.
23+
24+
#### Command Format
25+
26+
```shell
27+
./git-bisect-runner.sh \
28+
--flaky <FLAKY_COMMIT> \
29+
--fixed <FIXED_COMMIT> \
30+
--module <MODULE_PATH> \
31+
--test <TEST_CASE> \
32+
--nondex-version <NONDEX_VERSION> \
33+
[--mvn-install <MAVEN_OPTIONS>]
34+
```
35+
36+
#### Argument Details
37+
38+
| Argument | Format | Description | Example |
39+
|----------|--------|-------------|---------|
40+
| `--flaky` | Git commit hash (full or short) | The commit where the test **was flaky** | `ecf41be2ecd007853c2db19e1c6a038cf356cb9e` or `ecf41be` |
41+
| `--fixed` | Git commit hash (full or short) | The commit where the test **was fixed** | `f69557e325c5bb9e4e250bb0ec2db12d85298211` or `f69557e` |
42+
| `--module` | Maven module path | The Maven module containing the test (relative to project root) | `pinot-core` or `server/service` |
43+
| `--test` | Fully qualified test name | Format: `package.ClassName#methodName` or `package.ClassName` | `org.apache.pinot.queries.QueryTest#testMethod` |
44+
| `--nondex-version` | Version string | The NonDex Maven plugin version to use | `"2.1.7"` or `"1.1.2"` |
45+
| `--mvn-install` | Maven CLI options (optional) | Additional Maven options for build | `"-Dspotless.skip"` or `"-Dlicense.skip -DskipTests"` |
46+
47+
**Important Notes:**
48+
- **Module Path**: Use the Maven module identifier (same as `-pl` argument in Maven)
49+
- For single module projects: use `.` or the module name
50+
- For multi-module projects: use the relative path like `sub-module/nested-module`
51+
- **Test Case Format**:
52+
- Single test method: `com.example.TestClass#testMethodName`
53+
- Entire test class: `com.example.TestClass`
54+
- **Commit Hashes**: Both short (7+ chars) and full (40 chars) SHA hashes are acceptable
55+
56+
#### Examples
57+
58+
**Example 1: Basic usage**
59+
```shell
60+
./git-bisect-runner.sh \
61+
--flaky ecf41be2ecd007853c2db19e1c6a038cf356cb9e \
62+
--fixed f69557e325c5bb9e4e250bb0ec2db12d85298211 \
63+
--module pinot-core \
64+
--test org.apache.pinot.queries.ForwardIndexHandlerReloadQueriesTest#testSelectQueries \
65+
--nondex-version "2.1.7" \
66+
--mvn-install "-Dspotless.skip"
67+
```
68+
69+
**Example 2: Using short commit hashes**
70+
```shell
71+
./git-bisect-runner.sh \
72+
--flaky ecf41be \
73+
--fixed f69557e \
74+
--module server/service \
75+
--test com.example.integration.ApiTest#shouldHandleTimeout \
76+
--nondex-version "2.1.7"
77+
```
78+
79+
**Example 3: Multi-module project with multiple Maven options**
2580
```shell
26-
nohup ./git-bisect-runner.sh --flaky <FLAKY_COMMIT> --fixed <FIXED_COMMIT> --module <MODULE_PATH> --test <TEST_CASE> --nondex-version "<NONDEX VERSION>" --mvn-install "<MAVEN_OPTIONS>" &> git_bisect_output.log
81+
./git-bisect-runner.sh \
82+
--flaky abc1234 \
83+
--fixed def5678 \
84+
--module galleon-pack/layer-metadata-tests \
85+
--test org.wildfly.feature.pack.layer.tests.opentelemetry.OpenTelemetryLayerMetaDataTestCase \
86+
--nondex-version "2.1.7" \
87+
--mvn-install "-Dlicense.skip -DskipFormat"
88+
```
89+
90+
**Example 4: Running in background with logging**
91+
```shell
92+
nohup ./git-bisect-runner.sh \
93+
--flaky ecf41be \
94+
--fixed f69557e \
95+
--module pinot-core \
96+
--test org.apache.pinot.queries.ForwardIndexHandlerReloadQueriesTest#testSelectQueries \
97+
--nondex-version "2.1.7" \
98+
--mvn-install "-Dspotless.skip" \
99+
&> git_bisect_output.log &
100+
```
101+
#### Alternative: Using Default Values
102+
103+
You can also set default values directly in `git-bisect-runner.sh` (lines 5-10) to avoid typing arguments every time:
104+
105+
```bash
106+
# Default values in git-bisect-runner.sh
107+
mvn_options="-Dspotless.skip"
108+
nondex_version="2.1.7"
109+
flaky_commit="ecf41be2ecd007853c2db19e1c6a038cf356cb9e"
110+
fixed_commit="f69557e325c5bb9e4e250bb0ec2db12d85298211"
111+
test_module="pinot-core"
112+
test_case="org.apache.pinot.queries.ForwardIndexHandlerReloadQueriesTest#testSelectQueries"
27113
```
28-
Example:
114+
115+
Then simply run:
29116
```shell
30-
nohup ./git-bisect-runner.sh --flaky ecf41be2ecd007853c2db19e1c6a038cf356cb9e --fixed f69557e325c5bb9e4e250bb0ec2db12d85298211 --module pinot-core --test org.apache.pinot.queries.ForwardIndexHandlerReloadQueriesTest#testSelectQueries --nondex-version "2.1.7" --mvn-install "-Dspotless.skip" &> git_bisect_output.log
117+
./git-bisect-runner.sh
31118
```
32119

33-
The output of this execution will give the commit where the flaky test was fixed. Messages within process can be found in log file within project directory.
120+
**Note:** Command-line arguments will override default values if both are provided.
121+
### Output
122+
123+
The script will:
124+
1. Display configuration settings
125+
2. Validate all inputs and commits
126+
3. Run the bisect process with colored status updates
127+
4. Show the commit that fixed the flaky test
128+
5. Display the bisect log
129+
6. Automatically reset the bisect session
130+
131+
### Troubleshooting
132+
133+
The script includes built-in error handling, but you can also use these git commands:
134+
135+
- Check bisect status: `git bisect log`
136+
- View current state: `git bisect visualize` or `git bisect view`
137+
- Manually reset if needed: `git bisect reset`
138+
139+
### Exit Codes
34140

35-
To check for errors during process, run: `git bisect status`
36-
To check log as process is running, run: `git bisect log`
37-
To stop and reset bisect process, run: `git bisect reset`
141+
- `0`: Test is flaky (old state)
142+
- `1`: Test is non-flaky (new state)
143+
- `125`: Build failed, skip this commit (git bisect special code)

‎auto-check-fix-commit/git-bisect-runner.sh‎

Lines changed: 136 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,44 +1,157 @@
11
#!/bin/bash
22

3-
mvn_options=""
4-
nondex_version=""
3+
set -euo pipefail # Exit on error, undefined variables, and pipe failures
4+
5+
# Default values
56
flaky_commit=""
67
fixed_commit=""
78
test_module=""
89
test_case=""
10+
nondex_version=""
11+
mvn_options=""
912

13+
# Usage function
14+
usage() {
15+
cat << EOF
16+
Usage: $0 --flaky <COMMIT> --fixed <COMMIT> --module <PATH> --test <TEST> --nondex-version <VERSION> [--mvn-install <OPTIONS>]
17+
18+
Required arguments:
19+
--flaky Commit hash where the test was flaky
20+
--fixed Commit hash where the test was fixed
21+
--module Maven module path
22+
--test Test case name
23+
--nondex-version NonDex version to use
24+
25+
Optional arguments:
26+
--mvn-install Additional Maven options (e.g., "-Dlicense.skip")
27+
28+
Example:
29+
$0 --flaky ecf41be2ecd007853c2db19e1c6a038cf356cb9e \
30+
--fixed f69557e325c5bb9e4e250bb0ec2db12d85298211 \
31+
--module pinot-core \
32+
--test org.apache.pinot.queries.ForwardIndexHandlerReloadQueriesTest#testSelectQueries \
33+
--nondex-version "2.1.7" \
34+
--mvn-install "-Dspotless.skip"
35+
EOF
36+
exit 1
37+
}
38+
39+
# Parse arguments
1040
while [[ $# -gt 0 ]]; do
1141
case "$1" in
12-
'--flaky') flaky_commit="$2"; shift 2 ;;
13-
'--fixed') fixed_commit="$2"; shift 2 ;;
14-
'--module') test_module="$2"; shift 2 ;;
15-
'--test') test_case="$2"; shift 2 ;;
16-
'--nondex-version') nondex_version="$2"; shift 2 ;;
17-
'--mvn-install') mvn_options="$2"; shift 2 ;;
18-
*) echo "Unknown argument: $1"; exit 1 ;;
42+
--flaky) flaky_commit="$2"; shift 2 ;;
43+
--fixed) fixed_commit="$2"; shift 2 ;;
44+
--module) test_module="$2"; shift 2 ;;
45+
--test) test_case="$2"; shift 2 ;;
46+
--nondex-version) nondex_version="$2"; shift 2 ;;
47+
--mvn-install) mvn_options="$2"; shift 2 ;;
48+
-h|--help) usage ;;
49+
*) echo "Error: Unknown argument: $1"; usage ;;
1950
esac
2051
done
2152

22-
echo "Flaky Commit: $flaky_commit"
23-
echo "Fixed Commit: $fixed_commit"
24-
echo "Test Module: $test_module"
25-
echo "Test Case: $test_case"
26-
echo "NonDex Version: $nondex_version"
27-
echo "Maven Options: $mvn_options"
53+
# Validate required arguments and report missing ones
54+
missing_args=()
55+
[[ -z "$flaky_commit" ]] && missing_args+=("--flaky")
56+
[[ -z "$fixed_commit" ]] && missing_args+=("--fixed")
57+
[[ -z "$test_module" ]] && missing_args+=("--module")
58+
[[ -z "$test_case" ]] && missing_args+=("--test")
59+
[[ -z "$nondex_version" ]] && missing_args+=("--nondex-version")
60+
61+
if [[ ${#missing_args[@]} -gt 0 ]]; then
62+
echo "Error: Missing required argument(s): ${missing_args[*]}"
63+
echo ""
64+
usage
65+
fi
66+
67+
# Display configuration
68+
echo "=== Git Bisect Configuration ==="
69+
echo "Flaky Commit: $flaky_commit"
70+
echo "Fixed Commit: $fixed_commit"
71+
echo "Test Module: $test_module"
72+
echo "Test Case: $test_case"
73+
echo "NonDex Version: $nondex_version"
74+
echo "Maven Options: ${mvn_options:-<none>}"
75+
echo "================================"
76+
echo ""
2877

29-
if [[ -z "$flaky_commit" || -z "$fixed_commit" || -z "$test_module" || -z "$test_case" || -z "$nondex_version" ]]; then
30-
echo "Error: Missing required argument(s)."
78+
# Verify git bisect script exists
79+
if [[ ! -f "./git-bisect-script.sh" ]]; then
80+
echo "Error: git-bisect-script.sh not found in current directory"
3181
exit 1
3282
fi
3383

34-
git checkout $fixed_commit
84+
# Verify commits exist
85+
if ! git rev-parse "$flaky_commit" >/dev/null 2>&1; then
86+
echo "Error: Flaky commit '$flaky_commit' not found"
87+
exit 1
88+
fi
3589

36-
git bisect start
90+
if ! git rev-parse "$fixed_commit" >/dev/null 2>&1; then
91+
echo "Error: Fixed commit '$fixed_commit' not found"
92+
exit 1
93+
fi
3794

38-
git bisect bad
95+
# Clean up any previous bisect session
96+
if git bisect log >/dev/null 2>&1; then
97+
echo "Warning: Previous bisect session found. Resetting..."
98+
git bisect reset
99+
fi
100+
101+
# Start bisect process
102+
echo "Starting git bisect..."
103+
104+
git checkout "$fixed_commit" || {
105+
echo "Error: Failed to checkout fixed commit"
106+
exit 1
107+
}
108+
109+
git bisect start --term-old=flaky --term-new=non-flaky || {
110+
echo "Error: Failed to start bisect"
111+
exit 1
112+
}
113+
114+
git bisect non-flaky || {
115+
echo "Error: Failed to mark commit as non-flaky"
116+
git bisect reset
117+
exit 1
118+
}
119+
120+
git checkout "$flaky_commit" || {
121+
echo "Error: Failed to checkout flaky commit"
122+
git bisect reset
123+
exit 1
124+
}
125+
126+
git bisect flaky || {
127+
echo "Error: Failed to mark commit as flaky"
128+
git bisect reset
129+
exit 1
130+
}
131+
132+
# Run bisect with the test script
133+
echo "Running bisect to find the fix commit..."
134+
git bisect run ./git-bisect-script.sh "$test_module" "$test_case" "$nondex_version" "$mvn_options"
135+
136+
# Capture bisect result
137+
bisect_exit_code=$?
138+
139+
echo ""
140+
echo "=== Bisect Complete ==="
141+
if [[ $bisect_exit_code -eq 0 ]]; then
142+
echo "Successfully identified the commit that fixed the flaky test"
143+
else
144+
echo "Bisect completed with exit code: $bisect_exit_code"
145+
fi
39146

40-
git checkout $flaky_commit
147+
# Show bisect log
148+
echo ""
149+
echo "=== Bisect Log ==="
150+
git bisect log
41151

42-
git bisect good
152+
# Reset bisect
153+
echo ""
154+
echo "Resetting bisect session..."
155+
git bisect reset
43156

44-
git bisect run ./git-bisect-script.sh $test_module $test_case "$nondex_version" "$mvn_options"
157+
exit $bisect_exit_code

‎auto-check-fix-commit/git-bisect-script.sh‎

Lines changed: 29 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,35 @@
11
#!/bin/bash
22

3-
echo "Started Execution"
3+
set -uo pipefail # Exit on undefined variables and pipe failures
44

5-
test_module=$1
6-
test_case=$2
7-
nondex_version=$3
8-
mvn_options=$4
5+
# Parse arguments
6+
test_module="$1"
7+
test_case="$2"
8+
nondex_version="$3"
9+
mvn_options="${4:-}" # Optional, default to empty
910

10-
echo "Started Build"
11-
mvn clean install -pl $test_module -am -DskipTests -Dlicense.skip $mvn_options
12-
if [[ "$?" -ne 0 ]]; then
13-
echo "Build Failed"
14-
exit 1
15-
else
16-
mvn -pl $test_module edu.illinois:nondex-maven-plugin:$nondex_version:nondex -Dtest=$test_case -DnondexRuns=5 -Dlicense.skip=true
17-
if [[ "$?" -ne 0 ]]; then
18-
echo "NonDex Test Failed"
19-
exit 0
20-
else
21-
exit 1
22-
fi
11+
# Log current commit being tested
12+
echo "=== Testing Commit: $(git rev-parse --short HEAD) ==="
13+
echo "Commit: $(git log -1 --oneline)"
14+
echo ""
15+
16+
# Build the project
17+
echo "Starting build..."
18+
if ! mvn clean install -pl "$test_module" -am -DskipTests -Dlicense.skip $mvn_options; then
19+
echo "Build Failed - Cannot determine if test is flaky"
20+
exit 125 # Git bisect special code: skip this commit
21+
fi
22+
23+
echo "Build successful"
24+
echo ""
25+
26+
# Run NonDex test
27+
echo "Running NonDex test..."
28+
if ! mvn -pl "$test_module" edu.illinois:nondex-maven-plugin:"$nondex_version":nondex \
29+
-Dtest="$test_case" -DnondexRuns=5 -Dlicense.skip=true $mvn_options; then
30+
echo "NonDex Test Failed - Test is FLAKY at this commit"
31+
exit 0 # Test is flaky (old state)
2332
fi
2433

25-
exit 0
34+
echo "NonDex Test Passed - Test is NON-FLAKY at this commit"
35+
exit 1 # Test is non-flaky (new state)

0 commit comments

Comments
 (0)