Skip to content

Commit 25f80d4

Browse files
cataphractcataphract
committed
Claude instructions to reproduce/troubleshoot CI failures
1 parent 42c7c25 commit 25f80d4

33 files changed

+8183
-0
lines changed

‎.claude/ci/appsec-gradle-integration.md‎

Lines changed: 288 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,288 @@
1+
# AppSec Gradle Integration Tests
2+
3+
## CI Jobs
4+
5+
**Source:** `.gitlab/generate-appsec.php` — generates the appsec-trigger
6+
child pipeline; all job `script:` sections are defined inline in this
7+
file.
8+
9+
| CI Job | Image | What it does |
10+
|--------|-------|-------------|
11+
| `appsec integration tests: [{target}]` | `docker:24.0.4-gbi-focal` | Gradle integration tests with C++ helper (release/zts variants) |
12+
| `appsec integration tests (helper-rust): [{target}]` | same | Same tests with Rust helper (`-PuseHelperRust`); includes `debug` variant |
13+
| `appsec integration tests (ssi): [{target}]` | same | SSI mode (PHP 8.3 only) |
14+
| `helper-rust build and test` | same | `cargo fmt --check` + build + unit tests |
15+
| `helper-rust code coverage` | same | Unit test coverage via `cargo-llvm-cov` |
16+
| `helper-rust integration coverage` | same | Integration coverage collection (not needed locally) |
17+
18+
Runner: `docker-in-docker:amd64`
19+
Matrix: PHP 7.0--8.5 × release/debug/zts/musl/ssi (varies by job group)
20+
21+
**Important:** `testX.Y-debug` are not gradle targets that are run on CI. They
22+
may, however, be useful for debugging.
23+
24+
CI passes `TERM=dumb` and `--scan -Pbuildscan` to Gradle. `TERM=dumb`
25+
suppresses progress animations in CI logs; both flags are optional
26+
locally.
27+
28+
## Prerequisites
29+
30+
- JDK 17+. If not available via your system package manager, use SDKMAN
31+
(`sdk install java 17`) or download from
32+
https://download.oracle.com/java/17/archive/.
33+
- Docker daemon running
34+
35+
## Working Directory
36+
37+
All `./gradlew` commands run from:
38+
39+
```
40+
appsec/tests/integration/
41+
```
42+
43+
## Running Tests
44+
45+
### Full suite for one PHP target
46+
47+
```bash
48+
./gradlew test8.3-debug --info
49+
```
50+
51+
### Single test (fastest feedback loop)
52+
53+
```bash
54+
./gradlew test8.3-debug --info \
55+
--tests "com.datadog.appsec.php.integration.Apache2FpmTests.Pool environment"
56+
```
57+
58+
The `--tests` filter accepts:
59+
- Full method: `"com.datadog.appsec.php.integration.Apache2FpmTests.Pool environment"`
60+
- Class only: `"*Apache2FpmTests*"` or `"com.datadog.appsec.php.integration.Apache2FpmTests"`
61+
- Wildcard: `"*FpmTests*"`
62+
63+
### With helper-rust (instead of C++ helper)
64+
65+
```bash
66+
./gradlew test8.3-debug -PuseHelperRust --info \
67+
--tests "com.datadog.appsec.php.integration.Apache2FpmTests.Pool environment"
68+
```
69+
70+
This builds the Rust helper via the `buildHelperRust` task (musl build, works on both glibc and musl targets) and stores the binary in the `php-helper-rust` Docker volume.
71+
72+
### C++ helper (default)
73+
74+
Omit `-PuseHelperRust` and `-PhelperBinary`. The C++ helper is built via the `buildAppsec-*` task.
75+
76+
## Image Tags
77+
78+
By default, Gradle resolves Docker images via pinned SHA256 digests in `gradle/tag_mappings.gradle`. To use floating tags (locally-built images or latest from Docker Hub):
79+
80+
```bash
81+
./gradlew test8.3-debug -PfloatingImageTags --info
82+
```
83+
84+
## Available Gradle Tasks
85+
86+
### Test tasks
87+
88+
Pattern: `test{version}-{variant}`
89+
90+
| Variant | Notes |
91+
|---|---|
92+
| `release` | Standard build |
93+
| `debug` | Debug build (assertions enabled) |
94+
| `release-zts` | Thread-safe build |
95+
| `release-musl` | Alpine/musl (only `8.5-release-musl`) |
96+
| `release-ssi` / `debug-ssi` | SSI mode (only PHP 8.3) |
97+
98+
Full list: `./gradlew tasks --group=Verification`
99+
100+
### Helper-rust tasks
101+
102+
| Task | Description |
103+
|---|---|
104+
| `buildHelperRust` | Build helper-rust with musl (universal binary). Output in `php-helper-rust` volume. |
105+
| `testHelperRust` | `cargo fmt --check` + `cargo build --release` + `cargo test --release` (runs inside `php-deps` image) |
106+
| `coverageHelperRust` | Unit test coverage via `cargo-llvm-cov`. Output: `php-helper-rust-coverage` volume. |
107+
| `buildHelperRustWithCoverage` | Build with `-C instrument-coverage` for integration coverage collection. |
108+
| `generateHelperRustIntegrationCoverage` | Merge `.profraw` files into lcov after integration run. |
109+
110+
### Build tasks
111+
112+
| Task | Description |
113+
|---|---|
114+
| `buildTracer-{v}-{var}` | Build ddtrace.so for given PHP version/variant |
115+
| `buildAppsec-{v}-{var}` | Build ddappsec.so (C++ extension + helper) |
116+
| `buildHelperRust` | Build Rust helper (musl, universal) |
117+
| `buildLibddwaf` | Build libddwaf shared library |
118+
119+
### Other tasks
120+
121+
| Task | Description |
122+
|---|---|
123+
| `loadCaches` | Restore Docker volume caches from tarball |
124+
| `saveCaches` | Save Docker volume caches to tarball |
125+
| `clean` | Delete build directory and clean Docker volumes |
126+
| `check` | Run all test tasks |
127+
128+
All tasks: `./gradlew tasks --all`
129+
130+
## Interactive Container (runMain)
131+
132+
Start a test container without running tests (for manual debugging):
133+
134+
```bash
135+
./gradlew runMain8.3-release -PtestClass=com.datadog.appsec.php.integration.Apache2FpmTests
136+
```
137+
138+
The `-PtestClass` property is required (the task is not created without it). Add `-PuseHelperRust` or `-PhelperBinary=...` as needed.
139+
140+
SSI variant:
141+
142+
```bash
143+
./gradlew runMain8.3-release-ssi -PtestClass=com.datadog.appsec.php.integration.Apache2FpmTests
144+
```
145+
146+
## Logs
147+
148+
After a test run, logs are in:
149+
150+
```
151+
build/test-logs/<TestClassName>-<version>-<variant>/
152+
```
153+
154+
For example:
155+
156+
```
157+
build/test-logs/com.datadog.appsec.php.integration.Apache2FpmTests-8.3-debug/
158+
├── access.log
159+
├── appsec.log # PHP extension appsec log
160+
├── error.log # Apache error log
161+
├── helper.log # Helper process log (C++ or Rust)
162+
├── php_error.log
163+
├── php_fpm_error.log
164+
└── sidecar.log
165+
```
166+
167+
To distinguish which helper ran, check `helper.log`:
168+
- Rust: starts with `[INFO] AppSec helper starting`
169+
- C++: starts with `[info]` lines like `Started listening on abstract socket`
170+
171+
## Musl/Alpine Target
172+
173+
The `test8.5-release-musl` target uses an Alpine-based nginx+fpm image. Tests tagged with `@Tag("musl")` are included; untagged tests are excluded.
174+
175+
```bash
176+
./gradlew test8.5-release-musl -PuseHelperRust --info
177+
```
178+
179+
The `buildHelperRust` task already produces a musl-linked binary (built on Alpine with `cargo +nightly`, using LLVM libunwind). The `patchelf --remove-needed libc.musl-*` step makes it load on both musl and glibc systems.
180+
181+
## CI Job Mapping
182+
183+
| CI Job | Gradle Command |
184+
|---|---|
185+
| `appsec integration tests: [test8.3-release]` | `./gradlew test8.3-release` |
186+
| `appsec integration tests (helper-rust): [test8.3-debug]` | `./gradlew test8.3-debug -PuseHelperRust` |
187+
| `appsec integration tests (ssi): [test8.3-release-ssi]` | `./gradlew test8.3-release-ssi` |
188+
| `helper-rust build and test` | `./gradlew testHelperRust` |
189+
| `helper-rust code coverage` | `./gradlew coverageHelperRust` |
190+
| `helper-rust integration coverage` | `./gradlew buildHelperRustWithCoverage` then integration test with `-PuseHelperRustCoverage` |
191+
192+
CI also passes `--scan -Pbuildscan` for Gradle build scans, which is optional locally.
193+
194+
## Docker Volumes
195+
196+
Gradle uses named Docker volumes for build artifacts and caches. Key volumes:
197+
198+
| Volume | Contents |
199+
|---|---|
200+
| `php-helper-rust` | `libddappsec-helper.so` (Rust helper binary) |
201+
| `php-tracer-{v}-{var}` | Built `ddtrace.so` |
202+
| `php-appsec-{v}-{var}` | Built `ddappsec.so` + C++ helper |
203+
| `php-tracer-cargo-cache` | Cargo registry cache |
204+
| `php-tracer-cargo-cache-git` | Cargo git cache |
205+
| `php-appsec-boost-cache` | Boost build cache |
206+
| `php-helper-rust-coverage` | Coverage-instrumented binary + profraw files |
207+
208+
To force a rebuild, remove the relevant volume:
209+
210+
```bash
211+
docker volume rm php-helper-rust
212+
```
213+
214+
To clean everything:
215+
216+
```bash
217+
./gradlew clean
218+
```
219+
220+
## Debugging
221+
222+
Attach a Java debugger to the test runner:
223+
224+
```bash
225+
./gradlew test8.3-debug --tests "*Apache2FpmTests*" --debug-jvm
226+
```
227+
228+
Enable PHP Xdebug in the test container:
229+
230+
```bash
231+
./gradlew test8.3-debug --tests "*Apache2FpmTests*" -PXDEBUG=1
232+
```
233+
234+
## Expected skips
235+
236+
A significant number of tests are skipped on any given target — this is
237+
normal. Skip conditions are `@EnabledIf` guards in the test classes:
238+
239+
| Test class | Skips on | Reason |
240+
|---|---|---|
241+
| `FrankenphpClassicTests`, `FrankenphpWorkerTests` | anything except `8.4-zts` | Requires ZTS + PHP 8.4 |
242+
| `Laravel8xTests` | anything except `7.4` (NTS) | Requires PHP 7.4 non-ZTS |
243+
| `Symfony62Tests` | anything except `8.1` (NTS) | Requires PHP 8.1 non-ZTS |
244+
| `RaspSqliTests` | no MySQL service | Requires a running MySQL |
245+
| `SsiStableConfigTests` | non-SSI variants | Requires `-DSSI=true` |
246+
247+
On `test8.3-debug` expect ~67 skips out of ~300 tests; all are expected.
248+
249+
## Test report
250+
251+
After a run, the HTML report is at:
252+
253+
```
254+
appsec/tests/integration/build/reports/tests/test8.3-debug/index.html
255+
```
256+
257+
(Replace `test8.3-debug` with your target.) Open in a browser for a
258+
structured pass/fail/skip breakdown.
259+
260+
# Debugging
261+
262+
Run gradle with `--debug-jvm`. This will stop for the debugger, indicating so in the output.
263+
When you see the message, start jdb in a tmux session.
264+
265+
If you need to inspect sidecar/helper or PHP issues:
266+
267+
* Put a breakpoint in jdb that stops the test in the appropriate place (usually
268+
just before a request is executed).
269+
* Inside the test container (determine first its id), attach gdb to sidecar
270+
(`pref -f dd-ipc-helper`) or to PHP (usually an apache or an FPM worker -- if
271+
you're investigating code run during processes it will not be the master
272+
process). sidecar requires as a first command `file /proc/<pid>/exe`).
273+
* See [gdb.md](../gdb.md) for more information on how to run gdb. Always read
274+
this file before attempting to use gdb.
275+
276+
## Gotchas
277+
278+
- The `test` task itself is disabled (`tasks['test'].enabled = false`). Use versioned tasks like `test8.3-debug`.
279+
- Docker images are pulled from `docker.io/datadog/dd-appsec-php-ci`. Without `-PfloatingImageTags`, images are resolved by SHA256 digest from `gradle/tag_mappings.gradle`. If a digest is not locally available, Docker will pull it.
280+
- The `buildHelperRust` task uses the `nginx-fpm-php-8.5-release-musl` image (Alpine with Rust nightly). This image must be available locally or pullable.
281+
- On first run, Gradle downloads its wrapper, dependencies, and Docker images. Expect 5-10 minutes. Subsequent runs with warm caches take ~20-50 seconds for a single test.
282+
- **c-ares DNS failure in Alpine containers.** Alpine's `curl` and `git` use
283+
c-ares for DNS, which fails to resolve hosts when the DNS server includes
284+
EDNS COOKIE options in responses (common with home routers). `wget` and
285+
`getent` are unaffected (they use musl's native resolver). This breaks the
286+
`buildAppsec-*-musl` task which needs to `git clone` cmake dependencies.
287+
Fix: pass `--dns 8.8.8.8` to the Docker command.
288+
- `--info` is recommended for seeing test output in the console. Without it, output goes only to the HTML report in `build/reports/tests/`.

0 commit comments

Comments
 (0)