chore(perf): capture failed-request bodies in tracker perf test#24507
Closed
enricocolasante wants to merge 3 commits into
Closed
chore(perf): capture failed-request bodies in tracker perf test#24507enricocolasante wants to merge 3 commits into
enricocolasante wants to merge 3 commits into
Conversation
The "Performance tests compare" workflow intermittently fails with a storm of 409 CONFLICT responses on every `POST /api/tracker` import (and, as a downstream effect, empty exports). The failure is environmental/flaky - it has hit both the baseline and the candidate side - but it has been undiagnosable because Gatling's simulation.log records only the status code, not the response body, and the harness does not capture DHIS2 server logs by default. So the actual import error (the 409 payload) is lost. Enable Gatling's `io.gatling.http.engine.response` logger at DEBUG, which dumps the full request + response (including body) for FAILED requests only. Green runs have no failures, so this is silent there and does not perturb the perf measurement. Override via -DGATLING_RESPONSE_LOG_LEVEL. This makes the next CI occurrence self-diagnosing. Also add scripts/reproduce-tracker-409.sh: runs the LOAD profile in a loop against a chosen image, preserving each iteration's full console output and stopping on the first failure so the captured 409 body can be inspected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Temporary diagnostic instrumentation (NOT for merge) to root-cause the flaky tracker-import 409 storm where preheat fails to resolve Program/OrgUnit that demonstrably exist. Prior SQL-log capture showed the failure is in-memory (Program never issues a uid query), so SQL logs can't see it. - AbstractSchemaStrategy.queryForIdentifiableObjects: [preheat-diag] logs requested-vs-resolved per metadata class (class, idScheme, requested/resolved counts, requested + resolved UIDs). Directly shows "class=Program requested=1 resolved=0" during the storm. - DefaultQueryService.queryObjects: [query-diag] logs DB-engine result size vs in-memory-filter outcome for Program/OrgUnit/TET/ProgramStage, to tell "DB returned nothing" apart from "in-memory filter dropped them". - docker/log4j2.xml: enable DEBUG for the two packages so the lines land in the container log (dhis.log) during the perf run. All gated on isDebugEnabled(); silent unless enabled. Revert once root cause is found. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The storm hits a DIFFERENT metadata class each time (Program+OrgUnit, then DataElement, then TEA) and the failing resolution issues NO SQL at all. Widen [query-diag] to fire on ANY empty result (not just a fixed type set), and log which branch (PRESET vs DBPLAN), preset size, DB-engine result size, in-memory filtering, and the planned db/memory queries so we can see where the id-in filter landed. Diagnostic only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Contributor
Author
|
Closing: this branch was diagnostic instrumentation only (Gatling failed-response body logging, Root cause found and fixed in #24508 (thread-safe eager store-map init in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Performance tests compare workflow intermittently fails with a storm of
409 CONFLICTon everyPOST /api/trackerimport (e.g. run 29728194277). When it happens, all 180 import requests are KO for the whole lifetime of the DHIS2 container (warmup and measured run), and the export scenarios then find 0 events.The failure is environmental / flaky, not a code regression — it has hit both the baseline and the candidate side across different runs. It has stayed undiagnosable because:
simulation.logrecords only the status code, not the response body — so the actual 409 payload (the import error) is discarded.CAPTURE_DHIS2_LOGSis off in the compare workflow).Changes
logback-test.xml— enable Gatling'sio.gatling.http.engine.responselogger atDEBUG. Gatling dumps the full request + response (URL, headers, body) for failed requests only at this level; green runs have no failures so it is silent and does not perturb the perf measurement. Overridable via-DGATLING_RESPONSE_LOG_LEVEL(TRACE= all requests,WARN= off). This makes the next CI occurrence self-diagnosing.scripts/reproduce-tracker-409.sh— runs the LOAD profile in a loop against a chosen image, preserving each iteration's full console output and stopping on the first failure so the captured 409 body can be inspected.Next steps
Once merged, re-run the compare workflow (or the repro script) until the storm recurs; the DEBUG dump will reveal the actual import error and let us fix the root cause. This PR only adds instrumentation — it does not attempt a fix.
🤖 Generated with Claude Code