Skip to content

fix: use millisecond precision for dataEntry metadata ETag - #24046

Merged
stian-sandvold merged 5 commits into
masterfrom
fix/dataset-metadata-etag-millis-precision
Jun 1, 2026
Merged

fix: use millisecond precision for dataEntry metadata ETag#24046
stian-sandvold merged 5 commits into
masterfrom
fix/dataset-metadata-etag-millis-precision

Conversation

@stian-sandvold

Copy link
Copy Markdown
Contributor

What

The GET /api/dataEntry/metadata ETag is built from the latest lastUpdated timestamp across the relevant metadata types, hashed together with the user UID. That timestamp was formatted with DateUtils.toLongDate, which truncates to whole seconds. This bumps it to DateUtils.toLongDateWithMillis so sub-second changes are reflected in the ETag.

Why

With second-level precision, two distinct metadata states that happen within the same wall-clock second produce the same ETag. Consequences:

  • Correctness: a client that fetched the metadata and then sees it change within the same second can be served a stale 304 Not Modified and miss the update until the next change.
  • CI flakiness: DataSetMetadataTest.dataSetMetadataEtagFunctionalityTest creates a dataset and asserts the ETag changed (line 103). When the create lands in the same second as the preceding read, the ETags collide and the assertion fails. This has been intermittently failing the Run api tests job on master (the same test/assertion across multiple unrelated commits, ~25-30% of runs).

Millisecond precision makes same-instant collisions astronomically unlikely and fixes both issues. The MD5 hash length is unchanged, so the existing 34-character ETag assertions still hold.

Testing

  • mvn -pl dhis-web-api spotless:check passes.
  • The change is a one-line precision bump to the existing ETag input; dataSetMetadataEtagFunctionalityTest continues to cover the behaviour.

🤖 Generated with Claude Code

… collisions

The dataEntry/metadata ETag was hashed from a last-modified timestamp
truncated to whole seconds (DateUtils.toLongDate). When a metadata change
landed in the same wall-clock second as a previous fetch, the ETag was
unchanged, so clients could be served a stale 304 Not Modified.

This also intermittently broke CI: DataSetMetadataTest.
dataSetMetadataEtagFunctionalityTest asserts the ETag changes after a
dataset is created, which collided whenever the create happened in the
same second as the prior read (~25-30% of master api-test runs).

Switch to DateUtils.toLongDateWithMillis so the timestamp keeps
millisecond precision. The MD5 hash length is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@stian-sandvold
stian-sandvold requested review from a team and jbee May 31, 2026 07:22
Update the expected hash in testGetEtag to match the millisecond-precision
input, and add testGetEtagDistinguishesSubSecondChanges to lock in that
two timestamps one millisecond apart now produce different ETags.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented May 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.96%. Comparing base (17f8133) to head (6970795).
⚠️ Report is 11 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (17f8133) and HEAD (6970795). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (17f8133) HEAD (6970795)
unit 1 0
Additional details and impacted files
@@              Coverage Diff              @@
##             master   #24046       +/-   ##
=============================================
- Coverage     69.04%   56.96%   -12.09%     
+ Complexity      709      555      -154     
=============================================
  Files          3684     3684               
  Lines        141646   141620       -26     
  Branches      16453    16452        -1     
=============================================
- Hits          97797    80668    -17129     
- Misses        36243    53524    +17281     
+ Partials       7606     7428      -178     
Flag Coverage Δ
integration 49.65% <100.00%> (+<0.01%) ⬆️
integration-h2 29.64% <0.00%> (-0.01%) ⬇️
unit ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
.../java/org/hisp/dhis/webapi/utils/ContextUtils.java 57.77% <100.00%> (ø)

... and 986 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 05b1dbc...6970795. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@stian-sandvold
stian-sandvold requested a review from netroms May 31, 2026 11:07

@jbee jbee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel this recognizes the issue and then applies the wrong fix. The cause was to not use the timestamp as it nature of being a number but to use some formatting which just makes is unnecessary complicated and error prone. So I suggest the fix is to use timestamp as numbers, not format the number differently.

@stian-sandvold

Copy link
Copy Markdown
Contributor Author

Do you mean @jbee to not use the md5 hash to generate the etag string? Because the main change with the timestamp here is not that we format it differently, but add more granular information by adding milisecs in addition to seconds, since seconds where to rough to pass the test.

In theory, this fix leaves the test still flaky, but to be able to trigger that failure we would need a significantly faster execution time of the tested code

@jbee

jbee commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

@stian-sandvold I am saying this

String value =
        String.format("%s-%s", DateUtils.toLongDateWithMillis(lastModified), user.getUid());

could and should just be

String value = String.format("%d-%s", lastModified, user.getUid());

Comment thread dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/utils/ContextUtils.java Outdated
@stian-sandvold
stian-sandvold force-pushed the fix/dataset-metadata-etag-millis-precision branch from 3c76cce to 6970795 Compare June 1, 2026 13:02
Co-authored-by: Stian Sandvold <stian@dhis2.org>
@sonarqubecloud

sonarqubecloud Bot commented Jun 1, 2026

Copy link
Copy Markdown

@stian-sandvold
stian-sandvold enabled auto-merge (squash) June 1, 2026 13:27
@stian-sandvold
stian-sandvold merged commit a7561f4 into master Jun 1, 2026
22 checks passed
@stian-sandvold
stian-sandvold deleted the fix/dataset-metadata-etag-millis-precision branch June 1, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants