Skip to content

fix flaky cookieAccess tests in slow CI#4613

Open
watson wants to merge 1 commit into
mainfrom
watson/fix-flake
Open

fix flaky cookieAccess tests in slow CI#4613
watson wants to merge 1 commit into
mainfrom
watson/fix-flake

Conversation

@watson
Copy link
Copy Markdown
Collaborator

@watson watson commented May 13, 2026

Motivation

The document.cookie fallback variant of cookieAccess.spec is occasionally flaky on CI. The failure mode is should notify when cookie is changed externally reporting:

Expected spy observer to have been called once. It was called 0 times.

The test is unrelated to any specific feature work — it can be reproduced from main on slow CI runs.

Changes

The root cause is a real-time / mock-time mismatch. When mockClock() is installed, new Date() returns the mocked time. setCookie() computes the cookie's expiry from new Date() + expireDelay, but since mockClock() initializes the mocked time to the current real time, the absolute expiry the browser sees is REAL_T_install + expireDelay. The browser then expires cookies against real wall-clock time — it doesn't know anything about the mock.

With the previous expireDelay = 1000 ms, any CI run slow enough for >1 s of real time to elapse between mockClock() and the polling interval reading document.cookie would see the cookie already expired. getCookies() returned [], matched previousCookieValues = [] captured at construction, and the observable never notified.

Two complementary fixes in packages/core/src/browser/cookieAccess.spec.ts:

  • Bump the test cookie expireDelay from 1000 ms to ONE_MINUTE at every call site, leaving real time comfortable headroom to elapse during a slow test without the cookie disappearing.
  • Register a generic deleteCookie(COOKIE_NAME) cleanup in each setup() variant. Three tests in the file (should pass empty array when cookie does not exist, should write the value returned by the callback, should notify the observable after writing) write the test cookie via getAllAndSet without registering their own cleanup. Previously the short expiry incidentally limited the blast radius of those leaks; under the longer expiry, the leaked cookies would otherwise persist into the next test and break should pass empty array when cookie does not exist under certain Jasmine random orderings. The generic cleanup is registered first so it runs last (LIFO) and is a no-op when a more specific cleanup already deleted the cookie.

No production code changes — this is test-only.

Test instructions

To stress the random ordering (including the originally failing seed 00743):

for seed in 00001 00100 00743 12345 99999 50000; do
  yarn test:unit --spec packages/core/src/browser/cookieAccess.spec.ts --seed $seed
done

Checklist

  • Tested locally
  • Tested on staging
  • Added unit tests for this change
  • Added e2e/integration tests for this change
  • Updated documentation and/or relevant AGENTS.md file

The `document.cookie fallback` variant of cookieAccess.spec was flaky
because of a mismatch between mocked time and real time. With
`mockClock()` installed, `new Date()` returns the mocked time, so
`setCookie()` computes the cookie's expiry as `MOCKED_T_install +
expireDelay`. `mockClock()` initializes the mocked time to the current
real time, so the absolute value the browser sees is
`REAL_T_install + expireDelay`. The browser then evaluates cookie
expiration against real wall-clock time.

With `expireDelay = 1000` ms, on slow CI runs more than ~1 s of real
time could elapse between `mockClock()` and the polling interval's
read of `document.cookie`. The browser would consider the cookie
expired and `getCookies()` would return []. With
`previousCookieValues` captured as [] at construction, the comparison
matched, the observable never notified, and `should notify when cookie
is changed externally` failed with 0 spy calls.

Two complementary changes:

- Bump the test cookie expireDelay from 1s to ONE_MINUTE everywhere,
  so real time can comfortably elapse during a slow test without the
  cookie disappearing.
- Register a generic `deleteCookie(COOKIE_NAME)` cleanup in each
  `setup()` variant. Three tests (`should pass empty array when cookie
  does not exist`, `should write the value returned by the callback`,
  `should notify the observable after writing`) write the cookie via
  `getAllAndSet` without registering their own cleanup; with the
  longer expiry those cookies would otherwise leak into the next test
  and break `should pass empty array when cookie does not exist` under
  certain random orderings. Registered first so it runs last (LIFO)
  and is a no-op when a more specific cleanup already deleted the
  cookie.
@watson watson requested a review from a team as a code owner May 13, 2026 05:01
Copy link
Copy Markdown
Collaborator Author

watson commented May 13, 2026

This stack of pull requests is managed by Graphite. Learn more about stacking.

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da Bot commented May 13, 2026

Bundles Sizes Evolution

📦 Bundle Name Base Size Local Size 𝚫 𝚫% Status
Rum 169.51 KiB 169.51 KiB 0 B 0.00%
Rum Profiler 5.97 KiB 5.97 KiB 0 B 0.00%
Rum Recorder 21.23 KiB 21.23 KiB 0 B 0.00%
Logs 54.70 KiB 54.70 KiB 0 B 0.00%
Rum Slim 127.85 KiB 127.85 KiB 0 B 0.00%
Worker 22.99 KiB 22.99 KiB 0 B 0.00%
🚀 CPU Performance
Action Name Base CPU Time (ms) Local CPU Time (ms) 𝚫%
RUM - add global context 0.0031 0.003 -3.23%
RUM - add action 0.0164 0.0136 -17.07%
RUM - add error 0.0137 0.0116 -15.33%
RUM - add timing 0.0007 0.0005 -28.57%
RUM - start view 0.0151 0.011 -27.15%
RUM - start/stop session replay recording 0.0013 0.001 -23.08%
Logs - log message 0.0234 0.0207 -11.54%
🧠 Memory Performance
Action Name Base Memory Consumption Local Memory Consumption 𝚫
RUM - add global context 38.75 KiB 37.87 KiB -904 B
RUM - add action 64.74 KiB 63.06 KiB -1.68 KiB
RUM - add timing 36.82 KiB 41.44 KiB +4.62 KiB
RUM - add error 70.81 KiB 73.59 KiB +2.78 KiB
RUM - start/stop session replay recording 41.37 KiB 41.81 KiB +459 B
RUM - start view 486.34 KiB 482.38 KiB -3.96 KiB
Logs - log message 55.35 KiB 53.74 KiB -1.61 KiB

🔗 RealWorld

@datadog-datadog-prod-us1-2
Copy link
Copy Markdown

datadog-datadog-prod-us1-2 Bot commented May 13, 2026

Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 76.96% (+0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 203f5c1 | Docs | Datadog PR Page | Give us feedback!

{
title: 'document.cookie fallback',
setup: () => {
registerCleanupTask(() => deleteCookie(COOKIE_NAME))
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Supposedly we already delete all cookies in the global forEach

If this is not working properly we should probably fix it there, no?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants