PoC: Use jsontext package to stream components from checkin requests

[michel-laterman]

PoC DO NOT MERGE

What is the problem this PR solves?

Checkin requests from agents running synthetic monitors can exceed the checkin request body size limit due to the length of the components attribute.

How does this PR solve the problem?

Use the upcoming jsontext package to handle parsing the json checkin request.
This allows the checkin validator to read the components array as a stream and issue intermediate updates to ES (set to flush once 4MB of objects have been read).

The fleet-server -> ES update function is currently missing from this PoC as it was intended to demo large request handling on the fleet-server.

This PoC is also missing other safeties - such as max size guards against the rest of the body.

How to test this PR locally

N/A - this is a partial implementation that does not update the components list in ES

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Relates Health check request should be chunked elastic-agent#11981

[mergify]

This pull request does not have a backport label. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[github-actions]

TL;DR

check-ci failed because CI regenerated tracked files and then Check:NoChanges detected a dirty tree. The PR needs regenerated dependency metadata committed (go.mod, NOTICE.txt, and NOTICE-fips.txt).

Remediation

• Run mage check:notice (or mage check:ci) on the PR branch and commit the resulting file updates.
• Ensure go.mod reflects the new github.com/go-json-experiment/json usage after go mod tidy, then re-run CI.

Investigation details

Root Cause

This is a configuration/dependency metadata drift failure in CI, not a test/runtime failure.

check-ci runs mage check:ci, which includes:

• Check.Ci chaining Check.Notice and Check.NoChanges (magefile.go lines 640-643)
• Check.NoChanges enforcing clean git state via git update-index --refresh and git diff-index --exit-code HEAD -- (magefile.go lines 611-631)
• genNotice regenerating NOTICE artifacts as part of Check.Notice (magefile.go lines 515-584)

The Buildkite job shows those generated changes were not committed in the PR branch, so Check.NoChanges failed.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/14848
• Job/step: Run check-ci (.buildkite/scripts/check_ci.sh)
• Key log excerpt (/tmp/gh-aw/buildkite-logs/fleet-server-white_check_mark-run-check-ci.txt):

NOTICE-fips.txt: needs update
NOTICE.txt: needs update
go.mod: needs update
Error: git update-index failure: running "git update-index --refresh" failed with exit code 1

The same log diff also shows CI wanting github.com/go-json-experiment/json moved/updated in go.mod, which is consistent with go mod tidy + notice regeneration requirements.

Verification

• Not run locally against PR branch (read-only detective workflow context); diagnosis is based on Buildkite failure log + repository CI target wiring.

Follow-up

• No matching flaky-test issue found for this failure pattern (repo:elastic/fleet-server is:issue label:flaky-test check-ci).

Note

🔒 Integrity filter blocked 4 items

The following items were blocked because they don't meet the GitHub integrity level.

• PoC: Use jsontext package to stream components from checkin requests #7087 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #7087 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #7087 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• PoC: Use jsontext package to stream components from checkin requests #7087 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[mergify]

This pull request is now in conflicts. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b poc/jsontext-checkin-request-stream upstream/poc/jsontext-checkin-request-stream
git merge upstream/main
git push upstream poc/jsontext-checkin-request-stream