osquery_manager: generate ECS section mappings from config by marc-gr · Pull Request #18989 · elastic/integrations

marc-gr · 2026-05-13T10:48:31Z

Proposed commit message

osquery_manager: generate ECS section mappings from config

Move the osquery-gen ECS keep list into config.yml and pass it through the generator instead of embedding a separate text file. This lets the generated ecs.yml include object/nested parent mappings such as dll.pe.sections, file.pe.sections, and related threat.*.file.pe.sections fields consistently when artifacts are regenerated.

Refresh generated osquery artifacts for osquery 5.23.0 and bump the package to 1.29.0 so the changelog captures both the ECS generation fix and osquery version update.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Regenerated osquery_manager artifacts with osquery-gen.
Confirmed the PR osquery_manager: update ECS mapping #16650 ECS parent *.sections fields are generated in data_stream/result/fields/ecs.yml.

How to test this PR locally

cd packages/osquery_manager/_dev/scripts/osquery-gen
go run . -config ./config.yml -skip-package-check
go test ./...

Also run:

git diff --check

Not run locally: full elastic-package check.

Related issues

Relates osquery_manager: update ECS mapping #16650

Screenshots

N/A

Move ECS keep fields into osquery-gen config so parent object mappings such as *.sections are regenerated consistently, and refresh generated artifacts for osquery 5.23.0.

github-actions · 2026-05-13T10:49:41Z

Vale Linting Results

Summary: 1 warning found

⚠️ Warnings (1)

File	Line	Rule	Message
packages/osquery_manager/_dev/scripts/osquery-gen/README.md	7	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

infra-vault-gh-plugin-prod · 2026-05-13T10:58:29Z

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

github-actions · 2026-05-13T11:14:02Z

TL;DR

The failure is configuration-related, not a test/code regression: CI tries to boot Elastic Stack 9.4.2, but Docker images for that tag are unavailable (manifest unknown).
The immediate fix is to avoid pinning osquery_manager to an unavailable patch version.

Remediation

Update packages/osquery_manager/manifest.yml to use a supported Kibana constraint (e.g. revert conditions.kibana.version from ^9.4.2 to ^9.4.0, or another currently published version).
Re-run the Buildkite job after the manifest constraint change.

Investigation details

Root Cause

packages/osquery_manager/manifest.yml:11 in this PR changed Kibana constraint from ^9.4.0 to ^9.4.2, which drives stack selection during package testing.

/.buildkite/scripts/common.sh:479-497 computes the stack version from package constraints when STACK_VERSION is unset. The test step then calls elastic-package stack up --version <resolved-version>.

Buildkite failed while pulling stack images for 9.4.2, so tests never reached package logic.

Evidence

Build: https://buildkite.com/elastic/integrations/builds/42784
Job/step: Check integrations osquery_manager
PR diff evidence:
- packages/osquery_manager/manifest.yml
  - conditions.kibana.version: "^9.4.0" → "^9.4.2"
Key log excerpts:
- Error response from daemon: manifest for docker.elastic.co/elastic-agent/elastic-agent-wolfi:9.4.2 not found: manifest unknown
- Error response from daemon: manifest for docker.elastic.co/elasticsearch/elasticsearch:9.4.2 not found: manifest unknown
- booting up the stack failed ... exit status 18: manifest ... not found

Verification

Not run locally in this workflow (read-only detective analysis).

Follow-up

If 9.4.2 is truly required for a specific API/behavior, use a stack version/tag that is actually published in CI (or wait for artifact availability) before re-enabling that constraint.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

#18989 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
osquery_manager: generate ECS section mappings from config #18989 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

elasticmachine · 2026-05-27T13:39:40Z

💚 Build Succeeded

Buildkite Build
Commit: 6f0d7cd

History

💔 Build #42784 failed fc6c4a1

tomsonpl

Solid fix for a real bug (#16650 follow-through). The generator refactor is clean and the config-driven model is consistent with the rest of the tool. Left a few comments inline — main ask is (1) a unit test covering the parent-object keep-list path.

tomsonpl · 2026-05-28T09:53:10Z

-			fieldName := joinPath(parent, name)
 			typ, _ := m["type"].(string)
-			if (inSet(keepFields, fieldName) || isAllowedECSType(typ)) && name != "@timestamp" {
+			if !kept && isAllowedECSType(typ) && name != "@timestamp" {


nit: the inverted condition (!kept && isAllowedECSType(typ)) reads as "skip the type check when kept," but the actual intent is "kept-list takes priority and also emits the parent object node; the leaf type check still applies to non-kept children." A one-line comment above the kept block on line 538 would save the next reader from re-deriving this.

Also, name != "@timestamp" is now checked on both branches — minor duplication, harmless, but could be lifted to a single guard at the top of the loop body.

tomsonpl · 2026-05-28T09:53:10Z

 			typ, _ := m["type"].(string)
-			if (inSet(keepFields, fieldName) || isAllowedECSType(typ)) && name != "@timestamp" {
+			if !kept && isAllowedECSType(typ) && name != "@timestamp" {
 				*ecsFields = append(*ecsFields, fieldName)


Suggest adding a unit test for collectECSFieldNames (or generateECSFieldsYAML) that feeds a synthetic ECS YAML with a file.pe.sections-shaped object node + a keep_fields containing that parent path, and asserts the parent appears in the output.

This is the substance of the fix and right now it's only validated indirectly via the regenerated ecs.yml. The repo already runs go test ./... for this tool, so a generator_test.go would be cheap and lock the contract in.

tomsonpl · 2026-05-28T09:53:10Z


 	log.Printf("Resolved versions: osquery=%s beats=%s ecs=%s", osqueryVersion, beatsRef, ecsVersion)
-	if err := generateArtifacts(outputRoot, osqueryVersion, ecsVersion, beatsRef, !*skipPackageCheck); err != nil {
+	if err := generateArtifacts(outputRoot, osqueryVersion, ecsVersion, beatsRef, cfg.ECS.KeepFields, !*skipPackageCheck); err != nil {


If ecs.keep_fields is omitted from config.yml, cfg.ECS.KeepFields is nil and behavior silently degrades to "leaves + isAllowedECSType only" — i.e. the parent-object fix this PR introduces disappears with no warning.

A log.Printf("ECS keep fields: %d entries", len(cfg.ECS.KeepFields)) line near the existing Resolved versions: log would catch accidental empty configs in CI output.

tomsonpl · 2026-05-28T09:53:10Z

+      link: https://github.com/elastic/integrations/pull/16650
+    - description: Upgrade osquery version to 5.23.0
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/16650


Minor: the "Upgrade osquery version to 5.23.0" entry links to #16650, but #16650 tracks the ECS *.sections mapping issue, not the osquery version bump. Consider linking this PR (#18989) itself, or leaving it without a link if there's no standalone tracking issue for the osquery bump.

…8989) Move ECS keep fields into osquery-gen config so parent object mappings such as *.sections are regenerated consistently, and refresh generated artifacts for osquery 5.23.0.

osquery_manager: generate ECS section mappings from config

fc6c4a1

Move ECS keep fields into osquery-gen config so parent object mappings such as *.sections are regenerated consistently, and refresh generated artifacts for osquery 5.23.0.

marc-gr requested a review from a team as a code owner May 13, 2026 10:48

marc-gr requested review from gergoabraham and pzl May 13, 2026 10:48

marc-gr added enhancement New feature or request Integration:osquery_manager Osquery Manager Team:Security-Windows Platform Security Windows Platform team [elastic/sec-windows-platform] labels May 13, 2026

This comment has been minimized.

Sign in to view

andrewkroh added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Team:Defend Workflows Security team for Endpoint and OSQuery workflows [elastic/security-defend-workflows] labels May 13, 2026

pzl requested review from tomsonpl and removed request for pzl May 14, 2026 13:18

Merge branch 'main' into fix/osquery-manager-ecs-generation

6f0d7cd

marc-gr enabled auto-merge (squash) May 27, 2026 13:41

tomsonpl approved these changes May 28, 2026

View reviewed changes

marc-gr merged commit 7000d19 into elastic:main May 28, 2026
9 checks passed

github-actions Bot mentioned this pull request May 28, 2026

[breaking-change] Undocumented breaking changes detected (2026-05-28) #19275

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

osquery_manager: generate ECS section mappings from config#18989

osquery_manager: generate ECS section mappings from config#18989
marc-gr merged 2 commits into
elastic:mainfrom
marc-gr:fix/osquery-manager-ecs-generation

marc-gr commented May 13, 2026

Uh oh!

github-actions Bot commented May 13, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented May 13, 2026

Uh oh!

This comment has been minimized.

github-actions Bot commented May 13, 2026

Root Cause

Evidence

Verification

Follow-up

Uh oh!

elasticmachine commented May 27, 2026

Uh oh!

tomsonpl left a comment

Uh oh!

tomsonpl May 28, 2026

Uh oh!

tomsonpl May 28, 2026

Uh oh!

tomsonpl May 28, 2026

Uh oh!

tomsonpl May 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

marc-gr commented May 13, 2026

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

github-actions Bot commented May 13, 2026

Vale Linting Results

Uh oh!

infra-vault-gh-plugin-prod Bot commented May 13, 2026

Uh oh!

This comment has been minimized.

github-actions Bot commented May 13, 2026

TL;DR

Remediation

Root Cause

Evidence

Verification

Follow-up

Uh oh!

elasticmachine commented May 27, 2026

💚 Build Succeeded

History

Uh oh!

tomsonpl left a comment

Choose a reason for hiding this comment

Uh oh!

tomsonpl May 28, 2026

Choose a reason for hiding this comment

Uh oh!

tomsonpl May 28, 2026

Choose a reason for hiding this comment

Uh oh!

tomsonpl May 28, 2026

Choose a reason for hiding this comment

Uh oh!

tomsonpl May 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants