Skip to content

osquery_manager: generate ECS section mappings from config #18989

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
marc-gr merged 2 commits into from
May 28, 2026
Merged

osquery_manager: generate ECS section mappings from config #18989

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fc6c4a1
osquery_manager: generate ECS section mappings from config
marc-gr May 13, 2026
6f0d7cd
Merge branch 'main' into fix/osquery-manager-ecs-generation
marc-gr May 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions packages/osquery_manager/_dev/scripts/osquery-gen/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

Config-driven generator for osquery_manager field/schema generation.

This tool reads `osquery` version and `beats` git selection from `config.yml`,
This tool reads `osquery` version, `beats` git selection, and ECS keep fields from `config.yml`,
reads the ECS git ref from `packages/osquery_manager/_dev/build/build.yml`
(`dependencies.ecs.reference`, e.g. `git@v9.3.0`), resolves the latest matching

Check warning on line 7 in packages/osquery_manager/_dev/scripts/osquery-gen/README.md

View workflow job for this annotation

GitHub Actions / Lint Documentation 

Elastic.Latinisms: Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
patch for osquery/beats when using semver config, and generates:

- `packages/osquery_manager/data_stream/result/fields/osquery.yml`
Expand All @@ -20,13 +20,16 @@

## Config

`config.yml` — osquery and beats only:
`config.yml`:

```yaml
osquery:
version: "5.21.0"
beats:
version: "9.3"
ecs:
keep_fields:
- file.pe.sections
```

**Beats** (extension specs under `elastic/beats`): set **either** an explicit git ref **or** a semver-style `version`. Precedence is **tag > branch > version**:
Expand All @@ -51,6 +54,10 @@
reference: git@v9.3.0
```

**ECS** `keep_fields` entries force specific ECS field names into the generated
`data_stream/result/fields/ecs.yml` even when the field is an object/nested node
or has a type outside the generator's default allow-list.

Notes:

- Output is always written to `packages/osquery_manager` under the detected `integrations` repo root.
Expand Down
53 changes: 52 additions & 1 deletion packages/osquery_manager/_dev/scripts/osquery-gen/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,61 @@
osquery:
# Exact patch (5.21.0) or prefix (5.21) to auto-pick latest patch.
version: "5.22.1"
version: "5.23.0"
beats:
# Git ref for elastic/beats osquery extension specs (raw.githubusercontent.com).
# Precedence: tag > branch > version (semver / prefix resolution).
# tag: "v9.4.2"
# branch: "main"
# Exact patch (9.3.1) or prefix (9.3) to auto-pick latest patch when tag/branch unset:
version: "9.4.0"
ecs:
# ECS field names to include even when they are objects/nested or otherwise
# filtered out by the default type allow-list.
keep_fields:
- dll.pe.sections
- email
- email.attachments
- email.attachments.file.extension
- email.attachments.file.hash.md5
- email.attachments.file.hash.sha1
- email.attachments.file.hash.sha256
- email.attachments.file.hash.sha384
- email.attachments.file.hash.sha512
- email.attachments.file.hash.ssdeep
- email.attachments.file.hash.tlsh
- email.attachments.file.mime_type
- email.attachments.file.name
- email.attachments.file.size
- email.bcc.address
- email.cc.address
- email.content_type
- email.delivery_timestamp
- email.direction
- email.from.address
- email.local_id
- email.message_id
- email.origination_timestamp
- email.reply_to.address
- email.sender.address
- email.subject
- email.to.address
- email.x_mailer
- file.elf.sections
- file.macho.sections
- file.pe.sections
- group.id
- network.iana_number
- network.type
- process.elf.sections
- process.macho.sections
- process.parent.elf.sections
- process.parent.macho.sections
- process.parent.pe.sections
- process.pe.sections
- threat.enrichments
- threat.enrichments.indicator.file.elf.sections
- threat.enrichments.indicator.file.pe.sections
- threat.indicator.file.elf.sections
- threat.indicator.file.pe.sections
- user.group.id
- user.id
39 changes: 0 additions & 39 deletions packages/osquery_manager/_dev/scripts/osquery-gen/ecs_keep_fields.txt
View file
Open in desktop

This file was deleted.

32 changes: 10 additions & 22 deletions packages/osquery_manager/_dev/scripts/osquery-gen/generator.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ package main

import (
"bytes"
_ "embed"
"encoding/csv"
"encoding/json"
"fmt"
Expand Down Expand Up @@ -45,9 +44,6 @@ var (
ecsRestrictedSet = makeSet(ecsRestrictedFields)
)

//go:embed ecs_keep_fields.txt
var ecsKeepFieldsContent string

type kibanaOsqueryTable struct {
Name string `json:"name"`
Description string `json:"description"`
Expand Down Expand Up @@ -123,7 +119,7 @@ type columnInfo struct {
Column osqueryCol
}

func generateArtifacts(outputRoot, osqueryVersion, ecsVersion, beatsRef string, runPackageCheck bool) error {
func generateArtifacts(outputRoot, osqueryVersion, ecsVersion, beatsRef string, ecsKeepFields []string, runPackageCheck bool) error {
outFields := filepath.Join(outputRoot, "packages", "osquery_manager", "data_stream", "result", "fields")
outSchemas := filepath.Join(outputRoot, "packages", "osquery_manager", "schemas")

Expand Down Expand Up @@ -165,7 +161,7 @@ func generateArtifacts(outputRoot, osqueryVersion, ecsVersion, beatsRef string,
if err != nil {
return fmt.Errorf("download ECS fields: %w", err)
}
ecsOut, err := generateECSFieldsYAML(ecsYAML)
ecsOut, err := generateECSFieldsYAML(ecsYAML, ecsKeepFields)
if err != nil {
return fmt.Errorf("generate ecs.yml: %w", err)
}
Expand Down Expand Up @@ -500,20 +496,8 @@ func marshalYAMLWithIndent(v any, indent int) ([]byte, error) {
return buf.Bytes(), nil
}

func loadKeepFields() []string {
var out []string
for _, line := range strings.Split(ecsKeepFieldsContent, "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
out = append(out, line)
}
return out
}

func generateECSFieldsYAML(ecsBeatsYAML []byte) ([]byte, error) {
keepFields := makeSet(loadKeepFields())
func generateECSFieldsYAML(ecsBeatsYAML []byte, ecsKeepFields []string) ([]byte, error) {
keepFields := makeSet(ecsKeepFields)
var root any
if err := yaml.Unmarshal(ecsBeatsYAML, &root); err != nil {
return nil, err
Expand Down Expand Up @@ -550,12 +534,16 @@ func collectECSFieldNames(root any, parent string, keepFields map[string]struct{
if parent == "" && inSet(excludeFromTopLevelSet, name) {
continue
}
fieldName := joinPath(parent, name)
kept := inSet(keepFields, fieldName)
if kept && name != "@timestamp" {
*ecsFields = append(*ecsFields, fieldName)
}
if childFields, hasFields := m["fields"]; hasFields {
collectECSFieldNames(childFields, joinPath(parent, name), keepFields, ecsFields)
} else {
fieldName := joinPath(parent, name)
typ, _ := m["type"].(string)
if (inSet(keepFields, fieldName) || isAllowedECSType(typ)) && name != "@timestamp" {
if !kept && isAllowedECSType(typ) && name != "@timestamp" {

@tomsonpl tomsonpl May 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the inverted condition (!kept && isAllowedECSType(typ)) reads as "skip the type check when kept," but the actual intent is "kept-list takes priority and also emits the parent object node; the leaf type check still applies to non-kept children." A one-line comment above the kept block on line 538 would save the next reader from re-deriving this.

Also, name != "@timestamp" is now checked on both branches — minor duplication, harmless, but could be lifted to a single guard at the top of the loop body.

*ecsFields = append(*ecsFields, fieldName)

@tomsonpl tomsonpl May 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest adding a unit test for collectECSFieldNames (or generateECSFieldsYAML) that feeds a synthetic ECS YAML with a file.pe.sections-shaped object node + a keep_fields containing that parent path, and asserts the parent appears in the output.

This is the substance of the fix and right now it's only validated indirectly via the regenerated ecs.yml. The repo already runs go test ./... for this tool, so a generator_test.go would be cheap and lock the contract in.

}
}
Expand Down
7 changes: 6 additions & 1 deletion packages/osquery_manager/_dev/scripts/osquery-gen/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,14 @@ type beatsConfig struct {
Version string `yaml:"version"`
}

type ecsConfig struct {
KeepFields []string `yaml:"keep_fields"`
}

type config struct {
Osquery versionConfig `yaml:"osquery"`
Beats beatsConfig `yaml:"beats"`
ECS ecsConfig `yaml:"ecs"`
}

type packageBuildYAML struct {
Expand Down Expand Up @@ -93,7 +98,7 @@ func main() {
}

log.Printf("Resolved versions: osquery=%s beats=%s ecs=%s", osqueryVersion, beatsRef, ecsVersion)
if err := generateArtifacts(outputRoot, osqueryVersion, ecsVersion, beatsRef, !*skipPackageCheck); err != nil {
if err := generateArtifacts(outputRoot, osqueryVersion, ecsVersion, beatsRef, cfg.ECS.KeepFields, !*skipPackageCheck); err != nil {

@tomsonpl tomsonpl May 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If ecs.keep_fields is omitted from config.yml, cfg.ECS.KeepFields is nil and behavior silently degrades to "leaves + isAllowedECSType only" — i.e. the parent-object fix this PR introduces disappears with no warning.

A log.Printf("ECS keep fields: %d entries", len(cfg.ECS.KeepFields)) line near the existing Resolved versions: log would catch accidental empty configs in CI output.

log.Fatalf("generate artifacts: %v", err)
}
log.Println("Done.")
Expand Down
8 changes: 8 additions & 0 deletions packages/osquery_manager/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
# newer versions go on top
- version: "1.29.0"
changes:
- description: Generate ECS `*.sections` parent mappings from osquery-gen config
type: bugfix
link: https://github.com/elastic/integrations/pull/16650
- description: Upgrade osquery version to 5.23.0
type: enhancement
link: https://github.com/elastic/integrations/pull/16650

@tomsonpl tomsonpl May 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: the "Upgrade osquery version to 5.23.0" entry links to #16650, but #16650 tracks the ECS *.sections mapping issue, not the osquery version bump. Consider linking this PR (#18989) itself, or leaving it without a link if there's no standalone tracking issue for the osquery bump.

- version: "1.28.1"
changes:
- description: Update ECS mappings for `*.sections` fields in results dataset
Expand Down
2 changes: 2 additions & 0 deletions packages/osquery_manager/data_stream/result/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,8 @@
name: dns.answers.ttl
- external: ecs
name: dns.resolved_ip
- external: ecs
name: email
- external: ecs
name: email.attachments
- external: ecs
Expand Down
Loading
Loading