Repository Feature

Core Repo - (rule management, validation, testing, lib, cicd, etc.)

Problem Description

The repo ships one MITRE ATT&CK version (v18) per rule. The threat field in each rule
TOML was passed verbatim to the Kibana API. Currently, no build-time transform exists to support multiple versions. Supporting v19 (and future versions/frameworks) requires a way to:

1. Store multiple ATT&CK version mappings per rule without duplicating the whole TOML structure.
2. Select which version is shipped at build time, with a sensible default.
3. Generate a first-pass v19 mapping from an existing v18 mapping, without inventing incorrect data.
4. Keep everything DaC-compatible (custom rules + custom config).
5. Remain extensible to additional versions and frameworks without touching core schema code.

Desired Solution

TOML (repo side, authored):           API artifact (shipped to Kibana):
  threat            = [v18 ...]   ──┐
  threat_mappings   = [              │   threat = <selected version's list>
    {framework, version="19", threat=[...]}, │   (threat_mappings stripped, never shipped)
    {framework, version="20", threat=[...]}, │
  ]                                  ──┘

• threat remains the canonical default (v18) mapping. Existing rules are untouched; existing
  Kibana consumers see no change.
• threat_mappings is a new, optional, repo-side-only field holding additional version-tagged
  blocks. Adding it to a rule does not change the rule hash or trigger a version bump until the
  output version is actually flipped.
• At build time (_post_dict_conversion), the configured (framework, version) pair is resolved and
  the matching threat list is emitted as threat. threat_mappings is always stripped before the
  API artifact is finalized.

Key design decisions

1. Static storage: dual mapping in the TOML, not generated at build time

Decision: Both the v18 and v19 mappings are authored and stored statically in each rule's TOML
file. The build does not generate mappings on the fly from a translation table.

Why: A generated-at-build-time approach would mean the repo never has an authoritative v19
mapping that can be reviewed, tested, or diffed. Static storage makes the v19 mapping a first-class
artifact: reviewable in PRs, auditable in git history, and testable independently of the build
pipeline.

2. threat keeps v18; additional versions go in threat_mappings

Decision: The existing threat field holds the default (v18 / "MITRE ATT&CK") mapping
unchanged. New versioned blocks are placed in a separate threat_mappings list.

Why: Changing the semantics of threat would break every existing rule and Kibana consumer.
Keeping threat as the v18 baseline means zero migration cost for the existing rule corpus. The new
field is additive and optional.

3. framework is preserved inside each threat_mappings entry

Decision: Each VersionedThreatMapping wraps a framework string, a version string, and a
threat list whose entries each also carry their own framework. This mirrors the existing threat
field structure exactly.

Why: Reusing ThreatMapping for the inner list means all
existing URL validation, tactic/technique typing, and serialization logic applies automatically. A
@validates_schema check enforces that the inner framework values agree with the wrapper's
framework.

4. threat_mappings is always stripped from the API output

Decision: _post_dict_conversion unconditionally pops threat_mappings from the serialized
dict before it is returned as the API artifact.

Why: Kibana does not understand threat_mappings. Stripping it keeps the schema clean, avoids
API validation failures, and means no existing tooling (rule hash computation, export, etc.) needs
to be updated. The field is purely a repo-side authoring construct and follows a similar convention to DaC json payload preparation that also utilizes a similar mechanism for rule exception lists.

5. Build-time version selection via config + env var

Decision: The output (framework, version) pair is resolved in priority order:

1. DR_THREAT_MAPPING_FRAMEWORK / DR_THREAT_MAPPING_VERSION environment variables.
2. threat_mapping_framework / threat_mapping_version keys in _config.yaml.
3. Defaults: ("MITRE ATT&CK", "18").

When the default pair is resolved, _select_output_threat_mapping returns immediately; threat
is emitted unchanged (the existing behavior, with zero overhead).

Why: Environment variables give operators a low-friction way to flip the shipped version in CI
without modifying config files. Config keys give teams using DaC the ability to set the default
persistently per deployment. The early-return guard for the default pair avoids noise (spurious
warning logs) and preserves backward compatibility exactly.

6. Accuracy-first generation: config carries everything, absent entries are dropped

Decision: The dev attack convert-threat-mappings command is driven by a self-contained YAML
config that explicitly lists every source→destination mapping. Any source tactic/technique/subtechnique
absent from the config (or explicitly set to null) is dropped from the generated v19
block, never guessed or carried over unchanged.

Why: A false mapping (an ID that silently shifts to a wrong v19 target) is worse than a missing
mapping (a detectable gap). The dropped-entry report gives the rule author a clear list of items to
review.

7. Directory of per-pair config files

Decision: Mapping configs live in detection_rules/etc/attack-version-maps/, one YAML file per
(framework, source_version → target_version) triple. The directory is indexed at load time; the
key is (framework, source_version, target_version).

Why: A single flat file would not scale to multiple version pairs or multiple frameworks. A
directory of per-pair files keeps each config small and focused, makes additions zero-friction (drop
a new file, no registry to update), and allows multiple destinations to coexist for chained
upgrades (e.g. v18→v19 and v19→v20 can both be present without conflict).

8. DaC compatibility via @multi_collection and RulesConfig

Decision: The convert-threat-mappings command uses the @multi_collection decorator and
calls RuleCollection.default() by default, which automatically picks up CUSTOM_RULES_DIR and the
configured rule directories. Custom deployments can point at their own mapping configs via
attack_version_maps_dir in their _config.yaml, or pass --config explicitly.

Why: DaC support is required. Reusing the existing @multi_collection /
RuleCollection.default() pattern meant no new DaC plumbing was required; the command
automatically inherits all the existing rule-selection and config-discovery logic.

10. Scaffolding command for new version pairs

Decision: dev attack scaffold-version-map generates an identity-mapping skeleton from the
currently loaded ATT&CK data; every non-revoked, non-deprecated tactic/technique/subtechnique maps
to itself. The output file includes a review header and is intended as a starting point for human
curation, not a ready-to-ship artifact.

Why: Writing the mapping config by hand for 200+ techniques would be error-prone. An identity
skeleton captures the "nothing changed" baseline and makes it easy to spot what actually needs
editing (renames, deprecations, splits). The command also makes the workflow for future version
upgrades self-documenting: scaffold → diff against real v19 changelog → curate → run
convert-threat-mappings.

Considered Alternatives

No response

Additional Context

No response