Zeek OCSF: add MAC addresses, MITRE ATT&CK, alert category, and severity fixes (DataDog#23864)

* [Zeek] Extend OCSF v1.5 normalization with additional field mappings

- Notice pipeline: map proto to ocsf.evidence.connection_info.protocol_name;
  fix severity filter to use severity.level (not severity.id) per real Zeek logs
- Suricata pipeline: map service to evidence connection_info.protocol_name;
  extract MITRE ATT&CK tactic/technique from alert.metadata into
  finding_info.attacks; map alert.category to finding_info.types;
  add risk_level mapping from signature_severity metadata
- Conn pipeline: map orig_l2_addr/resp_l2_addr to src/dst endpoint MAC addresses

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Add changelog entry for PR DataDog#23864

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Fix grok pattern performance: replace data captures with word, drop changelog

- Replace 4x %{data} capture groups with %{word} for MITRE IDs/names
  (MITRE tactic/technique values are alphanumeric+underscore only)
- Drop trailing %{data} (not needed; grok matches without consuming tail)
- Reduces expensive quantifier count from 6 to 1, under the 3-pattern limit
- Remove changelog entry (log asset changes don't require one)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Refactor OCSF intermediary fields to ocsf.* namespace per style guide

- Replace tmp_md5/sha1/sha256.* with ocsf.file.hash.* (style guide §7.2/8.3)
- Replace tmp_attack_str/tmp_attack.* with ocsf.finding_info.attack_raw/attack.*
- Remove grok-parser integer coercions for algorithm_id; schema-processor
  handles type coercion for ocsf.file.hashes elements per OCSF schema
- Update string-builder names to follow Add <field> convention (§6.1)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Remove attack_raw intermediary: grok alert.metadata directly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix OCSF validator errors for algorithm_id type and MITRE extraction

- Restructure hash building to use unique named objects (ocsf.file.hash_md5,
  ocsf.file.hash_sha1, ocsf.file.hash_sha256) with an intermediate
  schema-processor to coerce algorithm_id from string to integer via
  targetFormat: integer before appending to ocsf.file.hashes array
- Store alert.metadata stringification in alert.metadata_str (outside the
  ocsf namespace) to avoid unknown-attribute validation errors; grok reads
  from this field to extract MITRE tactic/technique into ocsf.finding_info.attack

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix MITRE grok pattern: add trailing %{data} anchor and correct sample

Datadog's grok engine uses m.matches() which requires the pattern to consume
the entire input string. The pattern was missing a trailing %{data} to absorb
remaining content after the last MITRE capture (e.g. ,performance_impact:Low,...).
Also update the sample to match the actual string-builder output format:
comma-joined without brackets or spaces, as TemplateEvaluator produces.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Restore original grok sample format with brackets

The pattern already handles both bracket/space and comma-only formats via the
leading and trailing %{data} captures; the sample documents the original
alert.metadata stringified representation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Use lazy quantifier at start of MITRE grok rule

Replace leading %{data} with .*? to avoid validate-logs Greedy At Start
warning; trailing %{data} is kept to consume remaining content and satisfy
the full-string m.matches() requirement.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Replace grok type cast for direction_id with schema-remapper

Use schema-remapper targetFormat: integer inside an intermediate schema-processor
instead of a grok parser for coercing evidence connection_info.direction_id
from string to integer in both Detection Finding sub-pipelines.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix intermediate schema-processor names to follow naming convention

Rename to bare 'Apply OCSF schema for <class_uid>' per style guide [NAMING-7].

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Update pipeline

* Replace intermediate schema-processors with grok type coercion

Use grok parsers to coerce string values to integer instead of
intermediate schema-processors: direction_id in Detection Finding
evidence and algorithm_id in File Hosting Activity file hashes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Update type casting

* Fix attribute-remapper targetType and support MITRE sub-techniques

- Add targetType: attribute to three attribute-remappers that perform
  integer coercion (direction_id, hash_sha1/sha256 algorithm_id);
  validate-logs requires the field.
- Replace %{word} captures in the Suricata MITRE grok rule with
  regex captures that accept dotted sub-technique IDs (e.g. T1059.001),
  so tactic/technique fields populate for sub-techniques as well.
- Add a sub-technique sample to exercise the new pattern.

* Map MITRE sub-technique to ocsf.finding_info.attacks[].sub_technique

Split dotted technique IDs (e.g. T1059.001) into a base technique uid
(T1059 → technique.uid) and a sub-technique object (T1059.001 →
sub_technique.uid). sub_technique is a sibling of technique within the
attack object, not nested inside it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Only populate sub_technique for dotted ATT&CK technique IDs

Restructure the MITRE grok into two rules: sub_technique (dotted IDs
like T1059.001) captures mitre_subtechnique_name separately, while
base_technique (plain IDs like T1071) does not. The sub_technique.name
remapper now sources from mitre_subtechnique_name, which only exists
for sub-techniques, so base techniques never get a spurious
sub_technique object. Adds an end-to-end test for T1059.001.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Remove sub-technique handling from Suricata MITRE extraction

Suricata metadata only emits base ATT&CK technique IDs, so the
sub_technique split logic was unnecessary. Revert to a single grok
rule mapping technique.uid and technique.name directly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Address PR review comments on Zeek OCSF pipeline

- Replace Suricata direction_id grok-parser with attribute-remapper to match Notice sub-pipeline pattern
- Map MITRE grok captures directly to ocsf.finding_info.attack.* fields, removing 4 intermediate attribute-remappers
- Use ocsf.metadata.event_code as temp field for alert.metadata stringification (overwritten by schema-processor), eliminating alert.metadata_str
- Add severity.id OR conditions to Notice severity_id category mapper

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix type-cast remapper names and convert md5 grok coercion to attribute-remapper

Rename all type-cast self-map processors to follow Map `source` to `target`
style guide convention. Convert remaining grok-parser type coercion for
hash_md5.algorithm_id to attribute-remapper with targetFormat: integer.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: cepolation-datadog

zeek/assets/logs/zeek.yaml

@@ -1714,6 +1714,33 @@ pipeline:
           grok:
             supportRules: ""
             matchRules: "to_bool %{boolean(\"true\",\"false\"):ocsf.is_alert}"
+        - type: attribute-remapper
+          name: Map `proto` to `ocsf.evidence.connection_info.protocol_name`
+          enabled: true
+          sources:
+            - proto
+          sourceType: attribute
+          target: ocsf.evidence.connection_info.protocol_name
+          targetType: attribute
+          preserveSource: true
+          overrideOnConflict: false
+        - type: string-builder-processor
+          name: Set evidence connection_info.direction_id to 0 (Unknown)
+          enabled: true
+          template: "0"
+          target: ocsf.evidence.connection_info.direction_id
+          replaceMissing: false
+        - type: attribute-remapper
+          name: Map `ocsf.evidence.connection_info.direction_id` to `ocsf.evidence.connection_info.direction_id`
+          enabled: true
+          sources:
+            - ocsf.evidence.connection_info.direction_id
+          sourceType: attribute
+          target: ocsf.evidence.connection_info.direction_id
+          targetType: attribute
+          preserveSource: false
+          overrideOnConflict: false
+          targetFormat: integer
         - type: attribute-remapper
           name: Map `id.orig_h` to `ocsf.evidence.src_endpoint.ip`
           enabled: true
@@ -1836,23 +1863,23 @@ pipeline:
               name: ocsf.severity_id
               categories:
                 - filter:
-                    query: "@severity.id:1"
+                    query: "@severity.level:1 OR @severity.id:1"
                   name: Informational
                   id: 1
                 - filter:
-                    query: "@severity.id:2"
+                    query: "@severity.level:2 OR @severity.id:2"
                   name: Low
                   id: 2
                 - filter:
-                    query: "@severity.id:3"
+                    query: "@severity.level:3 OR @severity.id:3"
                   name: Medium
                   id: 3
                 - filter:
-                    query: "@severity.id:4"
+                    query: "@severity.level:4 OR @severity.id:4"
                   name: High
                   id: 4
                 - filter:
-                    query: "@severity.id:5"
+                    query: "@severity.level:5 OR @severity.id:5"
                   name: Critical
                   id: 5
                 - filter:
@@ -1929,6 +1956,33 @@ pipeline:
           grok:
             supportRules: ""
             matchRules: "to_bool %{boolean(\"true\",\"false\"):ocsf.is_alert}"
+        - type: attribute-remapper
+          name: Map `service` to `ocsf.evidence.connection_info.protocol_name`
+          enabled: true
+          sources:
+            - service
+          sourceType: attribute
+          target: ocsf.evidence.connection_info.protocol_name
+          targetType: attribute
+          preserveSource: true
+          overrideOnConflict: false
+        - type: string-builder-processor
+          name: Set evidence connection_info.direction_id to 0 (Unknown)
+          enabled: true
+          template: "0"
+          target: ocsf.evidence.connection_info.direction_id
+          replaceMissing: false
+        - type: attribute-remapper
+          name: Map `ocsf.evidence.connection_info.direction_id` to `ocsf.evidence.connection_info.direction_id`
+          enabled: true
+          sources:
+            - ocsf.evidence.connection_info.direction_id
+          sourceType: attribute
+          target: ocsf.evidence.connection_info.direction_id
+          targetType: attribute
+          preserveSource: false
+          overrideOnConflict: false
+          targetFormat: integer
         - type: attribute-remapper
           name: Map `id.orig_h` to `ocsf.evidence.src_endpoint.ip`
           enabled: true
@@ -1977,6 +2031,37 @@ pipeline:
             target: ocsf.evidences
             preserveSource: false
             type: append
+        - type: array-processor
+          name: Append alert.category to ocsf.finding_info.types
+          enabled: true
+          operation:
+            source: alert.category
+            target: ocsf.finding_info.types
+            preserveSource: true
+            type: append
+        - type: string-builder-processor
+          name: Add ocsf.metadata.event_code from alert.metadata
+          enabled: true
+          template: "%{alert.metadata}"
+          target: ocsf.metadata.event_code
+          replaceMissing: false
+        - type: grok-parser
+          name: Parse ocsf.metadata.event_code to MITRE tactic and technique fields
+          enabled: true
+          source: ocsf.metadata.event_code
+          samples:
+            - "[attack_target:Client_Endpoint, confidence:High, created_at:2026_04_01, deployment:Internal, deployment:Perimeter, mitre_tactic_id:TA0011, mitre_tactic_name:Command_And_Control, mitre_technique_id:T1071, mitre_technique_name:Application_Layer_Protocol, performance_impact:Low, signature_severity:Major, updated_at:2026_04_09]"
+          grok:
+            supportRules: ""
+            matchRules: 'rule .*?mitre_tactic_id:%{regex("[^,\\]]+"):ocsf.finding_info.attack.tactic.uid},\s*mitre_tactic_name:%{regex("[^,\\]]+"):ocsf.finding_info.attack.tactic.name},\s*mitre_technique_id:%{regex("[^,\\]]+"):ocsf.finding_info.attack.technique.uid},\s*mitre_technique_name:%{regex("[^,\\]]+"):ocsf.finding_info.attack.technique.name}%{data}'
+        - type: array-processor
+          name: Move ocsf.finding_info.attack into ocsf.finding_info.attacks array
+          enabled: true
+          operation:
+            source: ocsf.finding_info.attack
+            target: ocsf.finding_info.attacks
+            preserveSource: false
+            type: append
         - type: schema-processor
           name: Apply OCSF schema for 2004
           enabled: true
@@ -2009,6 +2094,42 @@ pipeline:
               targets:
                 name: ocsf.confidence
                 id: ocsf.confidence_id
+            - type: schema-category-mapper
+              name: ocsf.risk_level_id
+              categories:
+                - filter:
+                    query: "@alert.metadata:\"signature_severity:Informational\""
+                  name: Info
+                  id: 0
+                - filter:
+                    query: "@alert.metadata:\"signature_severity:Minor\""
+                  name: Low
+                  id: 1
+                - filter:
+                    query: "@alert.metadata:\"signature_severity:Major\""
+                  name: High
+                  id: 3
+                - filter:
+                    query: "@alert.metadata:\"signature_severity:Critical\""
+                  name: Critical
+                  id: 4
+              targets:
+                name: ocsf.risk_level
+                id: ocsf.risk_level_id
+            - type: schema-remapper
+              name: Map `ocsf.finding_info.attacks` to `ocsf.finding_info.attacks`
+              sources:
+                - ocsf.finding_info.attacks
+              target: ocsf.finding_info.attacks
+              preserveSource: true
+              overrideOnConflict: true
+            - type: schema-remapper
+              name: Map `ocsf.finding_info.types` to `ocsf.finding_info.types`
+              sources:
+                - ocsf.finding_info.types
+              target: ocsf.finding_info.types
+              preserveSource: true
+              overrideOnConflict: true
             - type: schema-remapper
               name: Map `ocsf.evidences` to `ocsf.evidences`
               sources:
@@ -2348,6 +2469,13 @@ pipeline:
               target: ocsf.src_endpoint.ip
               preserveSource: true
               overrideOnConflict: true
+            - type: schema-remapper
+              name: Map `orig_l2_addr` to `ocsf.src_endpoint.mac`
+              sources:
+                - orig_l2_addr
+              target: ocsf.src_endpoint.mac
+              preserveSource: true
+              overrideOnConflict: true
             - type: schema-remapper
               name: Map `id.orig_p` to `ocsf.src_endpoint.port`
               sources:
@@ -2356,6 +2484,13 @@ pipeline:
               preserveSource: true
               overrideOnConflict: true
               targetFormat: integer
+            - type: schema-remapper
+              name: Map `resp_l2_addr` to `ocsf.dst_endpoint.mac`
+              sources:
+                - resp_l2_addr
+              target: ocsf.dst_endpoint.mac
+              preserveSource: true
+              overrideOnConflict: true
             - type: schema-remapper
               name: Map `conn_state` to `ocsf.status_detail`
               sources:
@@ -3450,119 +3585,125 @@ pipeline:
             supportRules: ""
             matchRules: 'g %{ip:ocsf.dst_endpoint.ip}(,%{data})?'
         - type: string-builder-processor
-          name: Set MD5 algorithm name
+          name: Add MD5 algorithm name
           enabled: true
           template: MD5
-          target: tmp_md5.algorithm
+          target: ocsf.file.hash_md5.algorithm
           replaceMissing: false
         - type: string-builder-processor
-          name: Set MD5 algorithm id
+          name: Add MD5 algorithm id
           enabled: true
           template: "1"
-          target: tmp_md5.algorithm_id
+          target: ocsf.file.hash_md5.algorithm_id
           replaceMissing: false
-        - type: grok-parser
-          name: Coerce tmp_md5.algorithm_id to integer
-          enabled: true
-          source: tmp_md5.algorithm_id
-          samples:
-            - "1"
-          grok:
-            supportRules: ""
-            matchRules: "to_int %{integer:tmp_md5.algorithm_id}"
         - type: attribute-remapper
-          name: Map `md5` to `tmp_md5.value`
+          name: Map `md5` to `ocsf.file.hash_md5.value`
           enabled: true
           sources:
             - md5
           sourceType: attribute
-          target: tmp_md5.value
+          target: ocsf.file.hash_md5.value
           targetType: attribute
           preserveSource: true
           overrideOnConflict: false
-        - type: array-processor
-          name: Append tmp_md5 to ocsf.file.hashes
-          enabled: true
-          operation:
-            source: tmp_md5
-            target: ocsf.file.hashes
-            preserveSource: false
-            type: append
         - type: string-builder-processor
-          name: Set SHA1 algorithm name
+          name: Add SHA1 algorithm name
           enabled: true
           template: SHA-1
-          target: tmp_sha1.algorithm
+          target: ocsf.file.hash_sha1.algorithm
           replaceMissing: false
         - type: string-builder-processor
-          name: Set SHA1 algorithm id
+          name: Add SHA1 algorithm id
           enabled: true
           template: "2"
-          target: tmp_sha1.algorithm_id
+          target: ocsf.file.hash_sha1.algorithm_id
           replaceMissing: false
-        - type: grok-parser
-          name: Coerce tmp_sha1.algorithm_id to integer
-          enabled: true
-          source: tmp_sha1.algorithm_id
-          samples:
-            - "2"
-          grok:
-            supportRules: ""
-            matchRules: "to_int %{integer:tmp_sha1.algorithm_id}"
         - type: attribute-remapper
-          name: Map `sha1` to `tmp_sha1.value`
+          name: Map `sha1` to `ocsf.file.hash_sha1.value`
           enabled: true
           sources:
             - sha1
           sourceType: attribute
-          target: tmp_sha1.value
+          target: ocsf.file.hash_sha1.value
           targetType: attribute
           preserveSource: true
           overrideOnConflict: false
-        - type: array-processor
-          name: Append tmp_sha1 to ocsf.file.hashes
-          enabled: true
-          operation:
-            source: tmp_sha1
-            target: ocsf.file.hashes
-            preserveSource: false
-            type: append
         - type: string-builder-processor
-          name: Set SHA256 algorithm name
+          name: Add SHA256 algorithm name
           enabled: true
           template: SHA-256
-          target: tmp_sha256.algorithm
+          target: ocsf.file.hash_sha256.algorithm
           replaceMissing: false
         - type: string-builder-processor
-          name: Set SHA256 algorithm id
+          name: Add SHA256 algorithm id
           enabled: true
           template: "3"
-          target: tmp_sha256.algorithm_id
+          target: ocsf.file.hash_sha256.algorithm_id
           replaceMissing: false
-        - type: grok-parser
-          name: Coerce tmp_sha256.algorithm_id to integer
-          enabled: true
-          source: tmp_sha256.algorithm_id
-          samples:
-            - "3"
-          grok:
-            supportRules: ""
-            matchRules: "to_int %{integer:tmp_sha256.algorithm_id}"
         - type: attribute-remapper
-          name: Map `sha256` to `tmp_sha256.value`
+          name: Map `sha256` to `ocsf.file.hash_sha256.value`
           enabled: true
           sources:
             - sha256
           sourceType: attribute
-          target: tmp_sha256.value
+          target: ocsf.file.hash_sha256.value
           targetType: attribute
           preserveSource: true
           overrideOnConflict: false
+        - type: attribute-remapper
+          name: Map `ocsf.file.hash_md5.algorithm_id` to `ocsf.file.hash_md5.algorithm_id`
+          enabled: true
+          sources:
+            - ocsf.file.hash_md5.algorithm_id
+          sourceType: attribute
+          target: ocsf.file.hash_md5.algorithm_id
+          targetType: attribute
+          preserveSource: false
+          overrideOnConflict: false
+          targetFormat: integer
+        - type: attribute-remapper
+          name: Map `ocsf.file.hash_sha1.algorithm_id` to `ocsf.file.hash_sha1.algorithm_id`
+          enabled: true
+          sources:
+            - ocsf.file.hash_sha1.algorithm_id
+          sourceType: attribute
+          target: ocsf.file.hash_sha1.algorithm_id
+          targetType: attribute
+          preserveSource: false
+          overrideOnConflict: false
+          targetFormat: integer
+        - type: attribute-remapper
+          name: Map `ocsf.file.hash_sha256.algorithm_id` to `ocsf.file.hash_sha256.algorithm_id`
+          enabled: true
+          sources:
+            - ocsf.file.hash_sha256.algorithm_id
+          sourceType: attribute
+          target: ocsf.file.hash_sha256.algorithm_id
+          targetType: attribute
+          preserveSource: false
+          overrideOnConflict: false
+          targetFormat: integer
+        - type: array-processor
+          name: Append ocsf.file.hash_md5 to ocsf.file.hashes
+          enabled: true
+          operation:
+            source: ocsf.file.hash_md5
+            target: ocsf.file.hashes
+            preserveSource: false
+            type: append
+        - type: array-processor
+          name: Append ocsf.file.hash_sha1 to ocsf.file.hashes
+          enabled: true
+          operation:
+            source: ocsf.file.hash_sha1
+            target: ocsf.file.hashes
+            preserveSource: false
+            type: append
         - type: array-processor
-          name: Append tmp_sha256 to ocsf.file.hashes
+          name: Append ocsf.file.hash_sha256 to ocsf.file.hashes
           enabled: true
           operation:
-            source: tmp_sha256
+            source: ocsf.file.hash_sha256
             target: ocsf.file.hashes
             preserveSource: false
             type: append