CrowdStrike Adding Identity Protection IDP Parsing for FDR#19805
CrowdStrike Adding Identity Protection IDP Parsing for FDR#19805jeff-bb wants to merge 3 commits into
Conversation
ReviewersBuildkite won't run for external contributors automatically; you need to add a comment:
NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details. |
andrewkroh
added
Integration:crowdstrike
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
|
👀 I have started reviewing the PR |
![vera-review-bot[bot]](https://avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
| replacement: "" | ||
| tag: gsub_context_timestamp | ||
| if: "ctx.crowdstrike?.ContextTimeStamp != null && ctx.crowdstrike.ContextTimeStamp.length() > 18" | ||
| - date: |
There was a problem hiding this comment.
🟠 HIGH data_stream/fdr/.../idp_detection_summary.yml:129
Date processors parse ISO 8601 timestamps as UNIX epoch
The ContextTimeStamp, StartTime, EndTime and AccountCreationTimeStamp date processors parse their input with the UNIX_MS / UNIX (numeric epoch) formats, but the values in the committed sample (test-fdr-idp-detection-summary.log) are ISO 8601 strings, e.g. ContextTimeStamp "2026-06-26T13:33:50.000Z", StartTime/EndTime "2026-06-26T13:31:39.000Z". The convert-to-string + gsub(\d{6}$) + UNIX_MS pattern is the one used elsewhere for numeric nanosecond epochs and does not apply here: the gsub never matches (the strings end in 'Z', not 6 digits), and the date processor then tries to read the full ISO string as milliseconds-since-epoch and fails. These date processors have no ignore_failure and no processor-level on_failure, so the failure propagates to the pipeline on_failure block, which sets event.kind=pipeline_error for every IDP detection event and leaves event.created/event.start/event.end unset. The pipeline and its own sample are inconsistent, so the pipeline test cannot pass.
Recommendation:
Parse the IDP timestamps as ISO 8601 instead of epoch (drop the convert/gsub/UNIX pair for these string fields):
- date:
field: crowdstrike.ContextTimeStamp
target_field: event.created
timezone: UTC
formats:
- ISO8601
tag: date_context_timestamp
if: ctx.crowdstrike?.ContextTimeStamp != null
- date:
field: crowdstrike.StartTime
target_field: event.start
timezone: UTC
formats:
- ISO8601
tag: date_event_start_time
if: ctx.crowdstrike?.StartTime != null
- date:
field: crowdstrike.EndTime
target_field: event.end
timezone: UTC
formats:
- ISO8601
tag: date_event_end_time
if: ctx.crowdstrike?.EndTime != nullIf CrowdStrike can emit either epoch or ISO for these fields, list both formats in a single formats list (e.g. - ISO8601, - UNIX_MS, - UNIX) so one processor handles all shapes.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| @@ -0,0 +1 @@ | |||
| {"AgentId":"c8d1292a8a904216aa25ab728f4b45fd","AggregateId":"aggind:c8d1292a8a904216aa25ab728f4b45fd:11111111111-2222-3333333","CompositeId":"44be50f58ccfcfcfcfcfcfcfcffc:ind:44be50f58ccfcfcfcfcfcfcfcffc:11111111111-2222-3333333","Confidence":20,"ContextTimeStamp":"2026-06-26T13:33:50.000Z","CreatedTimestamp":"2026-06-26T13:34:53.009548969Z","DataDomains":["Identity"],"Description":"A user logged in to a machine for the first time","DisplayName":"Unusual login to an endpoint","EndTime":"2026-06-26T13:31:39.000Z","FalconHostLink":"https://falcon.laggar.gcw.crowdstrike.com/identity-protection/detections/44be50f58ccfcfcfcfcfcfcfcffc:ind:44be50f58ccfcfcfcfcfcfcfcffc:11111111111-2222-3333333?_cid=99999999","HostName":["computer.ad.contoso.com"],"MitreAttack":[{"pattern_id":51135,"tactic_id":"TA0001","technique_id":"T1078","tactic":"Initial Access","technique":"Valid Accounts"}],"Name":"AnomalousNewEndpointUsage","Objective":"Gain Access","PatternId":51135,"Product":"idp","Scenario":"machine_learning","Severity":10,"SeverityName":"Informational","SourceAccountDomain":"ad.contoso.com","SourceAccountName":"JohnD","SourceAccountObjectGuid":"4D9D1676-D22D-4DE8-BB63-0C1661DA9236","SourceAccountObjectSid":"S-1-5-21-123456789-1234567890-1234567890-221133","SourceAccountUpn":"JohnD@ad.contoso.com","SourceEndpointAccountObjectGuid":"1415A515-7DE4-4A10-BF0F-CF5F5CD92812","SourceEndpointAccountObjectSid":"S-1-5-21-123456789-1234567890-1234567890-123456","SourceEndpointAddressIp4":"1.1.1.1","SourceEndpointHostName":"computer.ad.contoso.com","SourceEndpointIpAddress":"1.1.1.1","SourceEndpointSensorId":"c8d1292a8a904216aa25ab728f4b45fd","SourceHosts":["computer.ad.contoso.com"],"SourceIps":["1.1.1.1"],"SourceProducts":["Falcon Identity Protection"],"SourceVendors":["CrowdStrike"],"StartTime":"2026-06-26T13:31:39.000Z","TargetEndpointAccountObjectGuid":"6ECA688B-47F3-461C-AD46-73521AB8677D","TargetEndpointAccountObjectSid":"S-1-5-21-123456789-1234567890-1234567890-098765","TargetEndpointHostName":"DC01","TargetServiceAccessIdentifier":"HTTP/proxy.ad.contoso.com","Timestamp":"2026-06-26T13:33:52.274Z","Type":"idp-session-source-user-endpoint-target-info","UserName":"JohnD","UserNames":["JohnD","JohnD@ad.contoso.com"]} | |||
There was a problem hiding this comment.
🟠 HIGH data_stream/fdr/.../test-fdr-idp-detection-summary.log:1
New test input has no matching -expected.json
This PR adds the pipeline test input test-fdr-idp-detection-summary.log but no companion test-fdr-idp-detection-summary.log-expected.json exists in the checkout. elastic-package test pipeline compares each input against its expected document and fails when the expected file is absent, so this addition will break the pipeline test run in CI. It also means the new idp_detection_summary.yml pipeline has no asserted output coverage.
Recommendation:
Generate and commit the expected output, then verify it reflects correct mappings (not pipeline_error from finding #1):
cd packages/crowdstrike
elastic-package test pipeline --generate -d fdrCommit the resulting test-fdr-idp-detection-summary.log-expected.json alongside the input.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| field: event.kind | ||
| value: alert | ||
| - append: | ||
| field: event.category |
There was a problem hiding this comment.
🟡 MEDIUM data_stream/fdr/.../idp_detection_summary.yml:9
event.category 'malware' is wrong for identity-protection detections
The pipeline appends event.category: malware for all IDP (Identity Protection) detection summary events. These events describe identity/authentication anomalies, not malware — the sample is DisplayName "Unusual login to an endpoint" / Name "AnomalousNewEndpointUsage" with DataDomains ["Identity"]. The value appears copied from epp_detection_summary.yml (endpoint malware) but is not semantically appropriate for identity detections, so dashboards and detections filtering on event.category will misclassify these events.
Recommendation:
Use an identity-appropriate ECS category such as authentication (and intrusion_detection where applicable) instead of malware:
- append:
field: event.category
value: authentication
tag: append_authentication_category
- append:
field: event.category
value: intrusion_detection
tag: append_intrusion_detection_category🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| ignore_missing: true | ||
| tag: append_source_ip | ||
| - append: | ||
| field: threat.technique.name |
There was a problem hiding this comment.
🟡 MEDIUM data_stream/fdr/.../idp_detection_summary.yml:77
MITRE ATT&CK enrichment reads fields absent from IDP events
The threat.technique.name/id and threat.tactic.name/id appends read ctx.crowdstrike.Technique, .TechniqueId, .Tactic and .TacticId. IDP detection summary events carry their ATT&CK data in a MitreAttack array instead (the sample has MitreAttack:[{"pattern_id":...,"tactic_id":"TA0001","technique_id":"T1078","tactic":"Initial Access","technique":"Valid Accounts"}] and no flat crowdstrike.Technique* fields). Because the appends are guarded by if ... != null, they silently never fire, so threat.technique.* and threat.tactic.* are never populated for IDP events. The parent default.yml foreach over crowdstrike.MitreAttack does not cover this either — it reads differently-cased keys (_ingest._value.Tactic / TacticID / Technique / TechniqueID), which do not match this event's tactic_id / technique_id / tactic / technique keys.
Recommendation:
Iterate the MitreAttack array and read its actual keys:
- foreach:
field: crowdstrike.MitreAttack
if: ctx.crowdstrike?.MitreAttack instanceof List
tag: foreach_mitreattack_technique_id
processor:
append:
field: threat.technique.id
value: '{{{_ingest._value.technique_id}}}'
allow_duplicates: false
- foreach:
field: crowdstrike.MitreAttack
if: ctx.crowdstrike?.MitreAttack instanceof List
tag: foreach_mitreattack_technique_name
processor:
append:
field: threat.technique.name
value: '{{{_ingest._value.technique}}}'
allow_duplicates: false
- foreach:
field: crowdstrike.MitreAttack
if: ctx.crowdstrike?.MitreAttack instanceof List
tag: foreach_mitreattack_tactic_id
processor:
append:
field: threat.tactic.id
value: '{{{_ingest._value.tactic_id}}}'
allow_duplicates: false
- foreach:
field: crowdstrike.MitreAttack
if: ctx.crowdstrike?.MitreAttack instanceof List
tag: foreach_mitreattack_tactic_name
processor:
append:
field: threat.tactic.name
value: '{{{_ingest._value.tactic}}}'
allow_duplicates: false🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
/test |
|
@jeff-bb To generate the test expectations, you will need to run |
💔 Build Failed
Failed CI StepsHistory |
This comment has been minimized.
This comment has been minimized.
TL;DRThe Remediation
Investigation detailsRoot CauseThis is a pipeline test fixture issue introduced by the PR changes. The PR adds The PR also wires Evidence
VerificationNot run locally; the available Buildkite log only includes the step tail and artifact upload list, so the analysis used the failed job summary, PR file list, changed pipeline content, and repository pipeline-test documentation. What is this? | From workflow: PR Buildkite Detective Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
There was a problem hiding this comment.
The default.yml pipeline is configured to branch on ctx.crowdstrike?.ExternalApiType == 'Event_IdpDetectionSummaryEvent', but this field is missing from the sample. Additionally, the timestamp format in the sample consists of ISO 8601 strings, whereas the pipeline expects the UNIX epoch format. Is this sample from the FDR feed?
6 participants
Proposed commit message
Create a pipeline (similar to Epp) for Idp events. The Falcon pipeline path has this, but the FDR pipline path does not. I used the existing Falcon Idp as the starting point with a few minor changes based on a combination of our real world data and the current version of the "Identity Protection detection summary fields" document from CrowdStrike
I've created a test-fdr-idp-detection-summary.log file with a single sanitized event, but do not know how to test this to produce the expected output file.