Commit 525c79d
pipelines: backfill ingest_mode and auth_type on transform_ocsf/ entries
Adds the new metadata fields introduced by #59 to all 129 existing
transform_ocsf/ pipeline metadata.yaml files. The fields are inserted
immediately after the existing ingestion_method line in each file. No
serializer logic, no pipeline JSON, no other metadata changed.
Values were derived per entry by combining:
1. Bound parser metadata (parsers/community/<source_name>/metadata.yaml)
when the parser declares format=syslog/CEF/RFC/w3c/custom-syslog or
ingestion_method containing "Syslog" or "HEC" -- the parser is
authoritative when its declaration is unambiguous.
2. Vendor and product knowledge for the ~90 entries where the parser
metadata is unclear (gron format with "streaming" or "unknown"
ingestion_method, or no parser binding at all). Examples:
- Cisco network kit (firewalls, ASA, Meraki, ISE, etc.) -> Syslog
- Microsoft 365 / Entra / Defender management surfaces -> API Call (OAuth)
- AWS managed services delivering to S3 (CloudTrail, ELB, Route53
Resolver, GuardDuty export, VPC flow) -> Other - {object store with
SQS notifications} (IAM Role)
- Azure Event Hub-delivered streams (signin, defender email) ->
Other - {Azure Event Hub stream (AMQP/Kafka protocol)} (OAuth)
- SaaS REST APIs (Okta, Snyk, Wiz, Tenable, Mimecast, Netskope,
Proofpoint, GitHub, Google Workspace, Cloudflare, etc.) -> API Call
with the vendor's typical auth (Bearer Token, API Key & Secret,
or OAuth)
Confidence per entry is recorded in
.reorg-prep/inventory/transform_ocsf_classifications.tsv as one of
high (103), medium (17), or low (9). Low-confidence entries are
genuinely generic placeholders (json_generic_logs, sample_test_logs,
microservice_tracing_logs, etc.) where a more specific value is not
derivable; they use Other - {Explain: ...} with the reason inline.
palo_alto_networks_firewall/ is intentionally not modified because it is
being removed in PR #60 (open).
Resulting distribution:
Syslog 56
API Call 39
Other - {object store / Event Hub / agent / etc.} 34
Auth distribution:
N/A (syslog / file-based / generic) 75
API Key & Secret 20
OAuth 18
IAM Role 8
Bearer Token 7
Other (Kafka SASL) 1
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 79a947d commit 525c79d
129 files changed
Lines changed: 258 additions & 0 deletions
File tree
- pipelines/community/transform_ocsf
- agent_metrics_logs
- akamai_cdn
- akamai_dns
- akamai_general
- akamai_sitedefender
- apache_http_logs
- aws_cloudtrail
- aws_elasticloadbalancer_logs
- aws_guardduty_logs
- aws_guardduty
- aws_vpc_dns_logs
- aws_vpc_flow
- aws_waf
- axonius_asset_logs
- azure_ad
- azure_logs
- azure_nsg_flow_logs
- azure_platform
- barracuda_firewall_logs_latest
- beyondtrust_passwordsafe_logs
- cisco_asa_logs
- cisco_combo_logs
- cisco_duo
- cisco_firewall
- cisco_fmc_logs
- cisco_ios_logs
- cisco_ironport
- cisco_isa3000_logs
- cisco_ise_logs
- cisco_logs
- cisco_meraki_flow_logs
- cisco_meraki_logs
- cisco_meraki
- cisco_networks_logs
- cisco_umbrella_logs
- citrix_netscaler_logs
- cloudflare_general_logs
- cloudflare_inc_waf_lastest
- cloudflare_logs
- cloudflare_waf_logs
- confluent_kafka_logs
- crowdstrike_detections
- crowdstrike_endpoint
- crowdstrike_logs
- cyberark_conjur
- darktrace_darktrace_logs
- darktrace
- dhcp_logs
- dns_general_logs
- dns_ocsf_logs
- f5_networks_logs
- forcepoint_forcepoint_logs
- fortigate_logs
- fortimanager_logs
- fortinet_fortigate_candidate_logs
- fortinet_fortigate
- fortinet_logs
- gcp_audit_logs
- gcp_vpc_flow
- generic_access_logs
- github_audit_logs
- google_cloud_dns_logs
- google_workspace_logs
- haproxy_loadbalancer_logs
- hashicorp_hcp_vault_logs
- iis_w3c
- imperva_waf_logs
- incapsula_incapsula_logs
- infoblox_logs
- inngate_gateway_logs
- jruby_application_logs
- json_generic_logs
- json_nested_kv_logs
- juniper_logs
- leef_template_logs
- linux_auth
- log4shell_detection_logs
- m365_audit_logs
- mail_server_logs
- managedengine_ad_audit_plus
- manageengine_adauditplus_logs
- manageengine_general_logs
- meraki_logs
- microservice_tracing_logs
- microsoft_365_mgmt_api_logs
- microsoft_365
- microsoft_activedirectory_logs
- microsoft_defender_for_cloud
- microsoft_entra_logs
- microsoft_eventhub_azure_signin_logs
- microsoft_eventhub_defender_email_logs
- microsoft_eventhub_defender_emailforcloud_logs
- mimecast_mimecast_logs
- netskope
- nginx_error_logs
- nginx_kvlog_logs
- okta_logs
- okta_ocsf_logs
- okta
- paloalto_alternate_logs
- paloalto_logs
- paloalto_vpn_logs
- pfsense_firewall_logs
- proofpoint
- rubrik_backup_logs
- sample_test_logs
- singularityidentity_singularityidentity_logs
- snyk
- sonicwall_firewall_logs
- spam_detection_logs
- sql_database_logs
- squid_proxy_logs
- syslog_space_delimited_logs
- tailscale_tailscale_logs
- teleport_logs
- tenable_vulnerability_management_audit_logging
- ufw_firewall_logs
- vcenter_logs
- vectra_ai_logs
- vmware_vcenter_logs
- vpc_logs
- watchguard_firewall_logs
- windows_event_log_logs
- wiz_cloud_security_logs
- wiz_issue
- zscaler_dns_firewall
- zscaler_firewall_logs
- zscaler_logs
- zscaler_zia_logs
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
| 17 | + | |
16 | 18 | | |
17 | 19 | | |
18 | 20 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
17 | 19 | | |
18 | 20 | | |
19 | 21 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
17 | 19 | | |
18 | 20 | | |
19 | 21 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
17 | 19 | | |
18 | 20 | | |
19 | 21 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
17 | 19 | | |
18 | 20 | | |
19 | 21 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
17 | 19 | | |
18 | 20 | | |
19 | 21 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
17 | 19 | | |
18 | 20 | | |
19 | 21 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
17 | 19 | | |
18 | 20 | | |
19 | 21 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
17 | 19 | | |
18 | 20 | | |
19 | 21 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
17 | 19 | | |
18 | 20 | | |
19 | 21 | | |
| |||
0 commit comments