From 2a9d8d215431b54b71c09df7ac522ea0e1df5fe3 Mon Sep 17 00:00:00 2001 From: Nate Smalley Date: Sun, 26 Apr 2026 19:49:07 -0700 Subject: [PATCH 1/3] pipelines: drop F-graded palo_alto_networks_firewall transform Removes pipelines/community/transform_ocsf/palo_alto_networks_firewall/. Reasons: - Graded F (analyzer_limit, 0% required_field_coverage_pct). - Uses non-standard class_uid=99602001 (SentinelOne Security Alert Extended), distinct from the rest of the PAN-OS cluster which uses class_uid=4001 (Network Activity). - No matching upstream parser in parsers/community/. Its source_name (palo_alto_networks_firewall) lacks the -latest versioning suffix used by every other PAN-OS entry, indicating it does not bind to a tracked parser. - Marked as imported "from Observo platform UI" in metadata, rather than via the standard contributor import path used by the rest of the PAN-OS entries. The three remaining PAN-OS transforms (paloalto_logs/, paloalto_alternate_logs/, paloalto_vpn_logs/) each bind cleanly to a corresponding parser in parsers/community/ and are unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../palo_alto_networks_firewall/metadata.yaml | 37 - .../palo_alto_networks_firewall.json | 60 -- .../palo_alto_networks_firewall/sample.json | 3 - .../serializer.lua | 741 ------------------ 4 files changed, 841 deletions(-) delete mode 100644 pipelines/community/transform_ocsf/palo_alto_networks_firewall/metadata.yaml delete mode 100644 pipelines/community/transform_ocsf/palo_alto_networks_firewall/palo_alto_networks_firewall.json delete mode 100644 pipelines/community/transform_ocsf/palo_alto_networks_firewall/sample.json delete mode 100644 pipelines/community/transform_ocsf/palo_alto_networks_firewall/serializer.lua diff --git a/pipelines/community/transform_ocsf/palo_alto_networks_firewall/metadata.yaml b/pipelines/community/transform_ocsf/palo_alto_networks_firewall/metadata.yaml deleted file mode 100644 index 43108ef..0000000 --- a/pipelines/community/transform_ocsf/palo_alto_networks_firewall/metadata.yaml +++ /dev/null @@ -1,37 +0,0 @@ -grade: - letter: F - score: 50 - verdict: analyzer_limit - required_field_coverage_pct: 0 - source_field_coverage_pct: 100 - tested_with_realistic_event: true - validated_at: '2026-04-19' -metadata_details: - purpose: OCSF-compliant Lua serializer for Palo Alto Networks Firewall. Maps source events to OCSF SentinelOne - Security Alert (extended) (class_uid=99602001) following the processEvent contract. - datasource_vendor: palo_alto - dataSource: Palo Alto Networks Firewall - format: source-specific JSON/KV/syslog - ocsf_version: 1.3.0 - ingestion_method: Observo OCSFSerializer (Lua-based transform) - sample_record: "{\n \"raw\": \",2026/04/20 03:40:52,598277911506582,TRAFFIC,end,,2026/04/20 03:39:58,10.190.36.125,201.14.75.196,10.190.36.125,201.14.75.196,allow-ssh,corp\\\ - \\melissa.brooks,,ssh,vsys1,internet,external,ethernet1/3,ethernet1/1,FORWARD,,60476,1,6271,22,6271,22,0x0,tcp,allow,7310097,4523132,2786965,8389,2026/04/20\ - \ 03:39:58,43,,,119316,0x0,,,,5033,3355,aged-out,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\"\ - \n}" - dependency_summary: Requires Observo OCSFSerializer template with Lua runtime (lupa). Source events - must match the field layout exercised by the Lua mappings. - performance_impact: Single-event transform, ~? lines. Field-for-field normalization with helper-based - nested access. - ocsf_mapping: - class_uid: 99602001 - class_name: SentinelOne Security Alert (extended) - category_uid: 996 - category_name: SentinelOne Extended - tags: palo_alto, ocsf, lua, serializer, transform - version: v1.0 - author: Community (imported from Observo platform UI) - validation: - harness_grade: F - harness_score: 50 - orion_reviewed: true - orion_verdict: signed_off diff --git a/pipelines/community/transform_ocsf/palo_alto_networks_firewall/palo_alto_networks_firewall.json b/pipelines/community/transform_ocsf/palo_alto_networks_firewall/palo_alto_networks_firewall.json deleted file mode 100644 index e6b9e40..0000000 --- a/pipelines/community/transform_ocsf/palo_alto_networks_firewall/palo_alto_networks_firewall.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "schema_version": "1.0", - "component": "transform", - "template_name": "OCSFSerializer", - "display_name": "Palo Alto Networks Firewall", - "grade": { - "letter": "F", - "score": 50, - "verdict": "analyzer_limit", - "required_field_coverage_pct": 0, - "source_field_coverage_pct": 100, - "tested_with_realistic_event": true, - "validated_at": "2026-04-19" - }, - "ocsf": { - "class_uid": 99602001, - "class_name": "SentinelOne Security Alert (extended)", - "category_uid": 996, - "category_name": "SentinelOne Extended", - "version": "1.3.0" - }, - "description": "OCSF-compliant Lua serializer for Palo Alto Networks Firewall. Maps source events to OCSF SentinelOne Security Alert (extended) class_uid 99602001.", - "vendor": "palo_alto", - "source_name": "palo_alto_networks_firewall", - "version": "v1.0", - "parameters": [ - { - "name": "serializer", - "type": "select", - "value": "palo-alto-networks-firewall-lua", - "description": "Serializer identifier used by Observo runtime" - }, - { - "name": "lua", - "type": "lua_code", - "value": "-- Palo Alto Networks Firewall to OCSF Mapping Script\n-- Maps PANW Firewall threat log events to OCSF S1 Security Alert format\n--\n-- Usage: processEvent(event) -> ocsf_event\n\nlocal FEATURES = {\n FLATTEN_EVENT_TYPE = true,\n}\n\nfunction mappedFields(fieldMappings)\n local mapped = {}\n for _, v in ipairs(fieldMappings) do\n source = v['source']\n mapped[source] = true\n end\n return mapped\nend\n\n-- Helper to check if a table is an array\nlocal function isArray(t)\n if type(t) ~= \"table\" then return false end\n local i = 0\n for _ in pairs(t) do\n i = i + 1\n if t[i] == nil then\n return false\n end\n end\n return true\nend\n\nfunction copyUnmappedFields(event, fieldMappings, result)\n -- copy everything else to unmapped\n flattenEvent = flattenObject(event)\n mapped = mappedFields(fieldMappings)\n for k, v in pairs(flattenEvent) do\n if k ~= \"_ob\" and not mapped[k] and v ~= nil and v ~= \"\" then\n setNestedField(result, \"unmapped.\" .. k, v)\n end\n end\n return result\nend\n\nfunction flattenObject(tbl, prefix, result)\n result = result or {}\n prefix = prefix or \"\"\n for k, v in pairs(tbl) do\n local keyPath = prefix ~= \"\" and (prefix .. \".\" .. tostring(k)) or tostring(k)\n local vtype = type(v)\n if vtype == \"table\" then\n if isArray(v) then\n -- Keep arrays as is\n result[keyPath] = v\n else\n flattenObject(v, keyPath, result)\n end\n elseif vtype == \"userdata\" then\n -- Handle userdata safely\n local ok, s = pcall(tostring, v)\n if not ok then\n result[keyPath] = nil\n end\n if s == \"userdata: (nil)\" then\n result[keyPath] = nil\n end\n if s == \"userdata: 0x0\" then\n result[keyPath] = nil\n end\n else\n result[keyPath] = v\n end\n end\n return result\nend\n\nlocal THREAT_FIELD_ORDERS = {\n root = {\n \"@logid\",\n \"action\",\n \"actionflags\",\n \"app\",\n \"category\",\n \"cloud_hostname\",\n \"config_ver\",\n \"contenttype\",\n \"device_name\",\n \"direction\",\n \"domain\",\n \"dst\",\n \"dstloc\",\n \"dport\",\n \"dstuser\",\n \"filedigest\",\n \"filename\",\n \"filetype\",\n \"flags\",\n \"from\",\n \"http_method\",\n \"http2_connection\",\n \"inbound_if\",\n \"misc\",\n \"natdport\",\n \"natdst\",\n \"natsport\",\n \"natsrc\",\n \"outbound_if\",\n \"parent_session_id\",\n \"parent_start_time\",\n \"pcap_id\",\n \"proto\",\n \"receive_time\",\n \"repeatcnt\",\n \"reportid\",\n \"rule\",\n \"rule_uuid\",\n \"seqno\",\n \"serial\",\n \"sessionid\",\n \"severity\",\n \"sport\",\n \"src\",\n \"srcloc\",\n \"srcuser\",\n \"subtype\",\n \"thr_category\",\n \"threat_name\",\n \"threatid\",\n \"tid\",\n \"time_generated\",\n \"time_received\",\n \"to\",\n \"tunnel\",\n \"tunnelid\",\n \"tunneltype\",\n \"type\",\n \"url_idx\",\n \"user_agent\",\n \"vsys\",\n \"vsys_id\",\n \"vsys_name\",\n \"wildfire\"\n }\n}\n\nARRAY_FIELDS = {\n attack_surface_ids = true,\n observables = true,\n evidences = true,\n resources = true,\n}\n\n-- Optimized JSON encoding function with predefined ordering\nfunction encodeJson(obj, key, field_orders)\n if obj == nil or obj == \"NULL_PLACEHOLDER\" then\n return \"null\"\n elseif type(obj) == \"boolean\" then\n return tostring(obj)\n elseif type(obj) == \"number\" then\n return tostring(obj)\n elseif type(obj) == \"string\" then\n return '\"' .. obj:gsub('\"', '\\\\\"') .. '\"'\n elseif type(obj) == \"table\" then\n local isArray = true\n local maxIndex = 0\n for k, v in pairs(obj) do\n if type(k) ~= \"number\" then\n isArray = false\n break\n end\n maxIndex = math.max(maxIndex, k)\n end\n \n if isArray and maxIndex > 0 then\n local items = {}\n for i = 1, maxIndex do\n -- Use the parent key for predefined ordering if available\n local elementKey = key or tostring(i)\n table.insert(items, obj[i] ~= nil and encodeJson(obj[i], elementKey, field_orders) or \"null\")\n end\n return \"[\" .. table.concat(items, \", \") .. \"]\"\n elseif isArray and ARRAY_FIELDS[key] == true then\n -- case of empty array []\n return \"[]\"\n else\n local items = {}\n local fieldOrder = field_orders[key] or {}\n \n -- Phase 1: Process fields in predefined order\n for _, fieldName in ipairs(fieldOrder) do\n local v = obj[fieldName]\n if v ~= nil then\n table.insert(items, '\"' .. fieldName:gsub('\"', '\\\\\"') .. '\": ' .. encodeJson(v, fieldName, field_orders))\n else \n table.insert(items, '\"' .. fieldName:gsub('\"', '\\\\\"') .. '\": ' .. \"null\")\n end\n end\n \n -- Phase 2: Process remaining fields\n for k, v in pairs(obj) do\n local found = false\n for _, fieldName in ipairs(fieldOrder) do\n if k == fieldName then \n found = true\n break\n end\n end\n if not found then\n local keyStr = type(k) == \"string\" and k or tostring(k)\n table.insert(items, '\"' .. keyStr:gsub('\"', '\\\\\"') .. '\": ' .. encodeJson(v, keyStr, field_orders))\n end\n end\n \n return \"{\" .. table.concat(items, \", \") .. \"}\"\n end\n else\n return '\"' .. tostring(obj) .. '\"'\n end\nend\n\n\nfunction setNestedField(obj, path, value)\n if value == nil or path == nil or path == '' then return end\n\n local keys = {}\n for key in string.gmatch(path, '[^.]+') do\n if key and key ~= '' then\n table.insert(keys, key)\n end\n end\n\n if #keys == 0 then return end\n\n local current = obj\n for i = 1, #keys - 1 do\n local key = keys[i]\n if key then\n local arrayIndex = string.match(key, '(.-)%[(%d+)%]')\n if arrayIndex then\n local baseName = string.match(key, '(.-)%[')\n local index = tonumber(string.match(key, '%[(%d+)%]')) + 1\n if current[baseName] == nil then\n current[baseName] = {}\n end\n if current[baseName][index] == nil then\n current[baseName][index] = {}\n end\n current = current[baseName][index]\n else\n if current[key] == nil then\n current[key] = {}\n end\n current = current[key]\n end\n end\n end\n\n local finalKey = keys[#keys]\n if finalKey then\n local arrayIndex = string.match(finalKey, '(.-)%[(%d+)%]')\n if arrayIndex then\n local baseName = string.match(finalKey, '(.-)%[')\n local index = tonumber(string.match(finalKey, '%[(%d+)%]')) + 1\n if current[baseName] == nil then\n current[baseName] = {}\n end\n current[baseName][index] = value\n else\n current[finalKey] = value\n end\n end\nend\n\nfunction getNestedField(obj, path)\n if obj == nil or path == nil or path == '' then return nil end\n\n local keys = {}\n for key in string.gmatch(path, '[^.]+') do\n if key and key ~= '' then\n table.insert(keys, key)\n end\n end\n\n if #keys == 0 then return nil end\n\n local current = obj\n for _, key in ipairs(keys) do\n if current == nil or key == nil then return nil end\n\n local arrayIndex = string.match(key, '(.-)%[(%d+)%]')\n if arrayIndex then\n local baseName = string.match(key, '(.-)%[')\n local index = tonumber(string.match(key, '%[(%d+)%]')) + 1\n if current[baseName] == nil or current[baseName][index] == nil then\n return nil\n end\n current = current[baseName][index]\n else\n if current[key] == nil then\n return nil\n end\n current = current[key]\n end\n end\n return current\nend\n\nfunction copyField(source, target, sourcePath, targetPath)\n if source == nil or target == nil or sourcePath == nil or targetPath == nil then\n return\n end\n if sourcePath == '' or targetPath == '' then\n return\n end\n local value = getNestedField(source, sourcePath)\n if value ~= nil then\n setNestedField(target, targetPath, value)\n end\nend\n\nfunction getValue(tbl, key, default)\n local value = tbl[key]\n if value == nil then\n return default\n else\n return value\n end\nend\n\nfunction getNestedValue(tbl, key1, key2, default)\n if tbl == nil then return default end\n local nested = tbl[key1]\n if nested == nil then return default end\n local value = nested[key2]\n if value == nil then return default end\n return value\nend\n\nfunction getSeverityId(severity)\n local severityId = 3\n if severity == \"informational\" then\n severityId = 1\n elseif severity == \"low\" then\n severityId = 2\n elseif severity == \"medium\" then\n severityId = 3\n elseif severity == \"high\" then\n severityId = 4\n elseif severity == \"critical\" then\n severityId = 5\n end\n return severityId\nend\n\nfunction getS1ClassificationId(subtype)\n local classificationIdMapping = {\n spyware = 34,\n virus = 37,\n [\"ml-virus\"] = 37,\n [\"wildfire-virus\"] = 37\n }\n if subtype == nil then return 0 end\n local subtypeLower = string.lower(subtype)\n for classification, classificationTypeId in pairs(classificationIdMapping) do\n if string.find(subtypeLower, classification) then\n return classificationTypeId\n end\n end\n return 0\nend\n\nfunction convertTimeToTimestamp(timeStr)\n if timeStr == nil or timeStr == \"\" then return 0 end\n -- Parse time in format \"YYYY/MM/DD HH:MM:SS\"\n local year, month, day, hour, min, sec = string.match(timeStr, \"(%d+)/(%d+)/(%d+) (%d+):(%d+):(%d+)\")\n if year == nil then return 0 end\n local t = {\n year = tonumber(year),\n month = tonumber(month),\n day = tonumber(day),\n hour = tonumber(hour),\n min = tonumber(min),\n sec = tonumber(sec)\n }\n local ts = os.time(t)\n local utc_ts = os.time(os.date(\"!*t\", ts))\n local offset = os.difftime(ts, utc_ts)\n return (ts + offset) * 1000\nend\n\nfunction getResources(event)\n local name = getValue(event, \"srcuser\", nil)\n if name == nil or name == \"\" then\n name = getValue(event, \"device_name\", nil)\n end\n local serial = getValue(event, \"serial\", \"\")\n local srcuser = getValue(event, \"srcuser\", \"\")\n local uid = serial\n if srcuser ~= nil and srcuser ~= \"\" then\n uid = serial .. \"-\" .. srcuser\n end\n local resources = {\n {name = name, uid = uid}\n }\n return resources\nend\n\nfunction getEvidences(event)\n local srcCountry = getNestedValue(event, \"srcloc\", \"#text\", \"\")\n local dstCountry = getNestedValue(event, \"dstloc\", \"#text\", \"\")\n local evidences = {\n {\n src_endpoint = {\n ip = getValue(event, \"src\", \"\"),\n port = tonumber(getValue(event, \"sport\", \"0\")) or 0,\n location = {\n country = srcCountry\n }\n },\n dst_endpoint = {\n ip = getValue(event, \"dst\", \"\"),\n port = tonumber(getValue(event, \"dport\", \"0\")) or 0,\n location = {\n country = dstCountry\n }\n },\n connection_info = {\n protocol_name = getValue(event, \"proto\", \"\"),\n direction_id = 0\n },\n process = {\n session = {\n uid = getValue(event, \"sessionid\", \"\")\n },\n parent_process = {\n session = {\n uid = getValue(event, \"parent_session_id\", \"\")\n }\n },\n tid = tonumber(getValue(event, \"tid\", \"0\")) or 0\n }\n }\n }\n return evidences\nend\n\nfunction getObservables(event)\n local observables = {\n {\n type = \"IP Address\",\n type_id = 2,\n name = \"src\",\n value = getValue(event, \"src\", \"\")\n },\n {\n type = \"IP Address\",\n type_id = 2,\n name = \"dst\",\n value = getValue(event, \"dst\", \"\")\n }\n }\n return observables\nend\n\nfunction getRelatedEvents(event)\n local receiveTime = getValue(event, \"receive_time\", \"\")\n local timestamp = convertTimeToTimestamp(receiveTime)\n local threatid = getValue(event, \"threatid\", \"\")\n local severity = getValue(event, \"severity\", \"\")\n local severityId = getSeverityId(severity)\n \n local commonFields = {\n time = timestamp,\n uid = threatid,\n severity = severity,\n severity_id = severityId\n }\n \n local relatedEvents = {\n {\n time = commonFields.time,\n uid = commonFields.uid,\n severity = commonFields.severity,\n severity_id = commonFields.severity_id,\n type = \"Source IP Address: \" .. getValue(event, \"src\", \"\")\n },\n {\n time = commonFields.time,\n uid = commonFields.uid,\n severity = commonFields.severity,\n severity_id = commonFields.severity_id,\n type = \"Destination IP Address: \" .. getValue(event, \"dst\", \"\")\n },\n {\n time = commonFields.time,\n uid = commonFields.uid,\n severity = commonFields.severity,\n severity_id = commonFields.severity_id,\n type = \"Rule: \" .. getValue(event, \"rule\", \"\")\n },\n {\n time = commonFields.time,\n uid = commonFields.uid,\n severity = commonFields.severity,\n severity_id = commonFields.severity_id,\n type = \"Rule UUID: \" .. getValue(event, \"rule_uuid\", \"\")\n },\n {\n time = commonFields.time,\n uid = commonFields.uid,\n severity = commonFields.severity,\n severity_id = commonFields.severity_id,\n type = \"Inbound Interface: \" .. getValue(event, \"inbound_if\", \"\")\n },\n {\n time = commonFields.time,\n uid = commonFields.uid,\n severity = commonFields.severity,\n severity_id = commonFields.severity_id,\n type = \"Outbound Interface: \" .. getValue(event, \"outbound_if\", \"\")\n }\n }\n return relatedEvents\nend\n\nfunction getThreatEvents(event)\n local result = {}\n local severity = getValue(event, \"severity\", \"\")\n local subtype = getValue(event, \"subtype\", \"\")\n local receiveTime = getValue(event, \"receive_time\", \"\")\n local timeGenerated = getValue(event, \"time_generated\", \"\")\n local timeReceived = getValue(event, \"time_received\", \"\")\n \n -- Set OCSF class fields\n result.class_uid = 99602001\n result.class_name = \"S1 Security Alert\"\n result.category_uid = 2\n result.category_name = \"Findings\"\n result.activity_id = 1\n result.activity_name = \"Create\"\n result.type_uid = 9960200101\n result.type_name = \"S1 Security Alert: Create\"\n \n -- Set severity\n result.severity = severity\n result.severity_id = getSeverityId(severity)\n \n -- Set status and state\n result.status_id = 1\n result.state_id = 1\n \n -- Set classification\n result.s1_classification = subtype\n result.s1_classification_id = getS1ClassificationId(subtype)\n \n -- Set attack surface\n result.attack_surface_ids = {1}\n \n -- Set metadata\n result.metadata = {\n logged_time = convertTimeToTimestamp(timeGenerated),\n original_time = timeReceived,\n event_code = getValue(event, \"@logid\", \"\"),\n log_name = getValue(event, \"type\", \"\"),\n product = {\n name = \"Palo Alto Networks Firewall\",\n vendor_name = \"Palo Alto Networks\"\n },\n version = \"1.1.0-dev\",\n extensions = {\n {uid = \"996\", name = \"s1\", version = \"0.1.0\"}\n }\n }\n \n -- Set dataSource\n result.dataSource = {\n name = \"Palo Alto Networks Firewall\",\n category = \"security\",\n vendor = \"Palo Alto Networks\"\n }\n \n -- Set finding_info\n local threatid = getValue(event, \"threatid\", \"\")\n local seqno = getValue(event, \"seqno\", \"\")\n result.finding_info = {\n uid = threatid .. \"-\" .. seqno .. \"-\" .. tostring(os.time() * 1000000000),\n title = getValue(event, \"threat_name\", \"\"),\n desc = getValue(event, \"characteristic_of_app\", \"\"),\n related_events = getRelatedEvents(event),\n analytic = {\n category = getValue(event, \"thr_category\", \"\"),\n type_id = 99\n }\n }\n \n -- Set risk level\n result.risk_level = getValue(event, \"risk_of_app\", \"\")\n \n -- Set count\n local repeatcnt = getValue(event, \"repeatcnt\", \"0\")\n result.count = tonumber(repeatcnt) or 0\n \n -- Set time\n result.time = convertTimeToTimestamp(receiveTime)\n \n -- Set resources, evidences, observables\n result.resources = getResources(event)\n result.evidences = getEvidences(event)\n result.observables = getObservables(event)\n \n -- Field mappings for unmapped fields\n local fieldMappings = {\n {source='@logid', target='metadata.event_code'},\n {source='subtype', target='s1_classification'},\n {source='type', target='metadata.log_name'},\n {source='characteristic_of_app', target='finding_info.desc'},\n {source='risk_of_app', target='risk_level'},\n {source='severity', target='severity'},\n {source='threat_name', target='finding_info.title'},\n {source='action', target='unmapped.action'},\n {source='actionflags', target='unmapped.actionflags'},\n {source='app', target='unmapped.app'},\n {source='category', target='unmapped.category'},\n {source='cloud_hostname', target='unmapped.cloud_hostname'},\n {source='config_ver', target='unmapped.config_ver'},\n {source='contenttype', target='unmapped.contenttype'},\n {source='device_name', target='unmapped.device_name'},\n {source='direction', target='unmapped.direction'},\n {source='domain', target='unmapped.domain'},\n {source='dst', target='unmapped.dst'},\n {source='dstloc', target='unmapped.dstloc'},\n {source='dport', target='unmapped.dport'},\n {source='dstuser', target='unmapped.dstuser'},\n {source='filedigest', target='unmapped.filedigest'},\n {source='filename', target='unmapped.filename'},\n {source='filetype', target='unmapped.filetype'},\n {source='flags', target='unmapped.flags'},\n {source='from', target='unmapped.from'},\n {source='http_method', target='unmapped.http_method'},\n {source='http2_connection', target='unmapped.http2_connection'},\n {source='inbound_if', target='unmapped.inbound_if'},\n {source='misc', target='unmapped.misc'},\n {source='natdport', target='unmapped.natdport'},\n {source='natdst', target='unmapped.natdst'},\n {source='natsport', target='unmapped.natsport'},\n {source='natsrc', target='unmapped.natsrc'},\n {source='outbound_if', target='unmapped.outbound_if'},\n {source='parent_session_id', target='unmapped.parent_session_id'},\n {source='parent_start_time', target='unmapped.parent_start_time'},\n {source='pcap_id', target='unmapped.pcap_id'},\n {source='proto', target='unmapped.proto'},\n {source='receive_time', target='unmapped.receive_time'},\n {source='repeatcnt', target='unmapped.repeatcnt'},\n {source='reportid', target='unmapped.reportid'},\n {source='rule', target='unmapped.rule'},\n {source='rule_uuid', target='unmapped.rule_uuid'},\n {source='seqno', target='unmapped.seqno'},\n {source='serial', target='unmapped.serial'},\n {source='sessionid', target='unmapped.sessionid'},\n {source='sport', target='unmapped.sport'},\n {source='src', target='unmapped.src'},\n {source='srcloc', target='unmapped.srcloc'},\n {source='srcuser', target='unmapped.srcuser'},\n {source='thr_category', target='unmapped.thr_category'},\n {source='threatid', target='unmapped.threatid'},\n {source='tid', target='unmapped.tid'},\n {source='time_generated', target='unmapped.time_generated'},\n {source='time_received', target='unmapped.time_received'},\n {source='to', target='unmapped.to'},\n {source='tunnel', target='unmapped.tunnel'},\n {source='tunnelid', target='unmapped.tunnelid'},\n {source='tunneltype', target='unmapped.tunneltype'},\n {source='url_idx', target='unmapped.url_idx'},\n {source='user_agent', target='unmapped.user_agent'},\n {source='vsys', target='unmapped.vsys'},\n {source='vsys_id', target='unmapped.vsys_id'},\n {source='vsys_name', target='unmapped.vsys_name'},\n {source='wildfire', target='unmapped.wildfire'},\n {source='metadata.product.name', target='metadata.product.name'},\n {source='metadata.product.vendor_name', target='metadata.product.vendor_name'},\n {source='metadata.version', target='metadata.version'},\n {source='metadata.extensions', target='metadata.extensions'},\n {source='dataSource.category', target='dataSource.category'},\n {source='dataSource.name', target='dataSource.name'},\n {source='dataSource.vendor', target='dataSource.vendor'},\n {source='class_uid', target='class_uid'},\n {source='class_name', target='class_name'},\n {source='category_uid', target='category_uid'},\n {source='category_name', target='category_name'},\n {source='type_uid', target='type_uid'},\n {source='type_name', target='type_name'},\n {source='activity_name', target='activity_name'},\n {source='activity_id', target='activity_id'},\n {source='severity_id', target='severity_id'},\n {source='status_id', target='status_id'},\n {source='state_id', target='state_id'},\n {source='s1_classification_id', target='s1_classification_id'},\n {source='attack_surface_ids', target='attack_surface_ids'},\n {source='resources', target='resources'},\n {source='evidences', target='evidences'},\n {source='observables', target='observables'},\n {source='finding_info', target='finding_info'},\n {source='time', target='time'},\n {source='count', target='count'},\n }\n\n result = copyUnmappedFields(event, fieldMappings, result)\n return result\nend\n\nfunction processSecurityFinding(event)\n local result = {}\n local field_order = THREAT_FIELD_ORDERS\n \n -- Process as threat event (PANW Firewall threat logs)\n result = getThreatEvents(event)\n \n -- preserve the original event in the message field\n local cleanEvent = {}\n for key, value in pairs(event) do\n if key ~= \"_ob\" then\n cleanEvent[key] = value\n end\n end\n result.message = encodeJson(cleanEvent, \"root\", field_order)\n\n if FEATURES.FLATTEN_EVENT_TYPE then\n if result and result.event then\n result['event.type'] = result.event.type\n end\n end\n return result\nend\n\n-- Main event processing function\nfunction processEvent(event)\n if event == nil then\n return {}\n end\n return processSecurityFinding(event)\nend\n\n", - "description": "Lua processEvent serializer \u2014 maps source fields to OCSF schema" - } - ], - "validation": { - "harness_grade": "F", - "harness_score": 50, - "harness_lint_score": 0.0, - "harness_required_coverage": 0, - "harness_field_coverage": 100, - "lua_validity": "pass", - "execution_passed": true, - "tested_with_realistic_event": true, - "orion_reviewed": true, - "orion_verdict": "analyzer_limit", - "validated_at": "2026-04-19", - "class_uid_concern": true, - "alternative_class_uid": 2004, - "concern_note": "Orion: 99602001 is non-standard SentinelOne extended UID; standard OCSF mapping is 2004 Detection Finding. Kept because this is SentinelOne-extended OCSF by design." - }, - "provenance": { - "tier": "ui", - "source": "Observo.ai Pipeline Manager UI (production template)" - } -} \ No newline at end of file diff --git a/pipelines/community/transform_ocsf/palo_alto_networks_firewall/sample.json b/pipelines/community/transform_ocsf/palo_alto_networks_firewall/sample.json deleted file mode 100644 index 5d18ba9..0000000 --- a/pipelines/community/transform_ocsf/palo_alto_networks_firewall/sample.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "raw": ",2026/04/20 03:40:52,598277911506582,TRAFFIC,end,,2026/04/20 03:39:58,10.190.36.125,201.14.75.196,10.190.36.125,201.14.75.196,allow-ssh,corp\\melissa.brooks,,ssh,vsys1,internet,external,ethernet1/3,ethernet1/1,FORWARD,,60476,1,6271,22,6271,22,0x0,tcp,allow,7310097,4523132,2786965,8389,2026/04/20 03:39:58,43,,,119316,0x0,,,,5033,3355,aged-out,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,," -} \ No newline at end of file diff --git a/pipelines/community/transform_ocsf/palo_alto_networks_firewall/serializer.lua b/pipelines/community/transform_ocsf/palo_alto_networks_firewall/serializer.lua deleted file mode 100644 index 143d4d7..0000000 --- a/pipelines/community/transform_ocsf/palo_alto_networks_firewall/serializer.lua +++ /dev/null @@ -1,741 +0,0 @@ --- Palo Alto Networks Firewall to OCSF Mapping Script --- Maps PANW Firewall threat log events to OCSF S1 Security Alert format --- --- Usage: processEvent(event) -> ocsf_event - -local FEATURES = { - FLATTEN_EVENT_TYPE = true, -} - -function mappedFields(fieldMappings) - local mapped = {} - for _, v in ipairs(fieldMappings) do - source = v['source'] - mapped[source] = true - end - return mapped -end - --- Helper to check if a table is an array -local function isArray(t) - if type(t) ~= "table" then return false end - local i = 0 - for _ in pairs(t) do - i = i + 1 - if t[i] == nil then - return false - end - end - return true -end - -function copyUnmappedFields(event, fieldMappings, result) - -- copy everything else to unmapped - flattenEvent = flattenObject(event) - mapped = mappedFields(fieldMappings) - for k, v in pairs(flattenEvent) do - if k ~= "_ob" and not mapped[k] and v ~= nil and v ~= "" then - setNestedField(result, "unmapped." .. k, v) - end - end - return result -end - -function flattenObject(tbl, prefix, result) - result = result or {} - prefix = prefix or "" - for k, v in pairs(tbl) do - local keyPath = prefix ~= "" and (prefix .. "." .. tostring(k)) or tostring(k) - local vtype = type(v) - if vtype == "table" then - if isArray(v) then - -- Keep arrays as is - result[keyPath] = v - else - flattenObject(v, keyPath, result) - end - elseif vtype == "userdata" then - -- Handle userdata safely - local ok, s = pcall(tostring, v) - if not ok then - result[keyPath] = nil - end - if s == "userdata: (nil)" then - result[keyPath] = nil - end - if s == "userdata: 0x0" then - result[keyPath] = nil - end - else - result[keyPath] = v - end - end - return result -end - -local THREAT_FIELD_ORDERS = { - root = { - "@logid", - "action", - "actionflags", - "app", - "category", - "cloud_hostname", - "config_ver", - "contenttype", - "device_name", - "direction", - "domain", - "dst", - "dstloc", - "dport", - "dstuser", - "filedigest", - "filename", - "filetype", - "flags", - "from", - "http_method", - "http2_connection", - "inbound_if", - "misc", - "natdport", - "natdst", - "natsport", - "natsrc", - "outbound_if", - "parent_session_id", - "parent_start_time", - "pcap_id", - "proto", - "receive_time", - "repeatcnt", - "reportid", - "rule", - "rule_uuid", - "seqno", - "serial", - "sessionid", - "severity", - "sport", - "src", - "srcloc", - "srcuser", - "subtype", - "thr_category", - "threat_name", - "threatid", - "tid", - "time_generated", - "time_received", - "to", - "tunnel", - "tunnelid", - "tunneltype", - "type", - "url_idx", - "user_agent", - "vsys", - "vsys_id", - "vsys_name", - "wildfire" - } -} - -ARRAY_FIELDS = { - attack_surface_ids = true, - observables = true, - evidences = true, - resources = true, -} - --- Optimized JSON encoding function with predefined ordering -function encodeJson(obj, key, field_orders) - if obj == nil or obj == "NULL_PLACEHOLDER" then - return "null" - elseif type(obj) == "boolean" then - return tostring(obj) - elseif type(obj) == "number" then - return tostring(obj) - elseif type(obj) == "string" then - return '"' .. obj:gsub('"', '\\"') .. '"' - elseif type(obj) == "table" then - local isArray = true - local maxIndex = 0 - for k, v in pairs(obj) do - if type(k) ~= "number" then - isArray = false - break - end - maxIndex = math.max(maxIndex, k) - end - - if isArray and maxIndex > 0 then - local items = {} - for i = 1, maxIndex do - -- Use the parent key for predefined ordering if available - local elementKey = key or tostring(i) - table.insert(items, obj[i] ~= nil and encodeJson(obj[i], elementKey, field_orders) or "null") - end - return "[" .. table.concat(items, ", ") .. "]" - elseif isArray and ARRAY_FIELDS[key] == true then - -- case of empty array [] - return "[]" - else - local items = {} - local fieldOrder = field_orders[key] or {} - - -- Phase 1: Process fields in predefined order - for _, fieldName in ipairs(fieldOrder) do - local v = obj[fieldName] - if v ~= nil then - table.insert(items, '"' .. fieldName:gsub('"', '\\"') .. '": ' .. encodeJson(v, fieldName, field_orders)) - else - table.insert(items, '"' .. fieldName:gsub('"', '\\"') .. '": ' .. "null") - end - end - - -- Phase 2: Process remaining fields - for k, v in pairs(obj) do - local found = false - for _, fieldName in ipairs(fieldOrder) do - if k == fieldName then - found = true - break - end - end - if not found then - local keyStr = type(k) == "string" and k or tostring(k) - table.insert(items, '"' .. keyStr:gsub('"', '\\"') .. '": ' .. encodeJson(v, keyStr, field_orders)) - end - end - - return "{" .. table.concat(items, ", ") .. "}" - end - else - return '"' .. tostring(obj) .. '"' - end -end - - -function setNestedField(obj, path, value) - if value == nil or path == nil or path == '' then return end - - local keys = {} - for key in string.gmatch(path, '[^.]+') do - if key and key ~= '' then - table.insert(keys, key) - end - end - - if #keys == 0 then return end - - local current = obj - for i = 1, #keys - 1 do - local key = keys[i] - if key then - local arrayIndex = string.match(key, '(.-)%[(%d+)%]') - if arrayIndex then - local baseName = string.match(key, '(.-)%[') - local index = tonumber(string.match(key, '%[(%d+)%]')) + 1 - if current[baseName] == nil then - current[baseName] = {} - end - if current[baseName][index] == nil then - current[baseName][index] = {} - end - current = current[baseName][index] - else - if current[key] == nil then - current[key] = {} - end - current = current[key] - end - end - end - - local finalKey = keys[#keys] - if finalKey then - local arrayIndex = string.match(finalKey, '(.-)%[(%d+)%]') - if arrayIndex then - local baseName = string.match(finalKey, '(.-)%[') - local index = tonumber(string.match(finalKey, '%[(%d+)%]')) + 1 - if current[baseName] == nil then - current[baseName] = {} - end - current[baseName][index] = value - else - current[finalKey] = value - end - end -end - -function getNestedField(obj, path) - if obj == nil or path == nil or path == '' then return nil end - - local keys = {} - for key in string.gmatch(path, '[^.]+') do - if key and key ~= '' then - table.insert(keys, key) - end - end - - if #keys == 0 then return nil end - - local current = obj - for _, key in ipairs(keys) do - if current == nil or key == nil then return nil end - - local arrayIndex = string.match(key, '(.-)%[(%d+)%]') - if arrayIndex then - local baseName = string.match(key, '(.-)%[') - local index = tonumber(string.match(key, '%[(%d+)%]')) + 1 - if current[baseName] == nil or current[baseName][index] == nil then - return nil - end - current = current[baseName][index] - else - if current[key] == nil then - return nil - end - current = current[key] - end - end - return current -end - -function copyField(source, target, sourcePath, targetPath) - if source == nil or target == nil or sourcePath == nil or targetPath == nil then - return - end - if sourcePath == '' or targetPath == '' then - return - end - local value = getNestedField(source, sourcePath) - if value ~= nil then - setNestedField(target, targetPath, value) - end -end - -function getValue(tbl, key, default) - local value = tbl[key] - if value == nil then - return default - else - return value - end -end - -function getNestedValue(tbl, key1, key2, default) - if tbl == nil then return default end - local nested = tbl[key1] - if nested == nil then return default end - local value = nested[key2] - if value == nil then return default end - return value -end - -function getSeverityId(severity) - local severityId = 3 - if severity == "informational" then - severityId = 1 - elseif severity == "low" then - severityId = 2 - elseif severity == "medium" then - severityId = 3 - elseif severity == "high" then - severityId = 4 - elseif severity == "critical" then - severityId = 5 - end - return severityId -end - -function getS1ClassificationId(subtype) - local classificationIdMapping = { - spyware = 34, - virus = 37, - ["ml-virus"] = 37, - ["wildfire-virus"] = 37 - } - if subtype == nil then return 0 end - local subtypeLower = string.lower(subtype) - for classification, classificationTypeId in pairs(classificationIdMapping) do - if string.find(subtypeLower, classification) then - return classificationTypeId - end - end - return 0 -end - -function convertTimeToTimestamp(timeStr) - if timeStr == nil or timeStr == "" then return 0 end - -- Parse time in format "YYYY/MM/DD HH:MM:SS" - local year, month, day, hour, min, sec = string.match(timeStr, "(%d+)/(%d+)/(%d+) (%d+):(%d+):(%d+)") - if year == nil then return 0 end - local t = { - year = tonumber(year), - month = tonumber(month), - day = tonumber(day), - hour = tonumber(hour), - min = tonumber(min), - sec = tonumber(sec) - } - local ts = os.time(t) - local utc_ts = os.time(os.date("!*t", ts)) - local offset = os.difftime(ts, utc_ts) - return (ts + offset) * 1000 -end - -function getResources(event) - local name = getValue(event, "srcuser", nil) - if name == nil or name == "" then - name = getValue(event, "device_name", nil) - end - local serial = getValue(event, "serial", "") - local srcuser = getValue(event, "srcuser", "") - local uid = serial - if srcuser ~= nil and srcuser ~= "" then - uid = serial .. "-" .. srcuser - end - local resources = { - {name = name, uid = uid} - } - return resources -end - -function getEvidences(event) - local srcCountry = getNestedValue(event, "srcloc", "#text", "") - local dstCountry = getNestedValue(event, "dstloc", "#text", "") - local evidences = { - { - src_endpoint = { - ip = getValue(event, "src", ""), - port = tonumber(getValue(event, "sport", "0")) or 0, - location = { - country = srcCountry - } - }, - dst_endpoint = { - ip = getValue(event, "dst", ""), - port = tonumber(getValue(event, "dport", "0")) or 0, - location = { - country = dstCountry - } - }, - connection_info = { - protocol_name = getValue(event, "proto", ""), - direction_id = 0 - }, - process = { - session = { - uid = getValue(event, "sessionid", "") - }, - parent_process = { - session = { - uid = getValue(event, "parent_session_id", "") - } - }, - tid = tonumber(getValue(event, "tid", "0")) or 0 - } - } - } - return evidences -end - -function getObservables(event) - local observables = { - { - type = "IP Address", - type_id = 2, - name = "src", - value = getValue(event, "src", "") - }, - { - type = "IP Address", - type_id = 2, - name = "dst", - value = getValue(event, "dst", "") - } - } - return observables -end - -function getRelatedEvents(event) - local receiveTime = getValue(event, "receive_time", "") - local timestamp = convertTimeToTimestamp(receiveTime) - local threatid = getValue(event, "threatid", "") - local severity = getValue(event, "severity", "") - local severityId = getSeverityId(severity) - - local commonFields = { - time = timestamp, - uid = threatid, - severity = severity, - severity_id = severityId - } - - local relatedEvents = { - { - time = commonFields.time, - uid = commonFields.uid, - severity = commonFields.severity, - severity_id = commonFields.severity_id, - type = "Source IP Address: " .. getValue(event, "src", "") - }, - { - time = commonFields.time, - uid = commonFields.uid, - severity = commonFields.severity, - severity_id = commonFields.severity_id, - type = "Destination IP Address: " .. getValue(event, "dst", "") - }, - { - time = commonFields.time, - uid = commonFields.uid, - severity = commonFields.severity, - severity_id = commonFields.severity_id, - type = "Rule: " .. getValue(event, "rule", "") - }, - { - time = commonFields.time, - uid = commonFields.uid, - severity = commonFields.severity, - severity_id = commonFields.severity_id, - type = "Rule UUID: " .. getValue(event, "rule_uuid", "") - }, - { - time = commonFields.time, - uid = commonFields.uid, - severity = commonFields.severity, - severity_id = commonFields.severity_id, - type = "Inbound Interface: " .. getValue(event, "inbound_if", "") - }, - { - time = commonFields.time, - uid = commonFields.uid, - severity = commonFields.severity, - severity_id = commonFields.severity_id, - type = "Outbound Interface: " .. getValue(event, "outbound_if", "") - } - } - return relatedEvents -end - -function getThreatEvents(event) - local result = {} - local severity = getValue(event, "severity", "") - local subtype = getValue(event, "subtype", "") - local receiveTime = getValue(event, "receive_time", "") - local timeGenerated = getValue(event, "time_generated", "") - local timeReceived = getValue(event, "time_received", "") - - -- Set OCSF class fields - result.class_uid = 99602001 - result.class_name = "S1 Security Alert" - result.category_uid = 2 - result.category_name = "Findings" - result.activity_id = 1 - result.activity_name = "Create" - result.type_uid = 9960200101 - result.type_name = "S1 Security Alert: Create" - - -- Set severity - result.severity = severity - result.severity_id = getSeverityId(severity) - - -- Set status and state - result.status_id = 1 - result.state_id = 1 - - -- Set classification - result.s1_classification = subtype - result.s1_classification_id = getS1ClassificationId(subtype) - - -- Set attack surface - result.attack_surface_ids = {1} - - -- Set metadata - result.metadata = { - logged_time = convertTimeToTimestamp(timeGenerated), - original_time = timeReceived, - event_code = getValue(event, "@logid", ""), - log_name = getValue(event, "type", ""), - product = { - name = "Palo Alto Networks Firewall", - vendor_name = "Palo Alto Networks" - }, - version = "1.1.0-dev", - extensions = { - {uid = "996", name = "s1", version = "0.1.0"} - } - } - - -- Set dataSource - result.dataSource = { - name = "Palo Alto Networks Firewall", - category = "security", - vendor = "Palo Alto Networks" - } - - -- Set finding_info - local threatid = getValue(event, "threatid", "") - local seqno = getValue(event, "seqno", "") - result.finding_info = { - uid = threatid .. "-" .. seqno .. "-" .. tostring(os.time() * 1000000000), - title = getValue(event, "threat_name", ""), - desc = getValue(event, "characteristic_of_app", ""), - related_events = getRelatedEvents(event), - analytic = { - category = getValue(event, "thr_category", ""), - type_id = 99 - } - } - - -- Set risk level - result.risk_level = getValue(event, "risk_of_app", "") - - -- Set count - local repeatcnt = getValue(event, "repeatcnt", "0") - result.count = tonumber(repeatcnt) or 0 - - -- Set time - result.time = convertTimeToTimestamp(receiveTime) - - -- Set resources, evidences, observables - result.resources = getResources(event) - result.evidences = getEvidences(event) - result.observables = getObservables(event) - - -- Field mappings for unmapped fields - local fieldMappings = { - {source='@logid', target='metadata.event_code'}, - {source='subtype', target='s1_classification'}, - {source='type', target='metadata.log_name'}, - {source='characteristic_of_app', target='finding_info.desc'}, - {source='risk_of_app', target='risk_level'}, - {source='severity', target='severity'}, - {source='threat_name', target='finding_info.title'}, - {source='action', target='unmapped.action'}, - {source='actionflags', target='unmapped.actionflags'}, - {source='app', target='unmapped.app'}, - {source='category', target='unmapped.category'}, - {source='cloud_hostname', target='unmapped.cloud_hostname'}, - {source='config_ver', target='unmapped.config_ver'}, - {source='contenttype', target='unmapped.contenttype'}, - {source='device_name', target='unmapped.device_name'}, - {source='direction', target='unmapped.direction'}, - {source='domain', target='unmapped.domain'}, - {source='dst', target='unmapped.dst'}, - {source='dstloc', target='unmapped.dstloc'}, - {source='dport', target='unmapped.dport'}, - {source='dstuser', target='unmapped.dstuser'}, - {source='filedigest', target='unmapped.filedigest'}, - {source='filename', target='unmapped.filename'}, - {source='filetype', target='unmapped.filetype'}, - {source='flags', target='unmapped.flags'}, - {source='from', target='unmapped.from'}, - {source='http_method', target='unmapped.http_method'}, - {source='http2_connection', target='unmapped.http2_connection'}, - {source='inbound_if', target='unmapped.inbound_if'}, - {source='misc', target='unmapped.misc'}, - {source='natdport', target='unmapped.natdport'}, - {source='natdst', target='unmapped.natdst'}, - {source='natsport', target='unmapped.natsport'}, - {source='natsrc', target='unmapped.natsrc'}, - {source='outbound_if', target='unmapped.outbound_if'}, - {source='parent_session_id', target='unmapped.parent_session_id'}, - {source='parent_start_time', target='unmapped.parent_start_time'}, - {source='pcap_id', target='unmapped.pcap_id'}, - {source='proto', target='unmapped.proto'}, - {source='receive_time', target='unmapped.receive_time'}, - {source='repeatcnt', target='unmapped.repeatcnt'}, - {source='reportid', target='unmapped.reportid'}, - {source='rule', target='unmapped.rule'}, - {source='rule_uuid', target='unmapped.rule_uuid'}, - {source='seqno', target='unmapped.seqno'}, - {source='serial', target='unmapped.serial'}, - {source='sessionid', target='unmapped.sessionid'}, - {source='sport', target='unmapped.sport'}, - {source='src', target='unmapped.src'}, - {source='srcloc', target='unmapped.srcloc'}, - {source='srcuser', target='unmapped.srcuser'}, - {source='thr_category', target='unmapped.thr_category'}, - {source='threatid', target='unmapped.threatid'}, - {source='tid', target='unmapped.tid'}, - {source='time_generated', target='unmapped.time_generated'}, - {source='time_received', target='unmapped.time_received'}, - {source='to', target='unmapped.to'}, - {source='tunnel', target='unmapped.tunnel'}, - {source='tunnelid', target='unmapped.tunnelid'}, - {source='tunneltype', target='unmapped.tunneltype'}, - {source='url_idx', target='unmapped.url_idx'}, - {source='user_agent', target='unmapped.user_agent'}, - {source='vsys', target='unmapped.vsys'}, - {source='vsys_id', target='unmapped.vsys_id'}, - {source='vsys_name', target='unmapped.vsys_name'}, - {source='wildfire', target='unmapped.wildfire'}, - {source='metadata.product.name', target='metadata.product.name'}, - {source='metadata.product.vendor_name', target='metadata.product.vendor_name'}, - {source='metadata.version', target='metadata.version'}, - {source='metadata.extensions', target='metadata.extensions'}, - {source='dataSource.category', target='dataSource.category'}, - {source='dataSource.name', target='dataSource.name'}, - {source='dataSource.vendor', target='dataSource.vendor'}, - {source='class_uid', target='class_uid'}, - {source='class_name', target='class_name'}, - {source='category_uid', target='category_uid'}, - {source='category_name', target='category_name'}, - {source='type_uid', target='type_uid'}, - {source='type_name', target='type_name'}, - {source='activity_name', target='activity_name'}, - {source='activity_id', target='activity_id'}, - {source='severity_id', target='severity_id'}, - {source='status_id', target='status_id'}, - {source='state_id', target='state_id'}, - {source='s1_classification_id', target='s1_classification_id'}, - {source='attack_surface_ids', target='attack_surface_ids'}, - {source='resources', target='resources'}, - {source='evidences', target='evidences'}, - {source='observables', target='observables'}, - {source='finding_info', target='finding_info'}, - {source='time', target='time'}, - {source='count', target='count'}, - } - - result = copyUnmappedFields(event, fieldMappings, result) - return result -end - -function processSecurityFinding(event) - local result = {} - local field_order = THREAT_FIELD_ORDERS - - -- Process as threat event (PANW Firewall threat logs) - result = getThreatEvents(event) - - -- preserve the original event in the message field - local cleanEvent = {} - for key, value in pairs(event) do - if key ~= "_ob" then - cleanEvent[key] = value - end - end - result.message = encodeJson(cleanEvent, "root", field_order) - - if FEATURES.FLATTEN_EVENT_TYPE then - if result and result.event then - result['event.type'] = result.event.type - end - end - return result -end - --- Main event processing function -function processEvent(event) - if event == nil then - return {} - end - return processSecurityFinding(event) -end - From 10b8a49ed911e50d32744be10d93649e6b5750a7 Mon Sep 17 00:00:00 2001 From: Nate Smalley Date: Sun, 26 Apr 2026 19:49:47 -0700 Subject: [PATCH 2/3] pipelines: document PAN-OS transform variants and upstream-parser binding Each PAN-OS OCSF transform in pipelines/community/transform_ocsf/ binds to a specific parser in parsers/community/ via the source_name field in the pipeline JSON. Until now the metadata.yaml purpose fields described all three variants identically, leaving users no way to choose between them. This commit rewrites each purpose field to declare: - The parser directory it is bound to (parsers/community/-latest/) - The field-name convention it expects from that parser's output - Its activity_id derivation strategy - Cross-references to sibling variants Specifically: - paloalto_logs/: bound to paloalto_logs-latest. Expects the standard PAN-OS field-name convention (sourceip / source_port / type). activity_id is action-derived. - paloalto_alternate_logs/: bound to paloalto_alternate_logs-latest. Expects an alternate field-name convention (srcip / srcport / logtype, plus verdict, threat_severity, threat_name, subtype). activity_id is logType-aware (TRAFFIC -> traffic activity, THREAT -> Malicious Activity 6, URL -> URL Activity 5). Also normalizes numeric protocol identifiers and accepts numeric severity strings. - paloalto_vpn_logs/: bound to paloalto_vpn_logs-latest. PAN-OS GlobalProtect VPN traffic logs. No serializer logic changes; this is documentation only. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../transform_ocsf/paloalto_alternate_logs/metadata.yaml | 9 +++++++-- .../community/transform_ocsf/paloalto_logs/metadata.yaml | 8 ++++++-- .../transform_ocsf/paloalto_vpn_logs/metadata.yaml | 6 ++++-- 3 files changed, 17 insertions(+), 6 deletions(-) diff --git a/pipelines/community/transform_ocsf/paloalto_alternate_logs/metadata.yaml b/pipelines/community/transform_ocsf/paloalto_alternate_logs/metadata.yaml index 960cedd..89b9cfc 100644 --- a/pipelines/community/transform_ocsf/paloalto_alternate_logs/metadata.yaml +++ b/pipelines/community/transform_ocsf/paloalto_alternate_logs/metadata.yaml @@ -7,8 +7,13 @@ grade: tested_with_realistic_event: true validated_at: '2026-04-19' metadata_details: - purpose: OCSF-compliant Lua serializer for Paloalto Alternate Logs. Maps source events to OCSF Network - Activity (class_uid=4001) following the processEvent contract. + purpose: OCSF-compliant Lua serializer bound to the upstream parser at parsers/community/paloalto_alternate_logs-latest/. + Maps that parser's output to OCSF Network Activity (class_uid=4001) for an alternate field-name + convention of PAN-OS firewall logs (uses srcip / srcport / logtype rather than sourceip / + source_port / type, and adds verdict, threat_severity, threat_name, subtype). activity_id is + logType-aware (TRAFFIC+allow=1, TRAFFIC+deny=2, THREAT=6 Malicious Activity, URL=5 URL Activity). + Also normalizes numeric protocol identifiers (6 -> tcp, 17 -> udp, 1 -> icmp) and accepts + numeric severity strings. For the standard upstream parser see paloalto_logs/. datasource_vendor: palo_alto dataSource: Paloalto Alternate Logs format: source-specific JSON/KV/syslog diff --git a/pipelines/community/transform_ocsf/paloalto_logs/metadata.yaml b/pipelines/community/transform_ocsf/paloalto_logs/metadata.yaml index 23ab39b..0944f89 100644 --- a/pipelines/community/transform_ocsf/paloalto_logs/metadata.yaml +++ b/pipelines/community/transform_ocsf/paloalto_logs/metadata.yaml @@ -7,8 +7,12 @@ grade: tested_with_realistic_event: true validated_at: '2026-04-19' metadata_details: - purpose: OCSF-compliant Lua serializer for Paloalto Logs. Maps source events to OCSF Network Activity - (class_uid=4001) following the processEvent contract. + purpose: OCSF-compliant Lua serializer bound to the upstream parser at parsers/community/paloalto_logs-latest/. + Maps that parser's output to OCSF Network Activity (class_uid=4001) for general PAN-OS firewall + logs (TRAFFIC, THREAT subtypes). Expects the standard PAN-OS field-name convention (sourceip / + source_port / type). activity_id is action-derived (allow=1, deny/drop/block=2, reset=3, + else=99). For the alternate field-name convention upstream parser see paloalto_alternate_logs/; + for PAN-OS GlobalProtect VPN traffic see paloalto_vpn_logs/. datasource_vendor: palo_alto dataSource: Paloalto Logs format: source-specific JSON/KV/syslog diff --git a/pipelines/community/transform_ocsf/paloalto_vpn_logs/metadata.yaml b/pipelines/community/transform_ocsf/paloalto_vpn_logs/metadata.yaml index fceedd2..8293264 100644 --- a/pipelines/community/transform_ocsf/paloalto_vpn_logs/metadata.yaml +++ b/pipelines/community/transform_ocsf/paloalto_vpn_logs/metadata.yaml @@ -7,8 +7,10 @@ grade: tested_with_realistic_event: true validated_at: '2026-04-19' metadata_details: - purpose: OCSF-compliant Lua serializer for Paloalto Vpn Logs. Maps source events to OCSF Network Activity - (class_uid=4001) following the processEvent contract. + purpose: OCSF-compliant Lua serializer bound to the upstream parser at parsers/community/paloalto_vpn_logs-latest/. + Maps that parser's output to OCSF Network Activity (class_uid=4001) for PAN-OS GlobalProtect + VPN traffic logs. For general PAN-OS firewall logs (TRAFFIC, THREAT subtypes) see paloalto_logs/ + or paloalto_alternate_logs/. datasource_vendor: palo_alto dataSource: Paloalto Vpn Logs format: source-specific JSON/KV/syslog From 77cf7534916d417ac0f664325b83de91bf131a4a Mon Sep 17 00:00:00 2001 From: Nate Smalley Date: Sun, 26 Apr 2026 19:50:09 -0700 Subject: [PATCH 3/3] CHANGELOG: drop F-graded PAN-OS firewall transform; document variant binding Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7680f36..a7eaf32 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,25 @@ removed. It is functionally subsumed by Activity, `class_uid=4001`) for a broader range of log types. The now-empty `pipelines/community/serializers/` umbrella has been removed alongside it. +### Removed - F-graded `palo_alto_networks_firewall` transform + +`pipelines/community/transform_ocsf/palo_alto_networks_firewall/` has been +removed. It was graded F (`analyzer_limit`, 0% required-field coverage), used +a non-standard `class_uid=99602001` (SentinelOne Security Alert Extended) that +diverged from the rest of the PAN-OS cluster (`class_uid=4001` Network +Activity), and had no matching upstream parser in `parsers/community/` (its +`source_name` lacked the `-latest` versioning suffix used by every other +PAN-OS entry). The three remaining PAN-OS transforms (`paloalto_logs/`, +`paloalto_alternate_logs/`, `paloalto_vpn_logs/`) are unaffected. + +### Documented - PAN-OS transform variant binding + +The three remaining PAN-OS OCSF transforms in +`pipelines/community/transform_ocsf/` now declare in their `metadata.yaml` +`purpose` field which upstream parser in `parsers/community/` they bind to +and the field-name convention each expects, so users can choose between them +without reading the Lua. No serializer logic changes. + ## [1.3.0] - 2025-10-28 ### Added