feat: ES intel integration + refactor input_type to boolean flags

[CTIBurn0ut]

Summary

This PR implements two related changes that were designed and discussed together:

1. Replace input_type singleSelect with boolean checkboxes (globalConfig.json)

Before: A mutually exclusive singleSelect (kvstore | index) that forced a choice between two output modes.

After: Two independent checkbox fields:

• index_output — Enable Index Output (default: off) — write events to a Splunk index for audit/replay
• es_intel_output — Enable Splunk ES Threat Intel Integration (default: off) — push indicators into ES intel KV stores

KV Store is now always the primary, always-on store. Index and ES intel are additive options.

---

2. Refactor opencti_stream_helper.py — additive pipeline + ES intel bridge

Architecture change

The old if input_type == "kvstore" / elif input_type == "index" branching is replaced with a three-stage additive pipeline:

Event → Enrich (once)
          │
          ├─ [Always]      Write/delete TA KV Store (opencti_indicators, etc.)
          ├─ [Conditional] Write/delete Splunk ES intel KV stores (ip_intel, domain_intel, etc.)
          └─ [Conditional] Write to Splunk index (audit/replay)

ES Intel Bridge (new)

• A secondary es_service connection is opened scoped to SplunkEnterpriseSecuritySuite — this resolves the app namespace write permission issue that prevented the TA from writing to ES intel collections.
• write_to_es_intel() upserts into ip_intel, domain_intel, http_intel, file_intel, or email_intel based on main_observable_type.
• delete_from_es_intel() removes indicators scoped by both value and source (opencti:<input_name>) — safe blast radius, never touches other feeds.
• source field is set to opencti:<input_name> for full provenance in the ES Threat Intelligence Management UI.
• OpenCTI score (0–100) is mapped to ES weight (1–3).

Bug fixes included

#	Fix
exist_in_kvstore	Bare except replaced — only 404 returns False, all other errors re-raise
Index mode delete key	Was using parsed_stix.get("id") (wrong); now uses _key consistently everywhere
enrich_generic_payload	extensions dict was never deleted — now removed to prevent KV store bloat
Timestamp parsing	Now handles both %Y-%m-%dT%H:%M:%S.%fZ and %Y-%m-%dT%H:%M:%SZ formats
STIX pattern skip	Unsupported patterns now log a warning instead of a silent print

Post-review fixes

#	Fix
ES_INTEL_MAP	"Url" mapping corrected from url_intel → http_intel (Splunk ES uses http_intel for URL indicators)
batch_save()	Restored correct splunklib API usage: batch_save(*[record]) instead of batch_save(record)
globalConfig.json	Help text updated to reference http_intel instead of url_intel

New helpers

• map_score_to_weight(score) — OpenCTI score → ES weight
• parse_event_timestamp(ts, logger) — robust multi-format timestamp parser
• _es_intel_key(value, source) — stable MD5 _key for ES intel records
• ES_INTEL_MAP — observable type → ES collection + field mapping
• write_to_es_intel() / delete_from_es_intel() — ES intel write/delete with lazy handle caching

---

Files Changed

• TA-opencti-for-splunk-enterprise/globalConfig.json
• TA-opencti-for-splunk-enterprise/package/bin/opencti_stream_helper.py

Testing Notes

• ES intel output requires Splunk Enterprise Security to be installed. If the ES app context connection fails, es_intel_output is automatically disabled for that run with an error log — the TA continues normally.
• Existing inputs with input_type = kvstore or input_type = index will default both new flags to 0 (off) on upgrade — no breaking change for existing deployments.

[Copilot]

Pull request overview

Implements additive output routing for the OpenCTI stream modular input, making KV Store writes always-on while optionally supporting Splunk index audit/replay output and Splunk Enterprise Security (ES) Threat Intel KV-store integration.

Changes:

• Replaces the prior input_type single-select with two independent boolean flags: index_output and es_intel_output.
• Refactors opencti_stream_helper.py into a single enrich-once pipeline that always writes/deletes to TA KV store, and conditionally writes to ES intel KV stores and/or to a Splunk index.
• Adds ES intel bridging helpers (collection mapping, deterministic _key, write/delete helpers) and includes several enrichment/logging/timestamp robustness fixes.

Reviewed changes

Copilot reviewed 1 out of 2 changed files in this pull request and generated 1 comment.

File	Description
TA-opencti-for-splunk-enterprise/package/bin/opencti_stream_helper.py	Adds ES intel integration helpers and refactors the streaming handler into an additive pipeline with optional index + ES outputs.
TA-opencti-for-splunk-enterprise/globalConfig.json	Updates UCC input schema to replace input_type with index_output and es_intel_output checkboxes and updates the input table fields.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[CTIBurn0ut]

@romain-filigran Whenever you're ready

[CTIBurn0ut]

🐛 Bug: url_intel should be http_intel

Splunk Enterprise Security does not have a url_intel KV store collection. The correct ES collection name for URL indicators is http_intel.

Files to fix

1. opencti_stream_helper.py — ES_INTEL_MAP

# Current (wrong):
"Url":         ("url_intel",    "url"),

# Should be:
"Url":         ("http_intel",   "url"),

This will cause a collection not found error at runtime when the add-on tries to upsert URL indicators into ES threat intel.

2. globalConfig.json — es_intel_output help text

// Current (wrong):
"help": "Write indicators into Splunk Enterprise Security threat intelligence KV stores (ip_intel, domain_intel, url_intel, file_intel, email_intel)."

// Should be:
"help": "Write indicators into Splunk Enterprise Security threat intelligence KV stores (ip_intel, domain_intel, http_intel, file_intel, email_intel)."

Additional note: batch_save() call

In the primary KV store write path (stage 1), batch_save(parsed_stix) is called with a bare dict. The batch_save() method expects positional args (one record per arg) — the correct call is batch_save(*[parsed_stix]) (which was the original code before this PR changed it). Passing a dict directly may work in some splunklib versions but is not the documented API.

[CTIBurn0ut]

✅ Fixes pushed (2 commits)

Commit 1 — opencti_stream_helper.py

• ES_INTEL_MAP: Changed "Url": ("url_intel", "url") → "Url": ("http_intel", "url") — Splunk ES uses http_intel, not url_intel
• batch_save(): Changed batch_save(record) → batch_save(*[record]) to match the splunklib API (positional args, not a bare dict)

Commit 2 — globalConfig.json

• Help text: Changed url_intel → http_intel in the es_intel_output checkbox description to match the actual ES collection name