feat(corelight-investigator): create external-import connector for alerts and detections (#6736)

[SamuelHassine]

Proposed changes

New external-import connector for Corelight Investigator (SaaS NDR).

On a schedule the connector:

• queries the Investigator "Detections and Alerts" API (bearer API key), incrementally using connector state (configurable initial look-back window);
• maps each alert / detection to a STIX Incident, with the normalized Investigator severity (1-10) mapped to the OpenCTI severity and the EventType recorded as the incident type;
• extracts the source / destination IP observables referenced by an alert and creates related-to relationships from the Incident to those observables;
• attributes the imported objects to a Corelight Investigator author identity and applies a configurable TLP marking.

Related issues

Closes #6736

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist

• Unit tests for the client, converter and connector (coverage > 80% on new code)
• Code formatting (black, isort) and linting (flake8, STIX id pylint) pass
• Generated __metadata__ config schema and documentation (via the SDK schema tooling)
• Added/updated the connector README.md, config.yml.sample, docker-compose.yml and connector_manifest.json

Further comments

The exact REST path/parameters of the Investigator Detections and Alerts API are documented in the in-product API reference (require a tenant account), so the query endpoint path is configurable via alerts_path with a documented default, and the response parser accepts the common envelope keys (data / alerts / detections / results). This is the import side of a Corelight integration; the Corelight open sensor consumes intel via Zeek/TAXII (e.g. OpenCTI's native TAXII2 live feed) rather than an outbound IOC API.

Community status and manager support

This connector is community-contributed work and is NOT officially verified by Filigran: the manifest intentionally ships "verified": false and "last_verified_date": null. Do not flip verified back to true as part of this PR. Note: the VC201 verified-status check has been removed from the connector-linter on master, so the advisory Connector Verified Linter job passes cleanly with this community status.

The connector IS manager-supported ("manager_supported": true): configuration is loaded through the connectors-sdk pydantic-settings mechanism (env vars + optional config.yml), __metadata__/connector_config_schema.json is generated from the settings model with the SDK schema generator and CONNECTOR_CONFIG_DOC.md is generated from that schema. Manager-compatibility pass applied on top of the initial review:

• connector_manifest.json: verified: false, manager_supported: true, support_version: >=7.260701.0 (in sync with the pinned pycti).
• Dockerfile: direct ENTRYPOINT ["python", "main.py"] instead of the entrypoint.sh indirection (VC402); entrypoint.sh removed.
• CONNECTOR_SCOPE now has a default (corelight-investigator), so the only mandatory variables are the OpenCTI URL/token and the Corelight API base URL/key; schema and doc regenerated accordingly.
• docker-compose.yml / config.yml.sample: exact ChangeMe placeholders, defaulted variables commented out (VC104 sample conventions).
• connector.py: a work is only initiated when the run actually collected data, so empty runs no longer create zero-bundle works (VC317).
• Local connector-linter run on the rebased branch: 36/36 checks pass (score 100%), zero errors and zero warnings; the CI Connector Verified Linter job on the connector is green as well.

Maintainer review (independent review-and-fix)

Independent senior review of the full connector, on top of the automated review threads. Substantive fixes:

• converter_to_stix.py: TLP:CLEAR and TLP:AMBER+STRICT are built as custom statement MarkingDefinitions via a shared _statement_marking helper (matching connectors-sdk TLPMarking.to_stix2_object), so OpenCTI shows the correct TLP label instead of aliasing stix2.TLP_WHITE / TLP_AMBER.
• converter_to_stix.py (deterministic ids): Incident.generate_id is seeded with the alert id in the name (so distinct alerts that share a name do not collapse into one Incident) plus the source timestamp; when an alert has no usable timestamp, created/modified fall back to a fixed sentinel instead of "now", so a re-imported alert keeps a stable id and is not re-sent with drifting timestamps each run.
• connector.py: works are finalized with in_error=True when a failure occurs after the work was initiated; a fetch failure raises (the client raises after retries), so last_run is not advanced on error and the next run retries.
• Docs: the README requirements and configuration tables reflect the actual defaults (OpenCTI >= 7.260701.0, scope / log_level / duration_period optional with defaults); the README Table of Contents anchor and logging example fixed.
• Dependencies: pycti pinned to the latest release 7.260701.0 (aligned with connectors-sdk on master, which is installed from master, never a pinned tag).

Tests

Unit tests cover the settings model, the API client (request shaping, retries, error wrapping), the converter (TLP:CLEAR / AMBER+STRICT marking shapes, severity mapping, _parse_timestamp, deterministic and collision-free incident ids, stable fallback timestamps, IP observable extraction + relationships) and the connector (no work initiated on empty runs, work finalized in_error when the bundle send fails). 44 unit tests pass; black / isort / flake8 clean locally.

Status

The branch is rebased onto the latest master (linear history, all commits GPG-signed). All CI checks are green, including the advisory Connector Verified Linter job (36/36 checks on this connector after VC201 removal upstream). 0 unresolved review threads.

[Copilot]

Pull request overview

Adds a new external-import connector under external-import/corelight-investigator/ to pull Corelight Investigator “Detections and Alerts” incrementally, convert them into STIX 2.1 Incidents plus related IP observables, and send bundles to OpenCTI with stateful scheduling and configurable TLP.

Changes:

• Implemented a Corelight Investigator HTTP client with basic retry/backoff and response-envelope normalization.
• Implemented STIX conversion + connector runtime (stateful incremental since, bundling, scheduling).
• Added full connector packaging (Docker, compose, sample config, metadata) and unit tests for client/settings/converter/connector.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
external-import/corelight-investigator/src/corelight_investigator_client/api_client.py	Requests-based API client with retries/backoff and payload extraction.
external-import/corelight-investigator/src/corelight_investigator_client/init.py	Exposes client + error types as a small import surface.
external-import/corelight-investigator/src/connector/converter_to_stix.py	Alert/detection → STIX Incident + IP observables + relationships conversion.
external-import/corelight-investigator/src/connector/connector.py	Connector orchestration: stateful “since”, bundling, send, work handling, scheduling.
external-import/corelight-investigator/src/connector/settings.py	Pydantic/connectors-sdk settings models for OpenCTI + connector + Corelight config.
external-import/corelight-investigator/src/connector/init.py	Exposes connector entry classes.
external-import/corelight-investigator/src/main.py	Runtime entrypoint wiring settings → helper → connector.
external-import/corelight-investigator/src/requirements.txt	Connector runtime dependencies.
external-import/corelight-investigator/tests/conftest.py	Pytest fixtures/mocking for OpenCTIConnectorHelper.
external-import/corelight-investigator/tests/test_main.py	Smoke tests for settings/helper/connector instantiation.
external-import/corelight-investigator/tests/test-requirements.txt	Test dependency pinning for this connector.
external-import/corelight-investigator/tests/tests_corelight_investigator_client/test_api_client.py	Unit tests for API client behavior (envelope parsing, params, errors, rate-limit retry).
external-import/corelight-investigator/tests/tests_connector/test_settings.py	Unit tests for settings validation behavior.
external-import/corelight-investigator/tests/tests_connector/test_converter.py	Unit tests for severity mapping, incident creation, observable creation.
external-import/corelight-investigator/tests/tests_connector/test_connector.py	Unit tests for connector orchestration, bundling, and state updates.
external-import/corelight-investigator/Dockerfile	Container build for the connector.
external-import/corelight-investigator/entrypoint.sh	Container entrypoint wrapper.
external-import/corelight-investigator/docker-compose.yml	Example deployment configuration.
external-import/corelight-investigator/config.yml.sample	Manual deployment sample configuration.
external-import/corelight-investigator/README.md	Connector documentation and configuration reference.
external-import/corelight-investigator/metadata/connector_manifest.json	Connector marketplace/manifest metadata.
external-import/corelight-investigator/metadata/connector_config_schema.json	Generated config schema for the connector’s env vars.
external-import/corelight-investigator/metadata/CONNECTOR_CONFIG_DOC.md	Generated connector configuration documentation.

[codecov]

Codecov Report

❌ Patch coverage is 92.97521% with 17 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...or/src/corelight_investigator_client/api_client.py	81.96%	11 Missing ⚠️
.../corelight-investigator/src/connector/connector.py	93.54%	4 Missing ⚠️
...ht-investigator/src/connector/converter_to_stix.py	97.89%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #6737       +/-   ##
===========================================
+ Coverage   35.64%   86.56%   +50.91%     
===========================================
  Files        2048        8     -2040     
  Lines      125310      588   -124722     
===========================================
- Hits        44672      509    -44163     
+ Misses      80638       79    -80559     

Files with missing lines	Coverage Δ
...t/corelight-investigator/src/connector/__init__.py	100.00% <100.00%> (ø)
...t/corelight-investigator/src/connector/settings.py	100.00% <100.00%> (ø)
...ator/src/corelight_investigator_client/__init__.py	100.00% <100.00%> (ø)
...ht-investigator/src/connector/converter_to_stix.py	97.89% <97.89%> (ø)
.../corelight-investigator/src/connector/connector.py	93.54% <93.54%> (ø)
...or/src/corelight_investigator_client/api_client.py	81.96% <81.96%> (ø)

... and 2046 files with indirect coverage changes

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[SamuelHassine]

Review-and-fix pass summary

Independent senior re-review of the full new connector plus the two open Copilot threads.

Code fixes (commit 40c75e5):

• converter_to_stix.py: TLP:CLEAR is no longer aliased to stix2.TLP_WHITE. TLP:CLEAR and TLP:AMBER+STRICT are now custom statement MarkingDefinitions built via a shared _statement_marking helper (matching connectors-sdk TLPMarking.to_stix2_object), so OpenCTI shows the correct TLP label instead of TLP:WHITE / TLP:AMBER.
• converter_to_stix.py: Incident.generate_id is seeded with the source timestamp only. With no usable timestamp the STIX created/modified still fall back to a timezone-aware now, but the id seed is None, so a re-imported alert keeps a stable Incident id instead of duplicating on every run.
• connector.py: the initiated work is finalized with in_error=True on the API-error and unexpected-error paths, so a failed run no longer leaves a dangling in-progress work in OpenCTI.
• Minor: _request typed Optional[dict]; removed a stray space in the pydantic version pin.

Tests (commits 40c75e5, 9988e06): added coverage for the distinct TLP:CLEAR marking, the deterministic Incident id, the source-timestamp passthrough, _parse_timestamp, and the in-error work finalize on both error paths.

Status: all CI checks are green (codecov/patch 88.44%, target 80%), black / isort / flake8 are clean locally, and there are 0 unresolved review threads (both Copilot threads resolved). The commits are GPG-signed.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me. As the author I cannot self-approve.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.

[SamuelHassine]

Review-and-fix pass summary

Independent senior re-review of the full external-import/corelight-investigator connector plus the 3 remaining open Copilot threads (commit 0d040c3).

• Converter determinism (proactive - same class of bug flagged on the sibling arcsight-incidents connector): Incident.generate_id is now seeded with the alert id in the name (f"{name} [{alert_id}]"), so distinct alerts that share a name no longer collapse into one Incident, and when the alert has no usable timestamp created/modified fall back to a fixed sentinel instead of "now" (no drift / needless updates on re-import). Covered by test_incidents_with_same_name_different_id_have_distinct_ids and test_incident_timestamps_use_stable_fallback_without_timestamp.
• Docs (the 3 open threads): scope / CONNECTOR_SCOPE marked mandatory in config.yml.sample and docker-compose.yml (required by the generated schema), and the README minimum version aligned to OpenCTI >= 6.8.12 to match the manifest support_version.
• Independent re-review confirmed the rest is sound: the client raises on fetch failure after retries (so last_run is not advanced and the run retries), the work item is finalized in_error on both error paths, and the TLP:CLEAR / AMBER+STRICT markings use the connectors-sdk statement-marking shape.
• Verification: black / isort / flake8 clean and all 42 unit tests pass locally.
• CI: all checks green on 0d040c3 (tests, lint/format, STIX ID linter, codecov/patch, codecov/project). filigran/cla satisfied (organization member).
• Review threads: 0 unresolved (all 3 replied to and resolved).
• Description refreshed to the final state; title already matches the repo's Conventional Commits convention.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me. As the author I cannot self-approve, so this requires another Filigran maintainer.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

[SamuelHassine]

Review-and-fix pass summary (rebase + pycti alignment)

Full pass on the external-import/corelight-investigator connector, head c8a586e1e8.

• Rebase: the branch is rebased onto the latest origin/master (clean linear history, no merge commits, all 5 commits GPG-signed) and force-with-lease pushed.
• Dependencies: pycti bumped to the latest release 7.260701.0 in src/requirements.txt (aligned with connectors-sdk on master; the SDK stays installed from master, never a pinned tag). The README pycti example was updated accordingly; the manifest support_version (>=6.8.12) and README minimum platform version are already in sync.
• Review threads: the 2 open Copilot threads (broken README ToC anchor, invalid logging snippet with a comma instead of attribute access) are fixed in c8a586e1e8, replied to and resolved - 0 unresolved threads remain.
• Independent re-review: re-checked the converter (statement markings for TLP:CLEAR / AMBER+STRICT, deterministic and collision-free incident ids, stable fallback timestamps), the client (retry/backoff, error wrapping) and the connector (work finalized in_error on both error paths, last_run not advanced on failure) - no new findings.
• Verification: 42 unit tests pass locally; black / isort / flake8 clean.
• CI: all checks green on c8a586e1e8 (tests, lint/format, STIX ID linter, unused-deps, identity uniqueness, codecov/patch, codecov/project, filigran/cla). mergeable: MERGEABLE, mergeStateStatus: CLEAN.

Remaining (non-CI) blocker: GitHub refuses self-approval ("Can not approve your own pull request"), so reviewDecision stays REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than the author. Everything else is ready to merge.