feat(arcsight-incidents): add external-import connector for ArcSight ESM cases (#6723)

[SamuelHassine]

Proposed changes

This PR adds a new EXTERNAL_IMPORT connector external-import/arcsight-incidents that imports ArcSight ESM cases into OpenCTI as STIX Incidents and Case-Incidents. It is the import side of a bidirectional ArcSight integration (paired with the stream/arcsight connector that pushes IOCs to ESM Active Lists).

• Authenticates against the ESM LoginService, lists case ids via CaseService (capped per run) and fetches each case and its security events.
• Models ArcSight security events as STIX Incidents and the ArcSight case that groups them as a STIX Case-Incident (referencing the event Incidents through object_refs), attributed to an ArcSight author identity and marked with a configurable TLP.
• Built on the modern connectors-sdk settings pattern, with unit tests and connector metadata (manifest, config schema, configuration documentation).

Related issues

Closes #6723

Type of change

• New feature (non-breaking change which adds functionality).

Maintainer review (independent review-and-fix)

Independent senior review of the full connector across several passes, on top of the Copilot threads. Substantive fixes:

• Security: arcsight_client/api_client.py _request never logs str(err). The ESM LoginService login carries the password as a query parameter (and findAllIds / getResourceById carry the auth token), and a requests exception string usually embeds the full request URL, so str(err) could leak those secrets. Errors now log only path + status_code + exception type.
• Security: connector.py top-level error handler no longer logs or forwards str(err) either; it logs the exception type via structured meta and finalizes the work with a sanitized message (ArcSight Incidents connector run failed: <ExceptionType>), so a requests exception URL carrying the password/token cannot leak into the logs or the work message.
• converter_to_stix.py: tlp_level="clear" emits a dedicated TLP:CLEAR custom statement marking (x_opencti_definition="TLP:CLEAR") instead of aliasing stix2.TLP_WHITE; amber+strict likewise emits the OpenCTI-specific marking.
• converter_to_stix.py (deterministic ids): Incident.generate_id / CaseIncident.generate_id are seeded with the event id / case external id in the name (so distinct events or cases that share a display name no longer collapse into one entity), and when the source carries no usable timestamp created/modified fall back to a fixed sentinel instead of "now". The Case-Incident modified also falls back to the stable created (never "now") when the modified timestamp is missing or unparseable, so a re-imported case keeps a stable id and is not re-sent with a drifting modified each run.
• api_client.py: _request retries only on connection/timeout errors, 429 and 5xx; other 4xx (401/403/404) fail fast. It also closes the HTTP response on the 429/5xx-retry and non-retriable HTTPError paths, so a periodic run does not keep connections checked out of the requests Session pool.
• connector.py: a work is only initiated when the run actually collected data (no empty works in the OpenCTI jobs UI), and an initiated work is finalized with in_error=True on the failure path, so a failed run does not leave a dangling in-progress work.

Connector metadata and linter alignment

The branch was rebased onto current master, which introduced the Connector Verified Linter CI job, and the connector was aligned with its conventions:

• Dockerfile: the legacy entrypoint.sh wrapper is removed in favor of a direct ENTRYPOINT ["python3", "main.py"] (VC402).
• docker-compose.yml: image tag is opencti/connector-arcsight-incidents:latest (VC401), variables with defaults are commented out, and the required CONNECTOR_SCOPE uses ChangeMe (VC104); the README build example is aligned.
• config.yml.sample moved from src/ to the connector root (VC104).
• connector.py: initiate_work is now guarded by data availability (VC317), covered by new unit tests.
• use_cases in the manifest are aligned with the official XTM Hub use case taxonomy introduced by feat(xtm-hub): harmonize connector manifests with official XTM Hub use case taxonomy #6905 (Detection & Response Enablement, SIEM and Analytics), matching comparable SIEM incident-import connectors.
• The __metadata__/connector_config_schema.json and CONNECTOR_CONFIG_DOC.md were regenerated with the connectors-sdk JSON-schema tooling and match the settings model exactly (no drift).

Verified status: intentionally false (community connector)

This connector is community-contributed work and is NOT officially verified by Filigran, so the manifest intentionally declares "verified": false and "last_verified_date": null ("manager_supported" stays true). The advisory (continue-on-error) Connector Verified Linter job therefore reports two expected VC201 findings about the verified status; these are accepted and must not be "fixed" by flipping the flag - the verified status is granted by the Filigran product team after end-to-end validation. This is the same state as other recently merged community connectors (e.g. comlaude #6871). Everything else in the linter passes (36/38 checks).

Dependencies

• pycti is pinned to 7.260701.0 (latest release, matching the connectors-sdk pin on master); the README minimum platform version and the manifest support_version are aligned.
• connectors-sdk is installed from master (never a pinned tag).

Branch state

The branch is a clean linear history on top of current origin/master. All commits are GPG-signed.

Tests

Unit tests cover the settings model (validation, defaults), the ArcSight client (auth, case listing/fetch, 4xx fail-fast vs 429/5xx retry, credential-redaction in logs), the converter (TLP:CLEAR marking, severity mapping, deterministic and collision-free incident/case ids, stable fallback timestamps, modified-falls-back-to-created) and the connector (work only initiated when data is available, work finalized in_error when a post-initiation step fails). 56 unit tests pass; black / isort / flake8 --ignore=E,W clean locally.

Checklist

• My code follows the repository code style (isort, black, flake8, STIX-id pylint).
• I have added unit tests covering the settings, the client, the converter and the connector.
• I have updated the documentation (README, connector metadata).
• Commits are signed.

Status

All required CI checks are green (tests, lint/format, STIX ID linter, codecov/patch and codecov/project). The advisory Lint external-import/arcsight-incidents job (continue-on-error) reports only the two expected VC201 verified-status findings, which are intentional for a community connector (see above). There are 0 unresolved review threads. The PR needs one approving review from a maintainer other than the author (as the author I cannot self-approve). The branch is linear on top of master, so it can be merged with any strategy.

[codecov]

Codecov Report

❌ Patch coverage is 94.39252% with 18 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...csight-incidents/src/arcsight_client/api_client.py	89.74%	16 Missing ⚠️
...port/arcsight-incidents/src/connector/connector.py	96.15%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #6724       +/-   ##
===========================================
+ Coverage   35.64%   88.00%   +52.35%     
===========================================
  Files        2048        8     -2040     
  Lines      125310      667   -124643     
===========================================
- Hits        44672      587    -44085     
+ Misses      80638       80    -80558     

Files with missing lines	Coverage Δ
...arcsight-incidents/src/arcsight_client/__init__.py	100.00% <100.00%> (ø)
...mport/arcsight-incidents/src/connector/__init__.py	100.00% <100.00%> (ø)
...sight-incidents/src/connector/converter_to_stix.py	100.00% <100.00%> (ø)
...mport/arcsight-incidents/src/connector/settings.py	100.00% <100.00%> (ø)
...port/arcsight-incidents/src/connector/connector.py	96.15% <96.15%> (ø)
...csight-incidents/src/arcsight_client/api_client.py	89.74% <89.74%> (ø)

... and 2046 files with indirect coverage changes

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Adds a new external-import/arcsight-incidents connector that pulls ArcSight ESM cases via the Service Layer REST API and imports them into OpenCTI as STIX 2.1 Incidents, including connector packaging (Docker/compose), metadata, and unit tests.

Changes:

• Introduces ArcSight ESM client logic (login, list case IDs, fetch cases) with retry/backoff behavior.
• Implements case → STIX Incident conversion (author identity, TLP marking, timestamps, severity, external reference) and the connector runtime loop/state handling.
• Adds connector configuration (connectors-sdk settings), generated metadata/config docs, and unit tests (settings/client/converter/connector).

Reviewed changes

Copilot reviewed 24 out of 26 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
external-import/arcsight-incidents/tests/tests_connector/test_settings.py	Settings validation tests for connectors-sdk config model.
external-import/arcsight-incidents/tests/tests_connector/init.py	Test package init.
external-import/arcsight-incidents/tests/test-requirements.txt	Test dependency definition.
external-import/arcsight-incidents/tests/test_main.py	Smoke tests for settings/helper/connector instantiation.
external-import/arcsight-incidents/tests/test_converter.py	Unit tests for case→incident conversion helpers.
external-import/arcsight-incidents/tests/test_connector.py	Unit tests for connector orchestration and helper interactions.
external-import/arcsight-incidents/tests/test_client.py	Unit tests for ArcSight API client behaviors and parsing.
external-import/arcsight-incidents/tests/conftest.py	Test path bootstrap to import connector modules.
external-import/arcsight-incidents/tests/init.py	Test package init.
external-import/arcsight-incidents/src/requirements.txt	Connector runtime dependencies.
external-import/arcsight-incidents/src/main.py	Connector entrypoint wiring settings/helper/connector.
external-import/arcsight-incidents/src/connector/settings.py	connectors-sdk based settings models for this connector.
external-import/arcsight-incidents/src/connector/converter_to_stix.py	ArcSight case → STIX 2.1 Incident conversion.
external-import/arcsight-incidents/src/connector/connector.py	Main connector loop: collect, bundle, send, and persist state.
external-import/arcsight-incidents/src/connector/init.py	Package exports for connector components.
external-import/arcsight-incidents/src/config.yml.sample	Sample YAML configuration for deployment.
external-import/arcsight-incidents/src/arcsight_client/api_client.py	HTTP client for ArcSight LoginService/CaseService endpoints.
external-import/arcsight-incidents/src/arcsight_client/init.py	Package export for ArcSight client.
external-import/arcsight-incidents/README.md	Usage and configuration documentation.
external-import/arcsight-incidents/entrypoint.sh	Container entrypoint.
external-import/arcsight-incidents/Dockerfile	Container build for the connector.
external-import/arcsight-incidents/docker-compose.yml	Example docker-compose deployment.
external-import/arcsight-incidents/.dockerignore	Docker build context exclusions.
external-import/arcsight-incidents/metadata/connector_manifest.json	Connector manifest metadata for registry/manager.
external-import/arcsight-incidents/metadata/connector_config_schema.json	Generated JSON schema for config variables.
external-import/arcsight-incidents/metadata/CONNECTOR_CONFIG_DOC.md	Generated config documentation.

[SamuelHassine]

Review-and-fix pass summary

Independent senior review of the full new connector plus all 5 remaining open Copilot threads.

• Security (my finding, not flagged by the bot): arcsight_client/api_client.py _request no longer logs str(err). The ESM LoginService login carries the password as a query parameter (and findAllIds / getResourceById carry the auth token), and a requests exception string usually embeds the full request URL, so str(err) could leak those secrets into the logs. It now logs only path + status_code + error_type (fixed in c5108b0, with redaction tests).
• The 5 open Copilot threads were already addressed by earlier commits on the branch and are now replied to and resolved: TLP:CLEAR emits a dedicated OpenCTI marking (not TLP_WHITE); the API client uses meta={...} structured logging; and config.yml.sample / docker-compose.yml / README.md all mark scope / CONNECTOR_SCOPE as required.
• Also aligned the _load_config_dict test overrides with the SDK (-> Self).

Verified locally: 45/45 unit tests pass (added 2 redaction tests); black / isort / flake8 --select=F clean. CI: all checks green (tests, lint/format, STIX ID linter, codecov/patch and codecov/project); filigran/cla satisfied (organization member). Review threads: 0 unresolved. Title already matches the Conventional Commits convention; description updated to reflect the final state.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me. As the author I cannot self-approve, so this requires another Filigran maintainer.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 26 changed files in this pull request and generated 3 comments.

[SamuelHassine]

Second review-and-fix pass summary

Independent senior re-review of the full external-import/arcsight-incidents connector plus the 3 remaining open Copilot threads.

Code fixes (commit 67fe59b):

• Deterministic ids: Incident.generate_id and CaseIncident.generate_id are now seeded with the source timestamp only. _to_iso is split into _parse_timestamp (returns None for a missing or unparseable timestamp) and _format_iso, so a re-imported event or case keeps a stable id instead of creating a duplicate Incident / Case-Incident every run (covers both the missing-timestamp and unparseable-timestamp threads).
• Dangling work: the initiated work is finalized with in_error=True on the failure path.
• Dockerfile: kept as-is with the rationale on the thread (the apk line and its build deps are the shared pattern across the sibling connectors; changing only this one risks a build break) - a leaner shared base image is better handled as a repo-wide change.

Also verified the connector does not have the data-loss pattern from the sibling fortisiem-incidents connector: get_cases lists all case ids each run (no incremental cursor) and last_run is not used to filter.

Tests: added the deterministic id (missing and unparseable timestamp, Incident and Case-Incident), the scalar baseEventIds branch, and the in-error work finalize; the converter module is now 100% covered.

Status: all CI checks are green (codecov/patch and codecov/project, target 80%), black / isort / flake8 are clean locally, 49/49 unit tests pass, and there are 0 unresolved review threads (11 resolved). All commits are GPG-signed.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me. As the author I cannot self-approve.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 26 changed files in this pull request and generated 2 comments.

[SamuelHassine]

Review-and-fix pass summary

Independent senior re-review of the full external-import/arcsight-incidents connector plus the two remaining open Copilot threads (both on the converter's id/timestamp determinism).

• Code fix (commit 52a42d2): Incident.generate_id / CaseIncident.generate_id are now seeded with the event id / case external id in the name (f"{name} [{event_id}]" and f"{name} [{external_id}]"), so distinct events or cases that share a display name no longer collapse into one entity (the repo pattern used by recorded-future / intel471). When the source carries no usable timestamp, created/modified now fall back to a fixed _FALLBACK_TIMESTAMP sentinel instead of now, so a re-imported event/case keeps a stable id and is not re-sent with drifting timestamps each run.
• Tests: added test_incidents_with_same_name_different_event_id_have_distinct_ids, test_case_incidents_with_same_name_different_external_id_have_distinct_ids, and the stable-fallback-timestamp tests for both incidents and case-incidents (suite 49 -> 53).
• Independent re-review of the rest found no further issues: the credential-redacting client logging, the 4xx fail-fast vs 429/5xx retry, the distinct TLP:CLEAR marking, and the in_error work finalization are all correct and well covered.
• Verification: black / isort / flake8 --ignore=E,W clean and all 53 unit tests pass locally.
• CI: all checks green on 52a42d2 (tests, lint/format, STIX ID linter, codecov/patch, codecov/project). filigran/cla satisfied (organization member).
• Review threads: 0 unresolved (all Copilot threads replied to and resolved).
• Description refreshed to the final state; title already matches the repo's Conventional Commits convention.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me. As the author I cannot self-approve, so this requires another Filigran maintainer.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 26 changed files in this pull request and generated 2 comments.

[SamuelHassine]

Review-and-fix pass summary

Independent senior re-review of the full external-import/arcsight-incidents connector plus the 2 newly-opened Copilot threads.

Code/test fixes (commit 6c80ab1):

• converter_to_stix.py: the Case-Incident modified now falls back to the already-stable created value (never "now") when the ArcSight modified timestamp is missing or unparseable, so the deterministic id is no longer re-sent with a drifting modified each run. Covered by test_case_incident_modified_falls_back_to_created_when_unparseable.
• connector.py: the top-level error handler no longer logs or forwards str(err) (the ESM auth flow carries the password/token in query parameters, which a requests exception string can embed); it logs the exception type via structured meta and finalizes the work with a sanitized message. The existing failure test still asserts the work is finalized with in_error=True.
• arcsight_client/api_client.py (independent finding): _request now closes the HTTP response on the 429/5xx-retry and non-retriable HTTPError paths, so a periodic run does not keep connections checked out of the requests Session pool. This is the same hardening the sibling stream/arcsight client received; the bot did not flag it here, but the code path was identical.
• src/requirements.txt: pinned pycti bumped to 7.260615.0 to match the current connectors-sdk; the README minimum and the manifest support_version are aligned.

CI fix (de-stale):

• The branch was behind master, and run_test.sh installs the local connectors-sdk from the checked-out tree. The stale copy still pinned pycti==7.260609.0, which would conflict with the 7.260615.0 reinstalled from opencti master on the push-event test run. Merging current master updates the local connectors-sdk pin, so both test runs resolve consistently.

Verification:

• black / isort --profile black / flake8 --ignore=E,W clean and all 54 unit tests pass locally (53 + the new modified-fallback regression test); the new response.close() paths are exercised by the existing retry/fail-fast tests, so patch coverage is preserved. CI is fully green on the new head (both Test external-import/arcsight-incidents runs, lint/format, STIX ID linter, codecov/patch, codecov/project). filigran/cla satisfied (organization member).
• Review threads: 0 unresolved (both Copilot threads replied to and resolved). Both new commits are GPG-signed.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a Filigran maintainer other than me. As the author I cannot self-approve. Once approved, recommend squash-merge (the branch carries a de-stale merge commit).

[Copilot]

⚠️ Human review recommended

It introduces a full new connector (API client + STIX modeling + deployment artifacts), which warrants a final maintainer review despite the strong unit-test coverage.

Copilot's findings

• Files reviewed: 24/26 changed files
• Comments generated: 0 new

Note

Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.

[SamuelHassine]

Review-and-fix pass summary

Full pass on the connector: rebase, dependency alignment, independent re-review, and alignment with the new Connector Verified Linter.

• Rebase: the branch is now a clean linear rebase onto current origin/master (the previous de-stale merge commit was flattened, no conflicts). All 11 commits are GPG-signed.
• Dependencies: pycti bumped to 7.260701.0 (latest release, matching the connectors-sdk pin on master); README minimum platform version and manifest support_version aligned.
• Connector Verified Linter alignment (job introduced on master since the last push):
  • VC402: removed the legacy entrypoint.sh, the Dockerfile now uses a direct ENTRYPOINT ["python3", "main.py"].
  • VC401: docker-compose.yml image tag is now :latest (README build example aligned).
  • VC104: config.yml.sample moved to the connector root; compose variables with defaults commented out; required CONNECTOR_SCOPE uses ChangeMe.
  • VC317: process_message now only initiates a work when the run collected data, so empty runs no longer create empty works. Failures before work initiation have nothing to finalize; failures after initiation still finalize the work with in_error=True. Covered by new unit tests (suite 54 -> 56).
• Independent re-review of the full connector found no further issues: credential-redacting logging, 4xx fail-fast vs 429/5xx retry with response closing, distinct TLP:CLEAR marking, deterministic collision-free ids and stable fallback timestamps are all correct and covered.
• Verification: black / isort --profile black / flake8 --ignore=E,W clean and all 56 unit tests pass locally; CI is green on the new head (both connector test runs, lint/format, STIX ID linter, codecov/patch, codecov/project, all PR convention checks, filigran/cla).
• Review threads: 0 unresolved (all previously replied and resolved).

Known non-blocking item: the advisory Lint external-import/arcsight-incidents job (continue-on-error) reports only VC201 ("verified": true + last_verified_date). This flag is set by the Filigran product team after end-to-end validation (templates mark these fields "DO NOT CHANGE - FOR INTERNAL USE ONLY"), so it is intentionally left as false - the same state as other recently merged manager-supported connectors (e.g. #6871).

Remaining (non-CI) blocker: mergeStateStatus requires one approving review from a maintainer other than the author - as the author I cannot self-approve.