[ti_recordedfuture] Add identity detection data stream by brijesh-elastic · Pull Request #19826 · elastic/integrations

brijesh-elastic · 2026-06-29T12:54:14Z

Proposed commit message

ti_recordedfuture: Add support for identity detection data stream.

This data stream retrieves identity exposure detections using [Detections API](1).

Test samples were derived from documentation, 
which were subsequently sanitized.

[1] https://docs.recordedfuture.com/reference/identity-detections

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/ti_recordedfuture directory.
Run the following command to run tests.

elastic-package test -v

Related issues

Closes [Recorded Future] Add Identity Detections Support #19406

Screenshots

github-actions · 2026-06-29T12:56:10Z

Elastic Docs Style Checker (Vale)

Summary: 1 warning, 2 suggestions found

⚠️ Warnings (1): Fix when the suggestion improves clarity or correctness.

File	Line	Rule	Message
packages/ti_recordedfuture/data_stream/identity_detection/manifest.yml	41	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.

💡 Suggestions (2): Optional style improvements. Apply when helpful.

File	Line	Rule	Message
packages/ti_recordedfuture/data_stream/identity_detection/manifest.yml	88	Elastic.WordChoice	Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI.
packages/ti_recordedfuture/data_stream/identity_detection/manifest.yml	97	Elastic.WordChoice	Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI.

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

elastic-vault-github-plugin-prod · 2026-06-29T13:18:24Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

infra-vault-gh-plugin-prod · 2026-06-30T10:24:03Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

vera-review-bot · 2026-06-30T10:38:02Z

👀 I have started reviewing the PR

vera-review-bot · 2026-06-30T11:00:26Z

👀 I have started reviewing the PR

vera-review-bot · 2026-06-30T11:17:02Z

Vera Review Bot

For the current commit state, I did not find any issues.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

kcreddy · 2026-06-30T11:39:02Z

+{{#if preserve_duplicate_custom_fields}}
+  - preserve_duplicate_custom_fields
+{{/if}}


Can remove these references since we are not defining preserve_duplicate_custom_fields inside manifest

kcreddy · 2026-06-30T12:01:45Z

+        secret: false
+        default: false
+        description: >-
+          When enabled, the leaked password cleartext value is replaced with a keyed hash (HMAC-SHA256) by the agent before the data reaches Elasticsearch. When disabled, the cleartext value is ingested as-is. Requires `Hashing Key` to be set.


Since by default (hash_password: false, hash_cookies: false) the leaked password cleartext and cookie values are written to ES, we should call this out in README as well and what users need to do to avoid it.

vera-review-bot · 2026-06-30T13:25:29Z

👀 I have started reviewing the PR

vera-review-bot · 2026-06-30T13:42:38Z

+  - append:
+      field: event.category
+      tag: append_malware_into_event_category
+      value: malware


🟡 MEDIUM data_stream/identity_detection/.../default.yml:269

event.category hardcoded to malware for non-malware detections

The pipeline unconditionally appends malware to event.category for every identity detection. Detections sourced from database breaches (source_type: DatabaseDumps and DatabaseCombolists) are not malware-derived, so this assigns an inaccurate ECS category to those records. This is visible in the pipeline test data: the record with id a1b2c3d4e5f6789012345678abcdef01 has source_type: DatabaseDumps yet receives event.category: [malware] in the expected output. Mixing accurate categories (stealer-malware logs) with inaccurate ones (database dumps) under a single hardcoded value reduces the precision of category-based filtering and detection rules.

Recommendation:

Categorize based on the detection's source_type so database-breach records are not labelled malware. For example, only set malware for malware-sourced detections and fall back to a threat-intel-appropriate category otherwise:

- append: field: event.category tag: append_malware_into_event_category value: malware allow_duplicates: false if: ctx.recordedfuture?.identity_detection?.source_type != null && ctx.recordedfuture.identity_detection.source_type.toLowerCase().contains('malware') - append: field: event.category tag: append_threat_into_event_category value: threat allow_duplicates: false if: ctx.recordedfuture?.identity_detection?.source_type != null && !ctx.recordedfuture.identity_detection.source_type.toLowerCase().contains('malware')

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

…ent categorizations according to vera comments

vera-review-bot · 2026-06-30T14:26:51Z

👀 I have started reviewing the PR

elastic-vault-github-plugin-prod · 2026-06-30T14:28:41Z

✅ All changelog entries have the correct PR link.

vera-review-bot · 2026-06-30T14:38:02Z

Vera Review Bot

For the current commit state, I did not find any issues.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

infra-vault-gh-plugin-prod · 2026-06-30T14:50:23Z

💚 Build Succeeded

Buildkite Build
Commit: d2049fe

History

💚 Build #45249 succeeded 4d9dd47
💚 Build #45200 succeeded b891b83

cc @brijesh-elastic

efd6 · 2026-07-02T01:37:52Z

+    state.?want_more.orValue(false) ?
+      state
+    :
+      state.drop(["offset"]).with(


Suggested change

state.drop(["offset"]).with(

state.drop("offset").with(

efd6 · 2026-07-02T01:39:12Z

+        "application/json",
+        {
+          "limit": int(state.batch_size),
+          ?"offset": (state.?offset.orValue("") != "") ? optional.of(string(state.offset)) : optional.none(),


Can we use absence instead of "" to indicate absence.

efd6 · 2026-07-02T01:39:30Z

+        {
+          "limit": int(state.batch_size),
+          ?"offset": (state.?offset.orValue("") != "") ? optional.of(string(state.offset)) : optional.none(),
+          ?"organization_id": state.?organization_id[0].hasValue() ? optional.of(state.organization_id) : optional.none(),


Suggested change

?"organization_id": state.?organization_id[0].hasValue() ? optional.of(state.organization_id) : optional.none(),

?"organization_id": state.?organization_id[0],

efd6 · 2026-07-02T01:40:10Z

+          "limit": int(state.batch_size),
+          ?"offset": (state.?offset.orValue("") != "") ? optional.of(string(state.offset)) : optional.none(),
+          ?"organization_id": state.?organization_id[0].hasValue() ? optional.of(state.organization_id) : optional.none(),
+          ?"include_enterprise_level": state.?include_enterprise_level.orValue(false) ? optional.of(true) : optional.none(),


Suggested change

?"include_enterprise_level": state.?include_enterprise_level.orValue(false) ? optional.of(true) : optional.none(),

?"include_enterprise_level": state.?include_enterprise_level,

efd6 · 2026-07-02T01:40:25Z

+          ?"include_enterprise_level": state.?include_enterprise_level.orValue(false) ? optional.of(true) : optional.none(),
+          "filter": {
+            "created": {"gte": string(state.start_time)},
+            ?"novel_only": state.?novel_only.orValue(false) ? optional.of(true) : optional.none(),


Suggested change

?"novel_only": state.?novel_only.orValue(false) ? optional.of(true) : optional.none(),

?"novel_only": state.?novel_only,

efd6 · 2026-07-02T01:46:47Z

+                ""
+            ).as(page_max,
+              (
+                (page_max > state.?cursor.max_created.orValue("")) ?


Avoid using string ordering when you are dealing with timestamps. In this case, you should have page_max be a timestamp (possibly optional, since you have a "" as a base case; either use timestamp(0) or optional.none() to signal absent max, there are trade-offs in both cases).

efd6 · 2026-07-02T01:47:27Z

+          "events": {
+            "error": {
+              "code": string(resp.StatusCode),
+              "id": string(resp.Status),


Suggested change

"id": string(resp.Status),

"id": resp.Status,

efd6 · 2026-07-02T01:47:39Z

+                (size(resp.Body) != 0) ?
+                  string(resp.Body)
+                :
+                  string(resp.Status) + " (" + string(resp.StatusCode) + ")"


Suggested change

string(resp.Status) + " (" + string(resp.StatusCode) + ")"

resp.Status + " (" + string(resp.StatusCode) + ")"

efd6 · 2026-07-02T01:53:50Z

+                  :
+                    [],
+                  "want_more": next_offset != "",
+                  ?"offset": (next_offset != "") ? optional.of(next_offset) : optional.none(),


Suggested change

?"offset": (next_offset != "") ? optional.of(next_offset) : optional.none(),

"next": {

?"offset": next_offset,

},

with the change that at L97 you have

body.?next_offset.as(next_offset,

and changes to make that work.

efd6 · 2026-07-02T02:00:45Z

Note that there are existing title mismatches (presumably this integration was made as a derivation of the AbuseCH integration originally since the titles refer to that data source); also in ti_recordedfuture-554321f4-a649-49da-a5ce-b3dfef1a179b.json, and others.

Add identity detection data stream

d7fa68d

brijesh-elastic self-assigned this Jun 29, 2026

update changelog entry

b891b83

add user configurable option to hash the sensitive value

4d9dd47

brijesh-elastic marked this pull request as ready for review June 30, 2026 10:24

brijesh-elastic requested review from a team as code owners June 30, 2026 10:24

kcreddy reviewed Jun 30, 2026

View reviewed changes

address krishna's comment

3068074

brijesh-elastic requested a review from kcreddy June 30, 2026 13:19

vera-review-bot Bot reviewed Jun 30, 2026

View reviewed changes

Remove addition of 1s in cursor to prevent any missing events, fix ev…

d2049fe

…ent categorizations according to vera comments

efd6 reviewed Jul 2, 2026

View reviewed changes

	?"organization_id": state.?organization_id[0].hasValue() ? optional.of(state.organization_id) : optional.none(),
	?"organization_id": state.?organization_id[0],

	?"include_enterprise_level": state.?include_enterprise_level.orValue(false) ? optional.of(true) : optional.none(),
	?"include_enterprise_level": state.?include_enterprise_level,

	?"novel_only": state.?novel_only.orValue(false) ? optional.of(true) : optional.none(),
	?"novel_only": state.?novel_only,

	string(resp.Status) + " (" + string(resp.StatusCode) + ")"
	resp.Status + " (" + string(resp.StatusCode) + ")"

-                  ?"offset": (next_offset != "") ? optional.of(next_offset) : optional.none(),
+                  "next": {
+                    ?"offset": next_offset,
+                  },

Uh oh!

Conversation

brijesh-elastic commented Jun 29, 2026

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

github-actions Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Elastic Docs Style Checker (Vale)

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jun 29, 2026

🚀 Benchmarks report

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jun 30, 2026

Uh oh!

vera-review-bot Bot commented Jun 30, 2026

Uh oh!

vera-review-bot Bot commented Jun 30, 2026

Uh oh!

vera-review-bot Bot commented Jun 30, 2026

Vera Review Bot

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vera-review-bot Bot commented Jun 30, 2026

Uh oh!

vera-review-bot Bot Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

vera-review-bot Bot commented Jun 30, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jun 30, 2026

Uh oh!

vera-review-bot Bot commented Jun 30, 2026

Vera Review Bot

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jun 30, 2026

💚 Build Succeeded

History

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Jun 29, 2026 •

edited

Loading