Skip to content

Commit 8ff2d57

Browse files
romain-filigranromain-filigran
authored
feat(google-ti-feeds): enable IOC fetching for campaigns (#5793)
1 parent ac82854 commit 8ff2d57

4 files changed

Lines changed: 72 additions & 34 deletions

File tree

‎external-import/google-ti-feeds/README.md‎

Lines changed: 24 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ The **Google Threat Intelligence (GTI) Feeds Connector** ingests threat intellig
5151
| Collection | Config toggle | Enabled by default | Main OpenCTI entity | Related OpenCTI entity(s) produced |
5252
|-------------------|--------------------------------|--------------------|---------------------|------------------------------------------------------------------------------------------------------------------------------------|
5353
| Reports | `GTI_IMPORT_REPORTS` | ✅ Yes | Report | Location, Sector, Malware, Tool, Intrusion-Set, Attack-Pattern, Vulnerability, Indicator, Observable (Domain, File, IP, URL), Note |
54-
| Campaigns | `GTI_IMPORT_CAMPAIGNS` | ❌ No | Campaign | Location, Sector, Intrusion-Set, Malware, Attack-Pattern, Vulnerabilities, Tool |
54+
| Campaigns | `GTI_IMPORT_CAMPAIGNS` | ❌ No | Campaign | Location, Sector, Intrusion-Set, Malware, Attack-Pattern, Vulnerabilities, Tool, Indicator, Observable (Domain, File, IP, URL) |
5555
| Threat Actors | `GTI_IMPORT_THREAT_ACTORS` | ❌ No | Intrusion-Set | Location, Sector, Attack-Pattern, Malware, Vulnerabilities, Tool |
5656
| Malware Families | `GTI_IMPORT_MALWARE_FAMILIES` | ❌ No | Malware | Location, Sector, Intrusion-Set, Attack-Pattern, Vulnerabilities |
5757
| Software Toolkits | `GTI_IMPORT_SOFTWARE_TOOLKITS` | ❌ No | Tool | Location, Sector, Malware, Attack-Pattern |
@@ -108,21 +108,25 @@ Fetches Google TI **[Campaigns](https://gtidocs.virustotal.com/reference/campaig
108108

109109
The connector calls **`GET /collections`** with a filter on `collection_type:campaign` and `last_modification_date` to retrieve only campaigns modified since the last successful execution. The date is persisted in the connector state as `campaign_next_cursor_start_date`; on first run, it is calculated from `GTI_CAMPAIGN_IMPORT_START_DATE`.
110110

111-
For each campaign returned, the connector fetches related sub-entities by calling **`GET /collections/{campaign_id}/{subentity_type}`** for: `malware_families`, `threat_actors`, `attack_techniques`, `vulnerabilities`, and `software_toolkits`.
111+
For each campaign returned, the connector fetches related sub-entities by calling **`GET /collections/{campaign_id}/{subentity_type}`** for: `malware_families`, `threat_actors`, `attack_techniques`, `vulnerabilities`, `domains`, `files`, `urls`, `ip_addresses`, and `software_toolkits`.
112112

113113
All sub-entities are converted to STIX 2.1 objects and linked to the parent Campaign entity.
114114

115115
> **Note:** Relationships between a campaign and reports are not fetched here. These links are established when the `reports` collection is imported — reports fetch `campaigns` as one of their sub-entities, which creates the link in OpenCTI.
116116
117117
#### Sub-entities mapping
118118

119-
| Sub-entity type | OpenCTI entity produced |
120-
|-----------------------|-------------------------|
121-
| `malware_families` | Malware |
122-
| `threat_actors` | Intrusion-Set |
123-
| `attack_techniques` | Attack-Pattern |
124-
| `vulnerabilities` | Vulnerability |
125-
| `software_toolkits` | Tool |
119+
| Sub-entity type | OpenCTI entity produced |
120+
|-----------------------|--------------------------------------------|
121+
| `malware_families` | Malware |
122+
| `threat_actors` | Intrusion-Set |
123+
| `attack_techniques` | Attack-Pattern |
124+
| `vulnerabilities` | Vulnerability |
125+
| `domains` | Domain-Name observable + Indicator |
126+
| `files` | File observable + Indicator |
127+
| `urls` | URL observable + Indicator |
128+
| `ip_addresses` | IPv4-Addr/IPv6-Addr observable + Indicator |
129+
| `software_toolkits` | Tool |
126130

127131

128132
#### Configurable filters
@@ -475,13 +479,17 @@ These are **direct attributes** of the GTI Campaign object, but they are modeled
475479

476480
These relationships are **only produced when the Campaigns collection is enabled**, as they require fetching additional sub-entities via dedicated API calls.
477481

478-
| GTI Concept | STIX Source Object | STIX Relationship | STIX Target Object |
479-
|--------------------------|--------------------|---------------------|--------------------|
480-
| `threat_actors` | Campaign | `attributed-to` | Intrusion-Set |
481-
| `malware_families` | Campaign | `uses` | Malware |
482-
| `software_toolkits` | Campaign | `uses` | Tool |
483-
| `attack_techniques` | Campaign | `uses` | Attack-Pattern |
484-
| `vulnerabilities` | Campaign | `targets` | Vulnerability |
482+
| GTI Concept | STIX Source Object | STIX Relationship | STIX Target Object |
483+
|--------------------------|--------------------|---------------------|------------------------------------|
484+
| `threat_actors` | Campaign | `attributed-to` | Intrusion-Set |
485+
| `malware_families` | Campaign | `uses` | Malware |
486+
| `software_toolkits` | Campaign | `uses` | Tool |
487+
| `attack_techniques` | Campaign | `uses` | Attack-Pattern |
488+
| `vulnerabilities` | Campaign | `targets` | Vulnerability |
489+
| `domains` | Indicator | `related-to` | Campaign (Domain-Name + Indicator) |
490+
| `files` | Indicator | `related-to` | Campaign (File + Indicator) |
491+
| `urls` | Indicator | `related-to` | Campaign (URL + Indicator) |
492+
| `ip_addresses` | Indicator | `related-to` | Campaign (IPv4/IPv6 + Indicator) |
485493

486494
---
487495

‎external-import/google-ti-feeds/connector/src/custom/orchestrators/campaign/orchestrator_campaign.py‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -83,12 +83,12 @@ async def run(self, initial_state: dict[str, Any] | None) -> None:
8383
"attack_techniques",
8484
"vulnerabilities",
8585
"threat_actors",
86-
# "reports",
87-
# "domains",
88-
# "files",
89-
# "urls",
90-
# "ip_addresses",
86+
"domains",
87+
"files",
88+
"urls",
89+
"ip_addresses",
9190
"software_toolkits",
91+
# "reports",
9292
]
9393
try:
9494
async for gti_campaigns in self.client_api.fetch_campaigns(initial_state):

‎external-import/google-ti-feeds/tests/custom/debug_responses/main_campaigns_e3b0c44298fc1c14.json‎

Lines changed: 27 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -56,8 +56,28 @@
5656
"domains_count": 1,
5757
"malware_roles": [],
5858
"field_sources": [],
59-
"recent_activity_summary": [7, 7, 7, 14, 3, 2, 7, 6, 3, 6, 4, 3, 3, 5],
60-
"targeted_regions": ["US", "AU", "BE", "CA"],
59+
"recent_activity_summary": [
60+
7,
61+
7,
62+
7,
63+
14,
64+
3,
65+
2,
66+
7,
67+
6,
68+
3,
69+
6,
70+
4,
71+
3,
72+
3,
73+
5
74+
],
75+
"targeted_regions": [
76+
"US",
77+
"AU",
78+
"BE",
79+
"CA"
80+
],
6181
"targeted_regions_hierarchy": [
6282
{
6383
"region": "Oceania",
@@ -229,10 +249,10 @@
229249
],
230250
"campaign_type": "GLOBAL",
231251
"counters": {
232-
"files": 0,
252+
"files": 1,
233253
"domains": 1,
234254
"ip_addresses": 27,
235-
"urls": 0,
255+
"urls": 1,
236256
"iocs": 28,
237257
"subscribers": 1,
238258
"attack_techniques": 106
@@ -255,7 +275,9 @@
255275
"workarounds": [],
256276
"first_seen": 1692648309,
257277
"private": true,
258-
"alt_names": ["GLOBAL.24.007"],
278+
"alt_names": [
279+
"GLOBAL.24.007"
280+
],
259281
"vendor_fix_references": [],
260282
"threat_scape": [],
261283
"recent_activity_relative_change": 0.14925373134328357,

‎external-import/google-ti-feeds/tests/custom/test_orchestrator.py‎

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -358,22 +358,30 @@ def expected_campaign_log_messages() -> list[str]:
358358
"Fetched 1 attack_techniques relationships from API - {'prefix': '[BaseFetcher]'}",
359359
"Fetched 1 vulnerabilities relationships from API - {'prefix': '[BaseFetcher]'}",
360360
"Fetched 1 threat_actors relationships from API - {'prefix': '[BaseFetcher]'}",
361+
"Fetched 1 domains relationships from API - {'prefix': '[BaseFetcher]'}",
362+
"Fetched 1 files relationships from API - {'prefix': '[BaseFetcher]'}",
363+
"Fetched 1 urls relationships from API - {'prefix': '[BaseFetcher]'}",
364+
"Fetched 1 ip_addresses relationships from API - {'prefix': '[BaseFetcher]'}",
361365
"Fetched 1 software_toolkits relationships from API - {'prefix': '[BaseFetcher]'}",
362-
"Found relationships - {'prefix': '[OrchestratorCampaign]', 'current': 1, 'total': 1, 'relationships': 'malware_families: 1, attack_techniques: 1, vulnerabilities: 1, threat_actors: 1, software_toolkits: 1'}",
366+
"Found relationships - {'prefix': '[OrchestratorCampaign]', 'current': 1, 'total': 1, 'relationships': 'malware_families: 1, attack_techniques: 1, vulnerabilities: 1, threat_actors: 1, domains: 1, files: 1, urls: 1, ip_addresses: 1, software_toolkits: 1'}",
363367
"Using ID-only approach for attack techniques (quota optimization) - {'prefix': '[OrchestratorCampaign]', 'attack_technique_count': 1}",
364368
"Converted entities to STIX format - {'prefix': '[GenericConverter]', 'count': 34, 'entity_type': 'malware families'}",
365369
"Converted entities to STIX format - {'prefix': '[GenericConverter]', 'count': 53, 'entity_type': 'vulnerabilities'}",
366370
"Converted entities to STIX format - {'prefix': '[GenericConverter]', 'count': 54, 'entity_type': 'threat actors'}",
371+
"Converted entities to STIX format - {'prefix': '[GenericConverter]', 'count': 5, 'entity_type': 'domains'}",
372+
"Converted entities to STIX format - {'prefix': '[GenericConverter]', 'count': 5, 'entity_type': 'files'}",
373+
"Converted entities to STIX format - {'prefix': '[GenericConverter]', 'count': 5, 'entity_type': 'URLs'}",
374+
"Converted entities to STIX format - {'prefix': '[GenericConverter]', 'count': 5, 'entity_type': 'IP addresses'}",
367375
"Converted entities to STIX format - {'prefix': '[GenericConverter]', 'count': 2, 'entity_type': 'software toolkits'}",
368376
"Converted entities to STIX format - {'prefix': '[GenericConverter]', 'count': 2, 'entity_type': 'attack techniques'}",
369-
"Converted to STIX entities - {'prefix': '[OrchestratorCampaign]', 'current': 1, 'total': 1, 'entities_count': 176, 'entities_summary': 'location: 23, identity: 34, campaign: 1, relationship: 86, malware: 1, vulnerability: 1, software: 24, note: 3, intrusion-set: 1, tool: 1, attack-pattern: 1'}",
370-
"Adding items to batch processor - {'prefix': '[GenericBatchProcessor]', 'count': 176, 'display_name': 'STIX objects'}",
371-
"Successfully added items - {'prefix': '[GenericBatchProcessor]', 'added_count': 176, 'total_count': 176, 'display_name': 'STIX objects'}",
372-
"Flushing remaining items - {'prefix': '[GenericBatchProcessor]', 'count': 178, 'display_name': 'STIX objects'}",
373-
"Processing batch - {'prefix': '[GenericBatchProcessor]', 'batch_num': 1, 'batch_size': 178, 'display_name': 'STIX objects', 'total_processed': 178}",
377+
"Converted to STIX entities - {'prefix': '[OrchestratorCampaign]', 'current': 1, 'total': 1, 'entities_count': 196, 'entities_summary': 'location: 23, identity: 34, campaign: 1, relationship: 98, malware: 1, vulnerability: 1, software: 24, note: 3, intrusion-set: 1, domain-name: 1, indicator: 4, file: 1, url: 1, ipv4-addr: 1, tool: 1, attack-pattern: 1'}",
378+
"Adding items to batch processor - {'prefix': '[GenericBatchProcessor]', 'count': 196, 'display_name': 'STIX objects'}",
379+
"Successfully added items - {'prefix': '[GenericBatchProcessor]', 'added_count': 196, 'total_count': 196, 'display_name': 'STIX objects'}",
380+
"Flushing remaining items - {'prefix': '[GenericBatchProcessor]', 'count': 198, 'display_name': 'STIX objects'}",
381+
"Processing batch - {'prefix': '[GenericBatchProcessor]', 'batch_num': 1, 'batch_size': 198, 'display_name': 'STIX objects', 'total_processed': 198}",
374382
"Sent batch to OpenCTI - {'prefix': '[GenericBatchProcessor]', 'batch_num': 1}",
375-
"Batch completed successfully - {'prefix': '[GenericBatchProcessor]', 'work_id': None, 'total_count': 178, 'type_summary': 'identity: 35, marking-definition: 1, location: 23, campaign: 1, relationship: 86, malware: 1, vulnerability: 1, software: 24, note: 3, intrusion-set: 1, tool: 1, attack-pattern: 1'}",
376-
"Successfully processed batch #1. Total STIX objects sent: 178 - {'prefix': '[GenericBatchProcessor]', 'batch_num': 1, 'total_items_sent': 178}",
383+
"Batch completed successfully - {'prefix': '[GenericBatchProcessor]', 'work_id': None, 'total_count': 198, 'type_summary': 'identity: 35, marking-definition: 1, location: 23, campaign: 1, relationship: 98, malware: 1, vulnerability: 1, software: 24, note: 3, intrusion-set: 1, domain-name: 1, indicator: 4, file: 1, url: 1, ipv4-addr: 1, tool: 1, attack-pattern: 1'}",
384+
"Successfully processed batch #1. Total STIX objects sent: 198 - {'prefix': '[GenericBatchProcessor]', 'batch_num': 1, 'total_items_sent': 198}",
377385
"State update: Setting next_cursor_date - {'prefix': '[GenericBatchProcessor]', 'latest_date': '2025-07-30T22:01:05+00:00'}",
378386
]
379387

0 commit comments

Comments
 (0)