|
3 | 3 | <img src="https://img.shields.io/pypi/v/iocx?logo=pypi&logoColor=white" alt="PyPI Version"> |
4 | 4 | </a> |
5 | 5 | <img src="https://img.shields.io/badge/coverage-97%25-brightgreen" alt="Coverage"> |
6 | | - <img src="https://img.shields.io/badge/tests-273_passed-brightgreen" alt="Tests"> |
| 6 | + <img src="https://img.shields.io/badge/tests-313_passed-brightgreen" alt="Tests"> |
7 | 7 | <img src="https://img.shields.io/badge/python-3.12-blue" alt="Python Version"> |
8 | | - <a href="https://github.com/malx-labs/malx-ioc-extractor/blob/main/LICENSE"> |
9 | | - <img src="https://img.shields.io/github/license/malx-labs/malx-ioc-extractor" alt="License"> |
| 8 | + <a href="https://github.com/iocx-dev/iocx/blob/main/LICENSE"> |
| 9 | + <img src="https://img.shields.io/github/license/iocx-dev/iocx" alt="License"> |
10 | 10 | </a> |
11 | | - <a href="https://github.com/malx-labs/malx-ioc-extractor/actions"> |
12 | | - <img src="https://img.shields.io/github/actions/workflow/status/malx-labs/malx-ioc-extractor/ci.yml?label=build" alt="Build Status"> |
| 11 | + <a href="https://github.com/iocx-dev/iocx/actions"> |
| 12 | + <img src="https://img.shields.io/github/actions/workflow/status/iocx-dev/iocx/ci.yml?label=build" alt="Build Status"> |
13 | 13 | </a> |
14 | 14 | <img src="https://img.shields.io/badge/v0.2.0_performance-1MB_in_0.0053s-brightgreen" alt="Performance"> |
15 | 15 | <img src="https://img.shields.io/badge/v0.2.0_throughput-~200MB%2Fs-brightgreen" alt="Throughput"> |
16 | 16 | <img src="https://img.shields.io/badge/v0.2.0_pathological_IPv6-0.0005s-brightgreen" alt="Pathological IPv6 Timing"> |
17 | 17 | </p> |
18 | 18 |
|
19 | | -# malx‑ioc‑extractor |
| 19 | +# IOCX — Static IOC Extraction for Binaries, Text, and Artifacts |
20 | 20 |
|
21 | | -**Static IOC extraction for binaries, text, and artifacts — fast, safe, and open‑source.** |
| 21 | +**Fast, safe, deterministic IOC extraction for DFIR, SOC automation, and large-scale threat analysis.** |
22 | 22 |
|
23 | | -malx‑ioc‑extractor is a lightweight, extensible engine for extracting Indicators of Compromise (IOCs) using pure static analysis. No execution. No sandboxing. No risk. |
24 | | -Built for DFIR workflows, SOC automation, and large‑scale threat analysis. |
| 23 | +IOCX is a lightweight, extensible engine for extracting Indicators of Compromise (IOCs) using **pure static analysis**. No execution. No sandboxing. No risk. |
25 | 24 |
|
26 | | -It’s designed to be: |
| 25 | +Built for: |
27 | 26 |
|
28 | | -- **Safe** — never executes untrusted code |
29 | | -- **Fast** — built for automation and pipelines |
30 | | -- **Extensible** — plug in your own regexes, parsers, and rules |
31 | | -- **Developer‑friendly** — clean API, CLI, and examples |
32 | | -- **Open‑source** — the extraction engine is free; enrichment lives in the MalX cloud platform |
| 27 | +- DFIR workflows |
| 28 | +- SOC automation |
| 29 | +- Threat-intel pipelines |
| 30 | +- CI/CD security checks |
| 31 | +- Large‑scale batch processing |
33 | 32 |
|
34 | 33 | This project is the foundation of the MalX Labs ecosystem for scalable, modern threat‑analysis tooling. |
35 | 34 |
|
36 | | -## Why malx‑ioc‑extractor? |
| 35 | +## Why IOCX? |
37 | 36 |
|
38 | | -malx‑ioc‑extractor is designed for environments where safety, determinism, and automation matter. While many IOC extractors operate only on raw text, malx‑ioc‑extractor includes binary‑aware static analysis and an extensible rule system, making it suitable for DFIR pipelines, CI systems, and high‑volume threat‑intel processing. |
| 37 | +IOCX is designed for environments where **safety, determinism, and automation** matter. Unlike extractors that operate only on raw text, IOCX includes **binary‑aware static analysis**, a **plugin-friendly rule system**, and a **stable JSON schema**. |
39 | 38 |
|
40 | | -**Key advantages** |
| 39 | +### Key advantages |
41 | 40 |
|
42 | | -- **Static‑only design** — no execution, no sandboxing, and no risk of running untrusted code |
43 | | -- **Binary parsing** — extracts indicators from Windows PE files in addition to raw text |
44 | | -- **Deterministic behaviour** — stable output and predictable performance, ideal for automated workflows |
45 | | -- **Extensible rule engine** — plug in custom detectors, parsers, and enrichment logic |
46 | | -- **Consistent JSON schema** — uniform output that integrates cleanly with SIEM, SOAR, and log pipelines |
47 | | -- **Low dependency footprint** — minimal attack surface and safe for enterprise environments |
48 | | -- **Designed for pipelines** — fast start‑up, fast throughput, and no heavyweight runtime requirements |
| 41 | +- **Static‑only design** — never executes untrusted code |
| 42 | +- **Binary parsing** — extracts IOCs from Windows PE files in addition to raw text |
| 43 | +- **Deterministic behaviour** — stable output and predictable performance, ideal for pipelines |
| 44 | +- **Extensible rule engine** — custom detectors, parsers, and plugins |
| 45 | +- **Consistent JSON schema** — clean integration with SIEM/SOAR |
| 46 | +- **Low dependency footprint** — safe for enterprise environments |
| 47 | +- **Pipeline-ready** — fast start‑up, fast throughput |
49 | 48 |
|
50 | | -## Use Cases |
| 49 | +--- |
| 50 | + |
| 51 | +## What IOCX *Is Not* |
| 52 | + |
| 53 | +To avoid confusion: |
51 | 54 |
|
52 | | -malx‑ioc‑extractor fits naturally into DFIR, security automation, and threat‑intelligence workflows. Typical usage patterns include: |
| 55 | +- Not a sandbox |
| 56 | +- Not a malware emulator |
| 57 | +- Not a behavioural analysis tool |
| 58 | +- Not an enrichment engine (that lives in the MalX Cloud platform) |
| 59 | + |
| 60 | +IOCX is **static extraction only**, by design. |
| 61 | + |
| 62 | +## Use Cases |
53 | 63 |
|
54 | 64 | ### SOC & Incident Response |
55 | | -- Extract indicators from suspicious emails, alerts, or analyst clipboard text |
56 | | -- Parse IOCs from incident reports and triage notes into structured JSON |
57 | | -- Safely inspect malware samples statically without executing anything |
| 65 | +- Extract indicators from emails, alerts, or analyst clipboard text |
| 66 | +- Parse IOCs from reports into structured JSON |
| 67 | +- Safely inspect malware samples without execution |
58 | 68 |
|
59 | 69 | ### Threat Intelligence Processing |
60 | | -- Normalize indicators from threat‑intel feeds |
61 | | -- Batch‑process dumps of unstructured text into machine‑readable IOC sets |
62 | | -- Build enrichment pipelines on top of the deterministic output format |
| 70 | +- Normalize indicators from feeds |
| 71 | +- Batch‑process unstructured text |
| 72 | +- Build enrichment pipelines on top of deterministic output |
63 | 73 |
|
64 | 74 | ### CI/CD & DevSecOps |
65 | | -- Scan new binaries for embedded indicators before publishing artifacts |
66 | | -- Integrate IOC extraction into automated security checks |
67 | | -- Detect accidental inclusion of URLs or addresses during build steps |
| 75 | +- Scan binaries for embedded indicators before publishing |
| 76 | +- Integrate IOC extraction into automated checks |
| 77 | +- Detect accidental inclusion of URLs or addresses in builds |
68 | 78 |
|
69 | 79 | ### Bulk Automation & Scripting |
70 | | -- Pipe logs, artifacts, or telemetry through malx‑ioc‑extractor to extract actionable indicators |
71 | | -- Use the Python API for batch workflows, ETL pipelines, or custom tooling |
72 | | -- Combine with rule extensions to tailor detection to internal patterns or datasets |
| 80 | +- Pipe logs or artifacts through IOCX |
| 81 | +- Use the Python API for ETL or batch workflows |
| 82 | +- Extend with custom detectors for internal patterns |
73 | 83 |
|
74 | | -## v0.3.0 — Stronger Architecture, New Crypto IOC Detection, Same Blazing Performance |
| 84 | +## Version Highlights |
75 | 85 |
|
76 | | -Version 0.3.0 is a foundational release that tightens the engine’s architecture, expands IOC coverage with Ethereum and Bitcoin wallet detection, and preserves the extractor’s extremely fast performance even on multi‑megabyte inputs. This release makes the engine more predictable, extensible, and resilient for the long-term extensibility. |
| 86 | +### v0.3.0 — Stronger Architecture, New Crypto IOC Detection |
77 | 87 |
|
78 | | -## v0.2.0 — High‑Reliability IP Detection in Hostile Data |
| 88 | +- Ethereum & Bitcoin wallet detection |
| 89 | +- Improved architecture for long-term extensibility |
| 90 | +- Same blazing performance on multi-MB inputs |
79 | 91 |
|
80 | | -Version 0.2.0 significantly improves IPv4/IPv6 extraction in noisy, malformed, mixed-content environments — the kind often seen in: |
| 92 | +### v0.2.0 — High‑Reliability IP Detection |
81 | 93 |
|
82 | | -- SIEM log lines |
83 | | -- network captures |
84 | | -- DFIR corpus samples |
85 | | -- pasted analyst dumps |
| 94 | +Significant improvements to IPv4/IPv6 extraction in noisy, malformed, mixed-content environments |
86 | 95 |
|
87 | | -### Real CLI Output (Chaos Corpus Sample) |
| 96 | +## Real CLI Output (Chaos Corpus Sample) |
88 | 97 |
|
89 | 98 | ```json |
90 | 99 | $ iocx chaos_corpus.json |
@@ -165,26 +174,21 @@ All measurements from the latest performance suite: |
165 | 174 |
|
166 | 175 | | Sample Type | Time | |
167 | 176 | |------------------------------|----------| |
168 | | -| IP | | |
169 | | -|==============================|==========| |
| 177 | +| **IP** | | |
170 | 178 | | 1 MB mixed‑content sample | 0.0070s | |
171 | 179 | | Pathological IPv6 blob | 0.0004s | |
172 | 180 | | 100 KB sample | 0.0008s | |
173 | 181 | | 300 KB sample | 0.0021s | |
174 | 182 | | 600 KB sample | 0.0038s | |
175 | 183 | | 1 MB sample | 0.0068s | |
176 | | -|------------------------------|----------| |
177 | | -| Filepath | | |
178 | | -|==============================|==========| |
| 184 | +| **Filepath** | | |
179 | 185 | | 1 MB mixed‑content sample | 0.0040s | |
180 | 186 | | Pathological deep unix path | 0.0237s | |
181 | 187 | | 300 KB sample | 0.0011s | |
182 | 188 | | 600 KB sample | 0.0022s | |
183 | 189 | | 1000 KB sample | 0.0038s | |
184 | 190 | | 1500 KB sample | 0.0055s | |
185 | | -|------------------------------|----------| |
186 | | -| Crypto | | |
187 | | -|==============================|==========| |
| 191 | +| **Crypto** | | |
188 | 192 | | 1 MB mixed‑content sample | 0.0021s | |
189 | 193 | | Pathological ETH-like blob | 0.0012s | |
190 | 194 | | 300 KB sample | 0.0006s | |
@@ -217,7 +221,7 @@ All measurements from the latest performance suite: |
217 | 221 | - Hashes (MD5 / SHA1 / SHA256 / SHA512 / Generic Hex) |
218 | 222 | - Email addresses |
219 | 223 | - Base64 |
220 | | -- Crypto wallets (Ethereum, Bitcoin) (new in v0.3.0) |
| 224 | +- Crypto wallets (Ethereum / Bitcoin) |
221 | 225 |
|
222 | 226 | ### Static PE Parsing |
223 | 227 |
|
@@ -338,64 +342,28 @@ print(results) |
338 | 342 | ## Architecture |
339 | 343 | ```plaintext |
340 | 344 |
|
341 | | -malx-ioc-extractor/ |
| 345 | +iocx/ |
342 | 346 | │ |
343 | 347 | ├── examples/ # Sample files + generators |
344 | | -├── tests/ # Unit and integration tests |
| 348 | +├── docs/ # Detector contracts, overlap suppression rules, and plugin authoring guidelines |
| 349 | +├── tests/ # Unit, integration, fuzz, robustness, and performance tests |
345 | 350 | ├── iocx |
346 | 351 | ├── detectors/ # Regex-based IOC detectors |
347 | 352 | ├── parsers/ # PE parsing, string extraction |
| 353 | + ├── plugins/ # Plugin API and registry |
348 | 354 | ├── cli/ # Command-line interface |
349 | 355 |
|
350 | 356 | ``` |
351 | 357 |
|
352 | 358 | The engine is intentionally modular so components can be extended or replaced easily. |
353 | 359 |
|
354 | | -## Extending the Engine |
355 | | - |
356 | | -You can add custom: |
357 | | - |
358 | | -- Regex detectors |
359 | | -- File parsers |
360 | | -- Normalisation logic |
361 | | - |
362 | | -### Register a custom detector |
363 | | -As an example, the following detector extracts the word "SECRET" from a string of text: |
364 | | -```python |
365 | | - |
366 | | -import re |
367 | | -from iocx.models import Detection |
368 | | -from iocx.detectors.registry import register_detector |
369 | | - |
370 | | -SECRET_REGEX = re.compile(r"\bSECRET\b") |
371 | | - |
372 | | -@register_detector("custom.secret") |
373 | | -def detect_secret(text: str): |
374 | | - results = [] |
375 | | - for m in SECRET_REGEX.finditer(text): |
376 | | - results.append( |
377 | | - Detection( |
378 | | - value=m.group(0), |
379 | | - start=m.start(), |
380 | | - end=m.end(), |
381 | | - category="custom.secret" |
382 | | - ) |
383 | | - ) |
384 | | - return results |
385 | | - |
386 | | -``` |
387 | | -Assuming the detector was saved as `my_detector.py`, consume the custom detector - if the detector fires, its results will be added to the list of detected IOCs: |
388 | | -```python |
389 | | - |
390 | | -from iocx.engine import Engine |
391 | | -import my_detector |
392 | | - |
393 | | -engine = Engine() |
| 360 | +## Extending IOCX |
394 | 361 |
|
395 | | -results = engine.extract("This is a SECRET message.") |
396 | | -print(results) |
| 362 | +See `docs/specs/` for: |
397 | 363 |
|
398 | | -``` |
| 364 | +- Detector contracts |
| 365 | +- Overlap suppression rules |
| 366 | +- Plugin authoring guidelines |
399 | 367 |
|
400 | 368 | ## Safe Testing (No Malware Required) |
401 | 369 |
|
@@ -423,14 +391,6 @@ See CONTRIBUTING.md for full guidelines. |
423 | 391 | If you discover a security issue, do not open a GitHub issue. |
424 | 392 | Please follow the instructions in SECURITY.md. |
425 | 393 |
|
426 | | -## Related Projects (MalX Labs) |
427 | | - |
428 | | -- malx-core — foundational primitives |
429 | | -- malx-utils — shared utilities |
430 | | -- malx-sandbox — dynamic analysis environment |
431 | | -- malx-forge — adversarial payload tooling |
432 | | -- malx-archive — research + PoCs |
433 | | - |
434 | 394 | ## License |
435 | 395 |
|
436 | 396 | Licensed under the MIT License. See LICENSE for details. |
0 commit comments