Skip to content

Commit 2a67c9c

Browse files
malx-labsmalx-labs
authored
Merge pull request #25 from iocx-dev/feature/v0.7.1
Feature/v0.7.1 into main
2 parents da914c3 + cf9d493 commit 2a67c9c

152 files changed

Lines changed: 13456 additions & 282 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# **v0.7.1 — Heuristics Engine Expansion & Structural Analysis Improvements**
2+
**Released: 2026‑05‑??**
3+
4+
v0.7.1 delivers a major upgrade to IOCX’s **PE heuristics engine**, **extractor correctness**, and **adversarial‑input resilience**. This release introduces six new structural heuristics, broad extractor hardening, and a significantly expanded adversarial test suite — including **full adversarial coverage for every IOC category**.
5+
6+
---
7+
8+
# **Extractor Hardening**
9+
10+
This release strengthens multiple IOC extractors with improved correctness, boundary handling, and adversarial‑text resilience. Updates span the **bare domain**, **strict URL**, **crypto**, and **hash** extractors, plus improved **URL normalisation**.
11+
12+
## **Bare Domain Extractor**
13+
14+
### **Improvements**
15+
- Expanded **TLD allow‑list** (e.g., `.ly`, `.gg`, `.sh`, `.app`, `.dev`, `.xyz`, `.online`) for broader real‑world coverage.
16+
- Strengthened **BAD_TLD deny‑list** to prevent file extensions, config keys, and log fields from being misclassified as domains.
17+
- Refined **boundary detection** to reduce false positives in noisy or punctuation‑heavy text.
18+
- Added **punycode + IDN homoglyph analysis**, including Unicode decoding, script classification, and confusable‑character detection.
19+
- Improved regex structure for **stability and predictable linear performance**, eliminating pathological backtracking cases.
20+
21+
### **Impact**
22+
- Higher recall for legitimate domains across modern TLDs.
23+
- Significant reduction in false positives from filepaths, dotted identifiers, and structured logs.
24+
- Richer, homoglyph‑aware metadata for downstream analysis and phishing detection.
25+
26+
---
27+
28+
## **Strict URL Extractor**
29+
30+
### Improvements
31+
- Added support for `ftp`, `ftps`, and `sftp`.
32+
- RFC‑compliant **userinfo parsing** (`user:pass@host`).
33+
- Full **punycode** domain support.
34+
- Improved **IPv6** handling (including zone indices).
35+
- More robust host matching aligned with the updated domain extractor.
36+
- Cleaner separation of path/query/fragment parsing.
37+
38+
### Impact
39+
- More complete URL extraction.
40+
- Fewer truncated or malformed URLs.
41+
- Better handling of obfuscated or credential‑embedded URLs.
42+
43+
---
44+
45+
## **Crypto Extractor**
46+
47+
### Improvements
48+
- Added **full Base58Check validation** for Bitcoin:
49+
- Double‑SHA256 checksum verification.
50+
- Version‑byte validation (`0x00`, `0x05`).
51+
- Rejects malformed Base58 sequences.
52+
- Preserved Bech32/Taproot and ETH detection.
53+
54+
### Impact
55+
- Dramatic reduction in Base58 false positives.
56+
- Only cryptographically valid BTC addresses are extracted.
57+
58+
---
59+
60+
## **Hash Extractor**
61+
62+
### Improvements
63+
- Increased short‑hex minimum length from **8 → 10** characters.
64+
- Strict MD5/SHA1/SHA256/SHA512 detection unchanged.
65+
66+
### Impact
67+
- Fewer false positives from small hex tokens.
68+
- Behaviour remains aligned with adversarial fixtures.
69+
70+
---
71+
72+
## **URL Normalisation**
73+
74+
- `normalise_url()` now wraps `urlparse()` in safe error handling.
75+
- Malformed URLs return `None` instead of raising.
76+
77+
### Impact
78+
- More robust behaviour on adversarial URL input.
79+
- Prevents crashes during bulk extraction.
80+
81+
---
82+
83+
# **Heuristics Engine Expansion (PE Structural Analysis)**
84+
85+
To support the expanded adversarial PE corpus, v0.7.1 introduces **six new deterministic heuristics** for detecting malformed or inconsistent PE structures:
86+
87+
- **Section overlap detection**
88+
`_analyse_section_overlap`
89+
- **Section alignment validation**
90+
`_analyse_section_alignment`
91+
- **Optional‑header consistency checks**
92+
`_analyse_optional_header_consistency`
93+
- **Entrypoint → section mapping validation**
94+
`_analyse_entrypoint_mapping`
95+
- **Data‑directory anomaly detection**
96+
`_analyse_data_directory_anomalies`
97+
- **Import‑directory validity checks**
98+
`_analyse_import_directory_validity`
99+
100+
### Impact
101+
- Clearer, reason‑coded anomaly reporting.
102+
- No false positives on benign binaries.
103+
- Deterministic behaviour across malformed PE structures.
104+
105+
---
106+
107+
# **Added**
108+
109+
### **1. Full adversarial fixtures for *all* IOC categories**
110+
New adversarial string corpora added for:
111+
112+
- **crypto wallets** (BTC/ETH, reversed, embedded, noisy, base58‑adjacent)
113+
- **domains** (Unicode homoglyphs, mixed‑script lookalikes)
114+
- **URLs** (broken schemes, nested encodings, truncated fragments)
115+
- **IPs** (malformed IPv4/IPv6, concatenated segments, invalid scopes)
116+
- **filepaths** (MAX_PATH‑breaking Windows paths, malformed UNC prefixes)
117+
- **hashes** (near‑miss hex sequences, truncated digests)
118+
- **base64** (invalid padding, embedded noise, extremely long runs)
119+
- **emails** (Unicode variants, malformed local parts)
120+
121+
Each fixture includes a deterministic snapshot.
122+
123+
### **2. Expanded adversarial PE corpus**
124+
Fixtures include:
125+
126+
- broken RVAs
127+
- overlapping/misaligned sections
128+
- corrupted data directories
129+
- malformed import tables
130+
- invalid optional headers (PE32 & PE32+)
131+
- truncated Rich headers
132+
- packed‑lookalike binaries
133+
- franken‑PE hybrids
134+
135+
### **3. Heuristics engine upgrades**
136+
- New structural heuristics (see above)
137+
- Unified internal analysis structure (`sections` + `data_directories`)
138+
- Deterministic, JSON‑safe anomaly reporting
139+
140+
---
141+
142+
# **Fixed**
143+
144+
- Improved stability when parsing malformed or adversarial PE files.
145+
- More robust handling of malformed URLs during normalisation.
146+
147+
---
148+
149+
# **Notes**
150+
151+
- Updated snapshot for `heuristic_rich.full.exe` to reflect new heuristics.
152+
- Previous snapshot predated directory‑range and RVA‑validation logic.
153+
154+
---

‎Makefile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,7 @@ dev: $(STAMP_DEV)
8484
# ===========================
8585
.PHONY: test
8686
test: dev
87-
$(PYTHON) -m pytest -q -m "not integration and not fuzz and not robustness and not performance"
87+
$(PYTHON) -m pytest -q -m "not integration and not fuzz and not robustness and not performance and not contract"
8888

8989
# ----------------------------------------
9090
# Integration tests only
@@ -132,7 +132,7 @@ test-coverage: dev
132132
.PHONY: test-contract
133133
test-contract: dev
134134
@echo "Running contract tests..."
135-
$(PYTEST) -m contract $(CONTRACT_DIR)
135+
$(PYTEST) -m contract $(CONTRACT_DIR) -sv
136136

137137
# ----------------------------------------
138138
# Static analysis and SCA

‎README-pypi.md‎

Lines changed: 31 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,37 @@ IOCX is a fast, safe, deterministic engine for extracting Indicators of Compromi
2424

2525
It performs **pure static analysis** — no execution, no sandboxing, no risk.
2626

27-
## What's new in v0.7.0
27+
## What's new in v0.7.1
28+
29+
### **Bare Domain Extractor Overhaul**
30+
- Expanded **TLD allow‑list** and strengthened **BAD_TLD deny‑list**
31+
- Refined boundary rules to reduce false positives in noisy text
32+
- Added **punycode decoding**, Unicode script classification, and homoglyph/confusable detection
33+
- Hardened regex for **predictable linear performance** under adversarial input
34+
- New metadata fields:
35+
- `punycode`, `punycode_decodes_to_unicode`
36+
- `decoded_unicode`
37+
- `contains_confusables`
38+
- `script`
39+
40+
### **Performance guarantees**
41+
- **~150-300 MB/s** for individual detectors (domains, crypto, filepaths, IPs)
42+
- **Strict linear scaling** across all detectors
43+
- Pathological punycode, IPv6, and filepath inputs complete in **< 15 ms**
44+
- End‑to‑end engine throughput: **20-30 MB/s**
45+
46+
### **Heuristic engine and adversarial fixture expansion**
47+
- Deterministic section overlap and alignment, optional header consistency, entrypoint mapping, data directory anomalies, and import directory validity heuristics
48+
- Adversarial fixtures covering all new heuristics and IOC subsystems.
49+
50+
### **Documentation updates**
51+
- New adversarial appendices
52+
- New Performance guarantees
53+
- Expanded schema‑contract guidance
54+
55+
## Recent changes
56+
57+
### v0.7.0
2858

2959
- **Deterministic heuristic engine**
3060

@@ -46,8 +76,6 @@ Deep hex‑encoding of nested byte structures prevents JSON serialization failur
4676

4777
New appendices and deterministic‑output guidance.
4878

49-
## Recent changes
50-
5179
### v0.6.0
5280

5381
- Stable JSON schema across all analysis levels

0 commit comments

Comments
 (0)