Add Sigma rule ID extraction to EntityExtractor#185
Conversation
…ule-id-extraction
The single-branch pattern applied the logsource-prefix whitelist even to explicitly labeled references, so sigma:lateral_movement_smb_admin_share was missed, while any two-token identifier starting with a whitelisted prefix (file_name, win_rate, net_income, apt_get) was extracted as a Sigma rule. Labeled forms now accept any snake_case rule name; bare IDs require a whitelisted prefix plus at least two more segments. The regex loop in extract_regex now flattens multi-group alternation matches. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TqYpmV8TABMzq27HBTLtNP
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…ule-id-extraction # Conflicts: # tests/test_basic.py
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…ata-suffix FPs Review feedback: the labeled form now accepts canonical Sigma YAML UUID rule ids (Sigma Rule: 929a690e-...), the bare-ID prefix whitelist gains SigmaHQ event-type filename prefixes (registry, dns, pipe, image, posh, wmi, driver), and bare IDs ending in generic metadata suffixes (aws_access_key_id, file_create_time, image_file_name) are excluded via lookbehinds. Regression tests for each. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TqYpmV8TABMzq27HBTLtNP
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 901c111ef5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "tool", | ||
| "campaign", | ||
| "attack_pattern", | ||
| "sigma_rule", |
There was a problem hiding this comment.
When users query text like Sigma Rule: <uuid> for a rule ingested through zettelforge.sigma.ingest, this new extractor emits sigma_rule:<uuid>. The ingest path persists detection KG edges from SigmaRule:<uuid> (src/zettelforge/sigma/entities.py builds from_type: "SigmaRule" and _persist_relations passes that through), and GraphRetriever looks up the exact (entity_type, entity_value), so traversal starts from a different node and misses the ingested rule relationships. Separately from the previously noted YAML-id extraction gap, the fresh evidence is that the lowercase entity type introduced here is incompatible with the existing SigmaRule endpoints.
Useful? React with 👍 / 👎.
Summary
Adds first-class
sigma_ruleextraction from CTI text so Sigma references are indexed as entities and carried throughremember()intonote.semantic.entities.Related issue
Linked automatically by the system.
Changes
Entity extractor
sigma_ruletoREGEX_PATTERNSin/src/zettelforge/entity_indexer.pysigma_ruletoENTITY_TYPESsigma:rule_nameSigma Rule: rule_nameprefix_segment_segment...) with Sigma-focused prefix constraints to reduce snake_case false positivesTest coverage (
tests/test_basic.py::TestEntityExtractor)sigma:apt28_cobalt_strikeSigma Rule: win_susp_powershell_encoded_cmdwin_susp_powershell_encoded_cmdsigma_ruleuser_id) are not extracted as Sigma rulesIntegration behavior
TestMemoryManager::test_remember_includes_sigma_rule_entitiesto verify extracted Sigma IDs are present innote.semantic.entitiesafterremember().Testing
pytest tests/ -v)ruff check src/zettelforge/)