docs: add 15 new documentation files — Sigma/YARA, MCP, detection rules, extensions, editions, integrations

Adds 15 new documentation files across how-to guides and reference docs. All checks passed. See PR for full file list.

Author: rolandpg

docs/how-to/build-extensions.md

@@ -0,0 +1,199 @@
+---
+title: "Build an Extension Package"
+description: "Create an optional extension package for ZettelForge that provides an alternative backend (TypeDB), an integration (OpenCTI), or an operational feature (multi-tenant auth)."
+diataxis_type: "how-to"
+audience: "Python developers extending ZettelForge with optional packages"
+tags: [extensions, enterprise, packages, development, optional-features]
+last_updated: "2026-04-27"
+version: "2.6.0"
+---
+
+# Build an Extension Package
+
+ZettelForge discovers installed extension packages at startup via `zettelforge.extensions.load_extensions()`. An extension is any Python package that registers itself under the `zettelforge.extensions` namespace or is importable as `zettelforge_enterprise`.
+
+## Prerequisites
+
+- ZettelForge installed (`pip install zettelforge`)
+- Python 3.12+
+- For enterprise features: separate `zettelforge-enterprise` package (not distributed on PyPI)
+
+## How Extensions Are Loaded
+
+The extension loader in `zettelforge.extensions` follows a two-check discovery:
+
+1. **Try importing `zettelforge_enterprise`** -- if the package is installed, it is loaded as the `"enterprise"` extension.
+2. **Legacy env var fallback** -- if no package was found, check `THREATENGRAM_LICENSE_KEY`. If it matches the `TG-XXXX-XXXX-XXXX-XXXX` pattern, a marker is stored so `has_extension("enterprise")` returns `True`.
+
+```python
+from zettelforge.extensions import load_extensions, has_extension, get_extension
+
+load_extensions()
+print(has_extension("enterprise"))  # True or False
+```
+
+The loader is idempotent -- subsequent calls return the cached result without re-scanning the environment.
+
+## Steps
+
+### 1. Name your package
+
+Use the `zettelforge_` prefix to keep naming consistent and avoid collisions:
+
+- `zettelforge_enterprise` -- enterprise features (TypeDB, OpenCTI, telemetry)
+- `zettelforge_myfeature` -- your custom feature
+
+### 2. Create the package structure
+
+```
+zettelforge-myfeature/
+  pyproject.toml
+  src/
+    zettelforge_myfeature/
+      __init__.py
+      feature.py
+```
+
+The `__init__.py` can be empty -- the extension loader only needs the package to be importable.
+
+### 3. Register as a ZettelForge extension (optional)
+
+If you want your extension to be discoverable beyond the `zettelforge_enterprise` naming convention, register via a plugin entry point in `pyproject.toml`:
+
+```toml
+[project.entry-points."zettelforge.extensions"]
+myfeature = "zettelforge_myfeature"
+```
+
+Then consumers can check for it by name:
+
+```python
+from zettelforge.extensions import has_extension
+
+if has_extension("myfeature"):
+    # activate custom behaviour
+```
+
+### 4. Respect the edition API
+
+Use the `zettelforge.edition` module to gate features behind the active edition:
+
+```python
+from zettelforge.edition import is_enterprise, EditionError
+
+if not is_enterprise():
+    raise EditionError("This feature requires ZettelForge Enterprise")
+```
+
+Available edition functions:
+
+| Function | Returns | Description |
+|:---------|:--------|:------------|
+| `is_enterprise()` | `bool` | True if enterprise extensions are loaded |
+| `is_community()` | `bool` | True if no enterprise extensions |
+| `get_edition()` | `Edition` | `Edition.ENTERPRISE` or `Edition.COMMUNITY` |
+| `edition_name()` | `str` | `"ZettelForge + Extensions"` or `"ZettelForge"` |
+
+### 5. Expose extension features
+
+Your extension package should provide the actual feature implementations. The `get_extension()` function lets core code access your extension module:
+
+```python
+from zettelforge.extensions import get_extension
+
+enterprise = get_extension("enterprise")
+if enterprise is not None:
+    # Access TypeDB backend, OpenCTI sync, telemetry, etc.
+    enterprise.register_backends()
+```
+
+### 6. Test your extension
+
+Use the `reset_extensions()` function in setup/teardown to clear cached state between tests:
+
+```python
+import os
+from unittest.mock import patch
+from zettelforge.extensions import load_extensions, has_extension, reset_extensions
+
+def test_extension_loaded():
+    reset_extensions()
+    # Simulate having the enterprise package
+    with patch.dict("sys.modules", {"zettelforge_enterprise": __import__("types")}):
+        load_extensions()
+        assert has_extension("enterprise") is True
+
+
+def test_extension_not_loaded():
+    reset_extensions()
+    # Simulate missing package
+    with patch.dict("sys.modules", {"zettelforge_enterprise": None}):
+        load_extensions()
+        assert has_extension("enterprise") is False
+
+
+def test_legacy_env_var_activates():
+    reset_extensions()
+    os.environ["THREATENGRAM_LICENSE_KEY"] = "TG-1234-5678-9abc-def0"
+    with patch.dict("sys.modules", {"zettelforge_enterprise": None}):
+        load_extensions()
+        assert has_extension("enterprise") is True
+
+
+def test_invalid_env_var_does_not_activate():
+    reset_extensions()
+    os.environ["THREATENGRAM_LICENSE_KEY"] = "invalid-key"
+    with patch.dict("sys.modules", {"zettelforge_enterprise": None}):
+        load_extensions()
+        assert has_extension("enterprise") is False
+
+
+def test_get_missing_returns_none():
+    reset_extensions()
+    with patch.dict("sys.modules", {"zettelforge_enterprise": None}):
+        assert get_extension("enterprise") is None
+```
+
+### 7. Use the optional-feature pattern for SDK dependencies
+
+If your extension depends on an optional SDK (e.g., `typedb-client`, `pycti`), follow the optional-feature pattern:
+
+```python
+class MyFeature:
+    def __init__(self):
+        self._sdk = None
+        self._lock = threading.Lock()
+
+    def _ensure_loaded(self):
+        if self._sdk is not None:
+            return
+        with self._lock:
+            if self._sdk is not None:
+                return
+            try:
+                import typedb  # lazy import
+            except ImportError as exc:
+                raise ImportError(
+                    "TypeDB feature requires typedb-client. "
+                    "Install with: pip install zettelforge-enterprise"
+                ) from exc
+            self._sdk = typedb
+```
+
+This ensures core ZettelForge never depends on your SDK, and the error surfaces only at the point of use.
+
+## LLM Quick Reference
+
+**Task**: Create a ZettelForge extension package.
+
+**Key functions**: `load_extensions()` (idempotent discovery), `has_extension(name)` (boolean check), `get_extension(name)` (module or None), `reset_extensions()` (test cleanup).
+
+**Edition module**: `is_enterprise()`, `is_community()`, `get_edition()`, `edition_name()` let core code gate features behind edition.
+
+**Activation paths**: Package import (`zettelforge_enterprise`) takes priority. Legacy env var (`THREATENGRAM_LICENSE_KEY=TG-XXXX-XXXX-XXXX-XXXX`) is the fallback for backward compatibility.
+
+**Test pattern**: `reset_extensions()` in setup, `patch.dict("sys.modules", ...)` to control whether the package exists, `patch.dict(os.environ, ...)` for env var tests.
+
+**Optional SDK pattern**: Lazy-import the SDK in a private `_ensure_loaded()` method. Never import at module level. Surface a clear `ImportError` with install instructions.
+
+**Entry point registration**: Add `[project.entry-points."zettelforge.extensions"]` in pyproject.toml for discovery by name beyond the `zettelforge_enterprise` convention.

docs/how-to/configure-sigma-ingestion.md

@@ -0,0 +1,183 @@
+---
+title: "Configure Sigma Rule Ingestion"
+description: "Ingest Sigma detection rules into ZettelForge memory with automatic entity extraction, tag resolution, and knowledge graph population."
+diataxis_type: "how-to"
+audience: "Detection engineers, SOC analysts integrating Sigma rules into ZettelForge"
+tags: [sigma, ingestion, detection-rules, knowledge-graph, sigmahq]
+last_updated: "2026-04-27"
+version: "2.7.0"
+---
+
+# Configure Sigma Rule Ingestion
+
+Ingest Sigma detection rules (SigmaHQ format) into ZettelForge memory. Each rule is parsed, validated against the vendored SigmaHQ JSON schema, mapped to a `SigmaRule` entity with typed knowledge graph relations, and persisted as a memory note.
+
+## Prerequisites
+
+- ZettelForge installed (`pip install zettelforge`)
+- Sigma rule files in `.yml` or `.yaml` format (SigmaHQ specification V2.0.0)
+- Embedding and LLM models available (download automatically on first use)
+
+## Steps
+
+### 1. Use the CLI (quick start)
+
+Dry-run a directory to validate rules before ingesting:
+
+```bash
+python -m zettelforge.sigma.ingest /path/to/sigma/rules/ --dry-run
+```
+
+Output shows each parsed rule with its id, rule type, tag count, and relation count:
+
+```
+OK  /path/to/sigma/rules/proc_creation_win_whoami.yml  id=sigma_a1b2c3d4e5f67890  type=detection  tags=3  edges=6
+OK  /path/to/sigma/rules/susp_ps_execution.yml  id=55043c5f-3c72-4fb6-aa22-70b6f7e98d4a  type=detection  tags=5  edges=9
+
+Dry-run summary: 2/2 parsed, 0 failed.
+```
+
+Live ingestion into a MemoryManager:
+
+```bash
+python -m zettelforge.sigma.ingest /path/to/sigma/rules/ --domain detection
+```
+
+### 2. Use the Python API
+
+```python
+from zettelforge import MemoryManager
+from zettelforge.sigma import ingest_rule, ingest_rules_dir
+
+mm = MemoryManager()
+
+# Ingest a single file
+note, relations = ingest_rule(
+    "/path/to/rule.yml",
+    mm,
+    domain="detection",
+)
+print(f"Ingested: {note.id}, relations: {len(relations)}")
+
+# Ingest a directory (walks recursively)
+ingested, skipped = ingest_rules_dir(
+    "/path/to/sigma/rules/",
+    mm,
+    glob="**/*.yml",
+    domain="detection",
+)
+print(f"Ingested: {ingested}, skipped: {skipped}")
+```
+
+### 3. Parse without persisting
+
+For validation pipelines or custom workflows, parse rules without memory storage:
+
+```python
+from zettelforge.sigma import parse_file, parse_yaml, from_rule_dict
+
+# Parse from file
+rule_dict = parse_file("rule.yml")
+
+# Parse from YAML string
+yaml_text = """
+title: Suspicious Whoami Execution
+logsource:
+  category: process_creation
+  product: windows
+detection:
+  selection:
+    Image|endswith: '\\whoami.exe'
+  condition: selection
+"""
+rule_dict = parse_yaml(yaml_text)
+
+# Map to entity and KG relations
+entity, relations = from_rule_dict(rule_dict)
+print(f"Rule: {entity.title}")
+print(f"Logsource: product={entity.logsource_product}, category={entity.logsource_category}")
+print(f"Relation types: {set(r['rel'] for r in relations)}")
+```
+
+### 4. Accept multiple input types
+
+`ingest_rule()` accepts a parsed dict, raw YAML string, or `Path`:
+
+```python
+# Dict (pre-parsed)
+note, rels = ingest_rule(rule_dict, mm)
+
+# YAML string
+note, rels = ingest_rule(yaml_text, mm)
+
+# Path object
+from pathlib import Path
+note, rels = ingest_rule(Path("rule.yml"), mm)
+
+# File path as string (auto-detected if it looks like a path)
+note, rels = ingest_rule("rule.yml", mm)
+```
+
+### 5. Validate against the Sigma schema
+
+Run standalone validation without entity mapping:
+
+```python
+from zettelforge.sigma import validate, parse_yaml, SigmaValidationError
+
+rule = parse_yaml(yaml_text)
+result = validate(rule)
+if not result.valid:
+    for error in result.errors:
+        print(f"  Validation error: {error}")
+```
+
+### 6. Understand idempotency
+
+Re-ingesting an unchanged rule returns the original note. The `source_ref` follows the pattern `sigma:<rule_id>:<content_sha256_prefix>` and is checked before any write:
+
+```python
+first, _ = ingest_rule("rule.yml", mm)
+second, _ = ingest_rule("rule.yml", mm)
+assert first.id == second.id  # same note, no duplicate
+```
+
+### 7. CLI flags reference
+
+```
+usage: python -m zettelforge.sigma.ingest [-h] [--domain DOMAIN] [--dry-run] [--glob GLOB] path
+
+positional arguments:
+  path              Sigma rule file or directory
+
+options:
+  --domain DOMAIN   Memory domain for ingested notes (default: detection)
+  --dry-run         Parse + validate + map without persisting to memory
+  --glob GLOB       Glob used when path is a directory (default: **/*.yml)
+```
+
+## LLM Quick Reference
+
+**Task**: Ingest Sigma rules into ZettelForge memory with automatic knowledge graph population.
+
+**Primary CLI**: `python -m zettelforge.sigma.ingest <path>` with optional `--dry-run`, `--domain`, `--glob`.
+
+**Primary Python API**: `ingest_rule(source, mm, domain="detection")` returns `(MemoryNote, relations_list)`. Accepts dict, string, or Path. `ingest_rules_dir(path, mm)` walks a directory tree and returns `(ingested_count, skipped_count)`.
+
+**Pipeline**: Input -> `parse_file()` or `parse_yaml()` (YAML load + JSON-schema validation against vendored SigmaHQ schemas) -> `from_rule_dict()` (map to `SigmaRule` entity + relations) -> `mm.remember()` (persist as memory note) -> KG edge persistence.
+
+**Validation**: `validate(rule_dict)` returns `ValidationResult(valid, errors)`. Two error types: `SigmaParseError` (bad YAML or I/O) and `SigmaValidationError` (schema violation). Both bubble through `ingest_rule()`.
+
+**Schema dispatch**: Detection rules validate against `sigma-detection-rule-schema.json`. Rules with a `correlation:` key use the correlation schema. Rules with a `filter:` key use the filters schema.
+
+**Tag resolution**: Sigma tags (`attack.t1059`, `cve.2021-44228`) upgrade to typed KG edges (`detects` -> `AttackPattern`, `references_cve` -> `Vulnerability`, `attributed_to` -> `IntrusionSet`/`Malware`). Raw `tagged_with` -> `SigmaTag` edges are always preserved alongside upgrade edges. `tlp.*` and `detection.*` tags are metadata-only (no upgrade).
+
+**Logsource edges**: Every populated logsource facet (product, service, category) generates an `applies_to` -> `LogSource` edge with the facet type and value in properties.
+
+**Related rule edges**: The `related:` block maps to `superseded_by` (type: `obsolete`) or `related_to` (all other types).
+
+**Idempotency**: Source ref pattern `sigma:<rule_id>:<content_sha256[:12]>`. A store lookup precedes every write. Unchanged rules return the original note.
+
+**Security**: File size capped at 1 MB. Symlinks are never followed during directory walk. Paths that resolve outside the rules root are skipped.
+
+**Edge tagging**: All KG edges emitted during ingest carry `edge_type: detection` and `source: sigma_ingest` properties for downstream filtering.