RFC-0001 OSINT Layer — scaffold (Phase 1–5) (#147)

* RFC-016 OSINT Layer: functional Phase 1 + Phase 2-5 stubs (amends #147)

Replaces the original Phase 1-5 scaffold commit (830156f) with an
implementation that integrates with the existing ontology validator and
ships working Phase 1 collectors with mocked tests.

What changed vs the original scaffold:

- ontology.py: Phase 1-5 entity / edge declarations, all using the
  from_types/to_types/cardinality shape that OntologyValidator actually
  validates against. The scaffold's "from": "*", "to": "A|B" syntax was
  invisible to the validator and silently waved every relation through.
- ontology.py: canonicalization helpers (asn, cidr, domain, ipv6, mx,
  port, url, web_title) used by collectors and tests.
- ontology.py: merge_into_global_ontology() — idempotent merge into core
  ENTITY_TYPES / RELATION_TYPES at import time, no schema migration.
- transform_registry.py: function-based register(metadata, fn) API,
  CollectorTuple NamedTuple, idempotent re-registration. Scaffold used a
  decorator pattern with no idempotency.
- collectors are sync (matches existing yara/sigma pipelines; the
  codebase has zero asyncio usage today). Scaffold used async def.
- Phase 1 collectors are functional with mocked tests:
    dns_collector  — A/AAAA/NS/MX with NXDOMAIN/Timeout absorbed
    whois_collector — domain via python-whois, IP via ipwhois RDAP
                       (graceful ImportError handling per AGENTS.OE
                        Override 4 — surface failures, no silent retry)
    cert_collector — crt.sh SAN enumeration, dedup, HTTP error → []
- Phase 1.5 collectors (bgp_collector, port_scanner) and Phase 2-5 stubs
  (Hunter, Holehe, Namechk, Wappalyzer, BuiltWith, Twitter, Hashtag,
  HIBP, Breach Directory) register their metadata so discovery works,
  return [] until their integration ships. port_scanner is gated behind
  ZETTELFORGE_OSINT_ACTIVE_SCAN=1.
- IPv6Address added to core ENTITY_TYPES (parity with IPv4Address).
- Top-level __init__.py: side-effect import of osint subpackage. The
  original scaffold's __init__ called __all__ += [...] before __all__
  was defined — would have raised NameError on import.
- Tests: 67 mocked tests in tests/test_osint_entities.py +
  tests/test_osint_collectors.py covering entity validation, edge
  validation, canonicalization, collector shape, registry dispatch.
- docs/rfcs/RFC-016-osint-layer.md: canonical RFC with Status block
  documenting the three deviations from the literal RFC text (single
  kg_nodes table, no CLI in this PR, sync collectors).
- SCOPING_DOC.md: Phase 1 planning notes.
- docs/rfc-0001-osint-layer.md (scaffold) removed; docs/rfcs/RFC-016
  is the canonical doc.

Verification:

- 67/67 osint tests pass (0.12s)
- 121/121 focused subset (basic + kg_edge_schema + extensions +
  consolidation + osint) pass (3.71s)
- ruff check src/zettelforge: clean
- ruff format --check src/zettelforge: clean
- Smoke import: 14 collectors registered, all entity/edge types merged.

Out of scope (deferred):

- Real BGPView / Wappalyzer / Hunter / etc. integrations (Phase 1.5+)
- Investigation workflow engine + state machine (Phase 4)
- Top-level zettelforge CLI (Phase 4 brings it alongside the workflow
  engine)
- Container packaging (RFC §10 already defers to vNext per docs/03)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(osint): install dnspython in CI; add coverage smoke tests

Addresses the two CI failures from ecb3f12:

- 3 dns_collector tests failed with ModuleNotFoundError: No module
  named 'dns'. The tests use dns.resolver.NXDOMAIN to construct realistic
  mocks; that requires dnspython to actually be installed even though the
  collector calls themselves are mocked. Adds a new [osint] optional
  dependency with dnspython, python-whois, and ipwhois (the runtime
  Phase 1 collectors need them anyway), and pulls it into [dev] so CI's
  pip install -e .[dev] resolves them.

- Total coverage was 66.74% vs 67% required (GOV-007). The OSINT layer
  added ~720 statements with most stub branches uncovered. Adds three
  parametrized tests over every registered collector:
    1. unsupported input returns a list (covers early-return)
    2. metadata is well-formed (covers TransformMetadata fields)
    3. every declared input_type is callable without raising
  That is 14 collectors x 3 tests = 42 new tests, all using mocked or
  short-circuited paths (no network, no API keys).

Local results:
- 109/109 osint tests pass (was 67; +42 from parametrized smoke tests)
- ruff check src/zettelforge/osint: clean
- OSINT package coverage: 70% (was untested for stubs)

No force-push needed; this is a follow-up commit on rfc/osint-layer-scaffold.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(osint/whois): swallow ipwhois IPDefinedError on reserved IPs

The new universal smoke test
test_every_collector_handles_each_declared_input_type[whois_collector]
exercised whois_collector with the canonical RFC 5737 documentation IP
192.0.2.1, which ipwhois treats as a poisoned input and raises
IPDefinedError before any network call. The collector wasn't catching
that exception, so the bare-bones probe call propagated it and CI failed
again on test (3.12) and test (3.13).

Real callers will hit the same exception any time an analyst feeds in a
private (10/8, 192.168/16), loopback (127/8), or documentation-net IP.
The right behaviour is fail-closed: log and return None / [] so the KG
doesn't pretend it has data it doesn't.

This commit catches IPDefinedError and BaseIpwhoisException in
_lookup_ip, logging the reserved-IP case at debug and the broader
failure case at warning. Per AGENTS.OE Override 4, the failure is
surfaced (not silently retried) — it just doesn't bubble as an
exception.

Local: 109/109 osint tests pass; ruff clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(osint): bgp edge direction; entity_resolver bug fixes from review

Addresses reviewer concerns on PR #147 that survived the amend (the rest
were against scaffold code my amend already replaced).

bgp_collector edge direction inverted:
The ontology declares part_of_as as Netblock -> ASNumber (with IP
families also valid as the from side). The collector was emitting
from_entity_type="ASNumber", to_entity_type="Netblock" which inverts
the declared direction. validate_relation would have rejected those
edges. Swap the from/to fields; the entity emitted is still the
Netblock (semantically: "given an ASN, here are netblocks that are
part_of_as that ASN"). Caught by Copilot review of the original
scaffold; the bug carried over into my amend.

entity_resolver canonicalise_ipv4 leading zeros:
ipaddress.IPv4Address("001.002.003.004") raises AddressValueError on
Python 3.10+. Replace with explicit octet parsing (split on '.',
int() each, validate 0-255). Raises ValueError on malformed input
instead of silently corrupting it.

entity_resolver canonicalise_asn hex strip:
Old behavior stripped non-digits, so "0x3039" became "AS03039" rather
than the expected hex-decoded value. Delegate to
zettelforge.osint.ontology.canonicalize_asn which uses int() and
raises ValueError on non-decimal input — fail-loud.

entity_resolver._canonical_key used "Domain" (typo, scaffold relic):
The core ontology uses DomainName. Fix to match. Also added IPv6Address
case (parity with IPv4Address).

entity_resolver late `from datetime import datetime` at module bottom:
Moved to module-top imports.

Same module: phone canonicalization preserved as-is (E.164 best-effort
heuristic; documented now). Netblock canonicalization upgraded to use
canonicalize_cidr so host bits get trimmed.

Local: 109/109 osint tests pass; ruff clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: rolandpg

SCOPING_DOC.md

@@ -110,6 +110,17 @@ crewai = [
     "crewai>=1.14.0",
 ]
 
+# RFC-016 OSINT layer runtime deps. Phase 1 collectors (DNS, WHOIS,
+# crt.sh) need these to function; tests need them to construct realistic
+# mocks (e.g. raise dns.resolver.NXDOMAIN). Optional so non-OSINT users
+# don't pay the cost. CI's [dev] install pulls these in via the line
+# below.
+osint = [
+    "dnspython>=2.4.0",
+    "python-whois>=0.9.0",
+    "ipwhois>=1.2.0",
+]
+
 dev = [
     "pytest>=7.0.0",
     "pytest-asyncio>=0.21.0",
@@ -121,6 +132,7 @@ dev = [
     "uvicorn>=0.20.0",      # for test_web_api.py
     "psutil>=5.9.0",        # for test_web_api.py
     "jinja2>=3.0.0",        # for test_web_api.py
+    "zettelforge[osint]",   # OSINT collectors need dnspython etc. for tests
 ]
 
 [project.urls]

docs/rfcs/RFC-016-osint-layer.md

@@ -22,6 +22,10 @@
     - OpenCTI integration
 """
 
+# RFC-016 OSINT layer (Phase 1, Infrastructure tier).
+# Side-effect import: merges OSINT entity / edge types into the global
+# ontology and registers Phase 1 collectors with TRANSFORM_REGISTRY.
+from zettelforge import osint
 from zettelforge.blended_retriever import BlendedRetriever
 from zettelforge.edition import (
     Edition,

pyproject.toml

@@ -207,6 +207,11 @@
         "optional": ["belongs_to_ref", "resolves_to_refs"],
         "properties": {},
     },
+    "IPv6Address": {
+        "required": ["value"],
+        "optional": ["belongs_to_ref", "resolves_to_refs"],
+        "properties": {},
+    },
     "DomainName": {
         "required": ["value"],
         "optional": ["resolves_to_refs"],

src/zettelforge/__init__.py

@@ -0,0 +1,80 @@
+"""
+ZettelForge OSINT layer (RFC-016 / RFC-0001).
+
+Importing the package merges the OSINT entity / edge types into the global
+ontology and imports each collector subpackage so collectors register with
+``TRANSFORM_REGISTRY`` at module load time.
+
+Phase 1 (Infrastructure) ships functional collectors (DNS, WHOIS, crt.sh)
+plus stubs for BGP and port scanning. Phases 2-5 ship as graceful stubs:
+each collector registers its metadata so callers can discover it, and
+returns ``[]`` until the underlying API integration lands.
+
+Public surface:
+
+- ``OSINT_ENTITY_TYPES`` / ``OSINT_RELATION_TYPES`` / ``ONTOLOGY`` — additive
+  ontology declarations.
+- ``TRANSFORM_REGISTRY`` — the singleton registry.
+- ``CollectorTuple`` — collector return-row shape.
+- ``TransformMetadata`` / ``TransformRegistry`` — types for adding new
+  collectors.
+- ``Investigation`` / ``EntityResolver`` — Phase 4 / Phase 1.5 utilities
+  (re-exported from their modules).
+"""
+
+from zettelforge.osint.ontology import (
+    ONTOLOGY,
+    OSINT_ENTITY_TYPES,
+    OSINT_RELATION_TYPES,
+    canonicalize_asn,
+    canonicalize_cidr,
+    canonicalize_domain,
+    canonicalize_ipv6,
+    canonicalize_mx,
+    canonicalize_port,
+    canonicalize_url,
+    canonicalize_web_title,
+    merge_into_global_ontology,
+)
+from zettelforge.osint.transform_registry import (
+    TRANSFORM_REGISTRY,
+    CollectorFn,
+    CollectorTuple,
+    TransformMetadata,
+    TransformRegistry,
+    get_transform_registry,
+)
+
+# Merge OSINT types into the global ontology before any collector runs.
+# Idempotent — safe under repeated imports (pytest, REPL re-imports, etc.).
+merge_into_global_ontology()
+
+# Trigger collector self-registration. Each subpackage's __init__ imports
+# the collector modules under it, and each module calls
+# ``TRANSFORM_REGISTRY.register(...)`` at import time.
+from zettelforge.osint.collectors import breach as _breach  # noqa: F401
+from zettelforge.osint.collectors import infrastructure as _infrastructure  # noqa: F401
+from zettelforge.osint.collectors import people as _people  # noqa: F401
+from zettelforge.osint.collectors import social as _social  # noqa: F401
+from zettelforge.osint.collectors import tech as _tech  # noqa: F401
+
+__all__ = [
+    "ONTOLOGY",
+    "OSINT_ENTITY_TYPES",
+    "OSINT_RELATION_TYPES",
+    "TRANSFORM_REGISTRY",
+    "CollectorFn",
+    "CollectorTuple",
+    "TransformMetadata",
+    "TransformRegistry",
+    "canonicalize_asn",
+    "canonicalize_cidr",
+    "canonicalize_domain",
+    "canonicalize_ipv6",
+    "canonicalize_mx",
+    "canonicalize_port",
+    "canonicalize_url",
+    "canonicalize_web_title",
+    "get_transform_registry",
+    "merge_into_global_ontology",
+]

src/zettelforge/ontology.py

@@ -0,0 +1,5 @@
+"""OSINT collector subpackages.
+
+Each tier (infrastructure, people, tech) is its own subpackage. Phase 1
+ships only ``infrastructure``.
+"""

src/zettelforge/osint/__init__.py

@@ -0,0 +1,17 @@
+"""
+Breach-data collectors (RFC-016 Phase 4 stubs).
+
+HaveIBeenPwned (k-anon password / breach lookup) and Breach Directory
+collectors. Both register their metadata at import time but return ``[]``
+until their integrations land.
+"""
+
+from zettelforge.osint.collectors.breach import (
+    breach_directory,
+    hibp_collector,
+)
+
+__all__ = [
+    "breach_directory",
+    "hibp_collector",
+]

src/zettelforge/osint/collectors/__init__.py

@@ -0,0 +1,46 @@
+"""
+Breach Directory collector — Phase 4 stub (RFC-016 §5).
+
+Looks up breach records via Breach Directory's API. Stub: requires
+``BREACH_DIRECTORY_API_KEY`` and returns ``[]`` without it. Phase 4
+ships the live lookup.
+"""
+
+from __future__ import annotations
+
+import os
+
+from zettelforge.log import get_logger
+from zettelforge.osint.transform_registry import (
+    TRANSFORM_REGISTRY,
+    CollectorTuple,
+    TransformMetadata,
+)
+
+_logger = get_logger("zettelforge.osint.collectors.breach_directory")
+
+API_KEY_ENV = "BREACH_DIRECTORY_API_KEY"
+
+
+def collect(input_entity_type: str, input_value: str) -> list[CollectorTuple]:
+    """Query Breach Directory for an EmailAddress. Stub: returns ``[]``."""
+    if input_entity_type != "EmailAddress":
+        return []
+    if not os.environ.get(API_KEY_ENV):
+        _logger.debug("breach_directory_no_api_key", env=API_KEY_ENV)
+        return []
+    # Phase 4: real Breach Directory call goes here. For now: fail closed.
+    return []
+
+
+_METADATA = TransformMetadata(
+    name="breach_directory",
+    description="Breach Directory: look up breach records for an email address.",
+    input_types=("EmailAddress",),
+    output_types=(),
+    api_dependencies=("breachdirectory.org",),
+    rate_limit=2.0,
+)
+
+
+TRANSFORM_REGISTRY.register(_METADATA, collect)

src/zettelforge/osint/collectors/breach/__init__.py

@@ -0,0 +1,46 @@
+"""
+HaveIBeenPwned collector — Phase 4 stub (RFC-016 §5).
+
+Looks up breach exposure for an email address via the HIBP API. Stub:
+requires ``HIBP_API_KEY`` and returns ``[]`` without it. Phase 4 will
+ship the live lookup and breach-record emission.
+"""
+
+from __future__ import annotations
+
+import os
+
+from zettelforge.log import get_logger
+from zettelforge.osint.transform_registry import (
+    TRANSFORM_REGISTRY,
+    CollectorTuple,
+    TransformMetadata,
+)
+
+_logger = get_logger("zettelforge.osint.collectors.hibp")
+
+API_KEY_ENV = "HIBP_API_KEY"
+
+
+def collect(input_entity_type: str, input_value: str) -> list[CollectorTuple]:
+    """Look up breaches associated with an EmailAddress. Stub: returns ``[]``."""
+    if input_entity_type != "EmailAddress":
+        return []
+    if not os.environ.get(API_KEY_ENV):
+        _logger.debug("hibp_collector_no_api_key", env=API_KEY_ENV)
+        return []
+    # Phase 4: real HIBP call goes here. For now: fail closed.
+    return []
+
+
+_METADATA = TransformMetadata(
+    name="hibp_collector",
+    description="HaveIBeenPwned: enumerate breach exposures for an email.",
+    input_types=("EmailAddress",),
+    output_types=(),
+    api_dependencies=("haveibeenpwned.com",),
+    rate_limit=2.0,
+)
+
+
+TRANSFORM_REGISTRY.register(_METADATA, collect)