Corpus reference enrichment: cross-corpus law linking, authority corpora, corpus-scoped analyzers

.gitignore

@@ -211,3 +211,9 @@ docs/superpowers/
 
 # Superpowers brainstorming scratch files
 .superpowers/
+
+# Demo corpus-import zips (large binaries, staged locally from ~/Code/EDGARx2)
+demo/import_zips/
+
+# Local-only governance-graph demo (kept on disk, excluded from PRs)
+/demo/

changelog.d/corpus-reference-enrichment.added.md

@@ -0,0 +1,68 @@
+- **Corpus reference enrichment agent.** A corpus-scoped agent tool pair that
+  crawls a corpus, inventories explicit references, and persists them — proven
+  on a real 55-document S-1 corpus (348 references created).
+  - New deterministic engine in `opencontractserver/enrichment/`
+    (`extractor.py`, `resolver.py`, `writer.py`, `services/`): regex/grammar
+    extraction of law citations (`Section 145 of the Delaware General
+    Corporation Law` → canonical key `dgcl:145`), document/exhibit references,
+    internal section references, and defined-term definition sites
+    (`(the "Company")` / `"Change of Control" means …` → `term:company` /
+    `term:change-of-control`, opt-in and capped); resolution to in-corpus target
+    documents and OC_SECTION annotations.
+  - New `CorpusReference` model (`opencontractserver/annotations/models.py`,
+    migration `annotations/0078_corpusreference`): a first-class
+    cross-document / cross-corpus / external-law connection with an indexed
+    `canonical_key` join key (`target_corpus` anticipates future cross-corpus
+    linking). Within-document section links continue to use `Relationship`;
+    every reference also gets a mention `Annotation` (law payloads stored in
+    `Annotation.data`, resolved document links in `Annotation.link_url` as
+    in-app site-relative paths).
+  - Two agent tools (`opencontractserver/llms/tools/core_tools/corpus_references.py`,
+    registered in `tool_registry.py`): `scan_corpus_references` (read-only
+    inventory) and `apply_corpus_reference_enrichment`
+    (`requires_approval=True`, `requires_write_permission=True`) — the
+    CAML-style scan → approval-gated apply pattern. Enrichment is idempotent.
+  - Read-only GraphQL `corpusReferences(corpusId)` query
+    (`config/graphql/annotation_types.py`, `annotation_queries.py`), visibility
+    scoped to corpus READ via `CorpusReferenceService`.
+  - Resolved document/exhibit references additionally roll up to
+    `DocumentRelationship` rows (`enrichment/writer.py`), feeding the corpus
+    document graph. Mention dedup is by (document, label, span start) so a
+    growing alias registry cannot duplicate mentions.
+- **Authority corpora + cross-corpus law linking.**
+  - `opencontractserver/enrichment/authorities.py`: `AuthorityCorpusBootstrapper`
+    materialises statute sections as keyed text documents
+    (`Document.custom_meta.canonical_key`, e.g. `dgcl:145`), idempotently
+    (skip / version-up on amendment / self-healing restamp after concurrent
+    pipeline saves). `find_authority_target` resolves citations with
+    subsection→section fallback (`dgcl:122(17)` → `dgcl:122`), visibility- and
+    current-version-aware.
+  - `EnrichmentService.link_external_references` upgrades EXTERNAL law
+    citations to RESOLVED cross-corpus links (`target_corpus`/`target_document`
+    + in-app `link_url` on the mention); runs automatically inside `apply()`
+    and is re-runnable as new authority corpora appear.
+  - Data-driven authority alias registry: `authority_alias_registry(user)`
+    merges static defaults with `custom_meta.authority_aliases` declared by
+    authority corpora — adding a body of law is a bootstrap call, zero code.
+- **Citation grammar coverage** (`enrichment/extractor.py`): suffix form
+  ("Section 145 of the DGCL"), prefix form ("Securities Act Section 4(a)(5)"),
+  statute-internal relative form ("§ 251 of this title", keyed via the
+  document's own authority context), and bare SEC rules ("Rule 506(b)",
+  "Rule 144A", "Rule 10b-5" → `sec-rule:*`).
+- **Analyzer framework: corpus-scoped analyzers.**
+  - New `@corpus_analyzer_task` decorator (`shared/decorators.py`) — the
+    corpus-scoped sibling of `@doc_analyzer_task`: runs once per Analysis,
+    owns its own writes, wrapper manages the Analysis lifecycle
+    (RUNNING/COMPLETED/FAILED, timestamps, result/error messages).
+  - `get_corpus_analyzer_task_by_name` / `get_analyzer_task_by_name`
+    (`utils/celery_tasks.py`); `run_task_name_analyzer` dispatches
+    corpus-scoped analyzers as a single task (no per-doc fan-out); analyzer
+    auto-sync and the `analyzer.W001` check cover both flavours.
+  - Enrichment adapter task
+    (`opencontractserver/tasks/corpus_analysis_tasks.py`) registers the engine
+    as a real task-based Analyzer (dispatchable via `CorpusAction`), attaching
+    to the framework-created Analysis instead of creating its own.
+- **Demo pipeline** (`demo/`, untracked zips excluded): authority seed JSONs
+  with real statutory text (DGCL, Securities Act, Exchange Act, IRC, ICA,
+  SEC Rules — 17 C.F.R.), bootstrap/import/export scripts, and a standalone
+  D3 governance-graph visualization (`governance_graph.html`).

changelog.d/corpusvote-index-state-reconcile.fixed.md

@@ -0,0 +1,8 @@
+- **Reconciled `CorpusVote` migration state with the model** (migration
+  `corpuses/0057`): migration `0049` created the three `CorpusVote` indexes
+  with explicit names (`corpuses_co_corpus__vote_idx`, …) while the model
+  declares them unnamed (Django auto-hashed names), and the `backend_lock`
+  field's `db_index=True` was missing from migration state. The drift made
+  `makemigrations --check` fail on every branch. Operations are three
+  Postgres `ALTER INDEX … RENAME` calls plus one index-state alter — no data
+  changes.

changelog.d/enrichment-review-fixes.fixed.md

@@ -0,0 +1,18 @@
+- **Enrichment mention links no longer 404** — the writer and
+  `link_external_references` wrote `link_url` as `/corpus/{id}/document/{id}`,
+  a shape no frontend route serves (it falls into the `*` catch-all → 404).
+  They now emit the canonical slug path
+  `/d/{corpus.creator.slug}/{corpus.slug}/{document.slug}` via the new
+  `opencontractserver/utils/frontend_paths.py::document_in_corpus_path`
+  helper (mirrors the frontend's `buildCanonicalPath`); cross-corpus law
+  links point into the authority corpus. Links with missing slugs are
+  skipped rather than written broken.
+- **`@corpus_analyzer_task` no longer marks a retrying Analysis FAILED**
+  (`opencontractserver/shared/decorators.py`) — `celery.exceptions.Retry`
+  extends `Exception`, so the wrapper's failure branch stamped
+  `Analysis.status=FAILED` before the retry ran. `Retry` is now re-raised
+  untouched, mirroring `doc_analyzer_task`; regression test pins the
+  RUNNING status surviving a retry.
+- **Enrichment perf**: OC_SECTION lookups batched to one query per corpus
+  (was one per document), and `link_external_references` collapses O(N)
+  row-by-row saves into two `bulk_update` calls (refs + mentions).

changelog.d/governance-graph-query.added.md

@@ -0,0 +1,19 @@
+- **`governanceGraph` GraphQL query** — the corpus-scoped reference web in
+  node-link form, the in-app successor to `demo/export_governance_graph.py`
+  (first integration step from #1976's callout). Nodes are documents (filing
+  primaries, exhibits, statute sections) plus "ghost" nodes for law citations
+  with no visible target document; edges are mention-weighted `LAW` (resolved,
+  possibly cross-corpus), `LAW_EXTERNAL` (rolled up to the citation's section
+  root), and `DOCUMENT` (`DocumentRelationship` rollups).
+  - Graph assembly lives in
+    `opencontractserver/enrichment/services/governance_graph_service.py`;
+    the resolver (`config/graphql/annotation_queries.py`) only encodes relay
+    ids — no inline Tier-0 (E001 green).
+  - Visibility: corpus READ gates the query (invisible corpus → empty graph);
+    every surfaced document is independently READ-checked — invisible source
+    documents drop their edges, invisible target documents degrade to external
+    ghost nodes so titles never leak, and only READ-visible target corpora are
+    listed (pinned by `opencontractserver/tests/test_governance_graph.py`).
+  - Node lists are degree-capped at `GOVERNANCE_GRAPH_MAX_NODES` (200,
+    `opencontractserver/constants/stats.py`) with full-graph counts +
+    `truncated` flag, mirroring `corpusDocumentGraph`'s contract.

config/graphql/annotation_queries.py

@@ -13,12 +13,17 @@
 from graphene_django.filter import DjangoFilterConnectionField
 from graphql import GraphQLError
 from graphql_jwt.decorators import login_required
-from graphql_relay import from_global_id
+from graphql_relay import from_global_id, to_global_id
 
 from config.graphql.filters import LabelFilter, LabelsetFilter, RelationshipFilter
 from config.graphql.graphene_types import (
     AnnotationLabelType,
     AnnotationType,
+    CorpusReferenceType,
+    GovernanceGraphCorpusType,
+    GovernanceGraphEdgeType,
+    GovernanceGraphNodeType,
+    GovernanceGraphType,
     LabelSetType,
     NoteType,
     PageAwareAnnotationType,
@@ -37,7 +42,9 @@
     DOCUMENT_ANNOTATION_INDEX_LIMIT,
     MANUAL_ANNOTATION_SENTINEL,
 )
+from opencontractserver.constants.stats import GOVERNANCE_GRAPH_MAX_NODES
 from opencontractserver.documents.models import Document
+from opencontractserver.enrichment import constants as enrichment_constants
 from opencontractserver.shared.services.base import BaseService
 from opencontractserver.types.enums import LabelType
 
@@ -47,6 +54,153 @@
 class AnnotationQueryMixin:
     """Query fields and resolvers for annotation, relationship, label, labelset, and note queries."""
 
+    # CORPUS REFERENCE RESOLVERS ###############################
+    corpus_references = DjangoConnectionField(
+        CorpusReferenceType,
+        corpus_id=graphene.ID(required=True),
+        reference_type=graphene.String(),
+        canonical_key=graphene.String(),
+    )
+
+    def resolve_corpus_references(self, info, corpus_id, **kwargs) -> Any:
+        """List enrichment cross-references for a corpus the user can read.
+
+        Visibility is enforced by ``CorpusReferenceService`` (corpus-derived);
+        no inline Tier-0 permission fusion here.
+        """
+        from opencontractserver.annotations.models import CorpusReference
+        from opencontractserver.enrichment.services import CorpusReferenceService
+
+        pk_str = from_global_id(corpus_id)[1]
+        if not str(pk_str).isdigit():
+            return CorpusReference.objects.none()
+        pk = int(pk_str)
+        qs = CorpusReferenceService.for_corpus(info.context.user, pk)
+        if kwargs.get("reference_type"):
+            qs = qs.filter(reference_type=kwargs["reference_type"])
+        if kwargs.get("canonical_key"):
+            qs = qs.filter(canonical_key=kwargs["canonical_key"])
+        # Pull the FK targets the type resolves in one pass — without this each
+        # CorpusReferenceType row fires a separate query per FK (N+1).
+        return qs.select_related(
+            "source_annotation",
+            "corpus",
+            "target_document",
+            "target_annotation",
+            "target_corpus",
+        )
+
+    governance_graph = graphene.Field(
+        GovernanceGraphType,
+        corpus_id=graphene.ID(required=True),
+        limit=graphene.Int(required=False),
+        description=(
+            "The corpus-scoped reference web in node-link form: documents, "
+            "statute sections, and external-citation ghost nodes, with "
+            "mention-weighted LAW / LAW_EXTERNAL / DOCUMENT edges. Powers the "
+            "Governance Graph panel on the Corpus Intelligence home."
+        ),
+    )
+
+    @graphql_ratelimit_dynamic(get_rate=get_user_tier_rate("READ_MEDIUM"))
+    def resolve_governance_graph(self, info, corpus_id, limit=None) -> Any:
+        """Build the governance graph through ``GovernanceGraphService``.
+
+        All visibility decisions (corpus READ gate, per-document READ checks,
+        ghost degradation for invisible targets) live in the service; this
+        resolver only translates raw PKs / canonical keys into relay ids.
+        """
+        # Function-local service import (services import GraphQL types
+        # transitively — module-level import would cycle).
+        from opencontractserver.enrichment.services import GovernanceGraphService
+
+        empty = GovernanceGraphType(
+            corpora=[],
+            nodes=[],
+            edges=[],
+            document_count=0,
+            external_key_count=0,
+            edge_count=0,
+            mention_count=0,
+            truncated=False,
+        )
+
+        corpus_pk = from_global_id(corpus_id)[1]
+        # Malformed/empty global ids decode to a non-numeric pk; treat as
+        # not-found (empty graph) rather than erroring.
+        if not str(corpus_pk).isdigit():
+            return empty
+
+        node_cap = GOVERNANCE_GRAPH_MAX_NODES
+        if limit is not None and 0 < limit < node_cap:
+            node_cap = limit
+
+        data = GovernanceGraphService.build(
+            info.context.user, int(corpus_pk), node_cap, request=info.context
+        )
+        if data is None:
+            return empty
+
+        def _node_id(endpoint) -> str:
+            kind, val = endpoint
+            if kind == "doc":
+                return to_global_id("DocumentType", val)
+            return f"key:{val}"
+
+        nodes = [
+            GovernanceGraphNodeType(
+                id=to_global_id("DocumentType", n["doc_pk"]),
+                document_id=to_global_id("DocumentType", n["doc_pk"]),
+                title=n["title"],
+                kind=n["kind"],
+                corpus_id=(
+                    to_global_id("CorpusType", n["corpus_pk"])
+                    if n["corpus_pk"]
+                    else None
+                ),
+                authority=n["authority"],
+                degree=n["degree"],
+            )
+            for n in data["doc_nodes"]
+        ] + [
+            GovernanceGraphNodeType(
+                id=f"key:{g['key']}",
+                document_id=None,
+                title=g["key"],
+                kind=enrichment_constants.GRAPH_NODE_EXTERNAL,
+                corpus_id=None,
+                authority=g["authority"],
+                degree=g["degree"],
+            )
+            for g in data["ghost_nodes"]
+        ]
+
+        return GovernanceGraphType(
+            corpora=[
+                GovernanceGraphCorpusType(
+                    id=to_global_id("CorpusType", c["corpus_pk"]),
+                    title=c["title"],
+                    kind=c["kind"],
+                )
+                for c in data["corpora"]
+            ],
+            nodes=nodes,
+            edges=[
+                GovernanceGraphEdgeType(
+                    source=_node_id(e["source"]),
+                    target=_node_id(e["target"]),
+                    edge_type=e["edge_type"],
+                    weight=e["weight"],
+                )
+                for e in data["edges"]
+            ],
+            document_count=data["document_count"],
+            external_key_count=data["external_key_count"],
+            edge_count=data["edge_count"],
+            mention_count=data["mention_count"],
+            truncated=data["truncated"],
+        )
+
     # ANNOTATION RESOLVERS #####################################
     annotations = DjangoConnectionField(
         AnnotationType,