Skip to content

Commit 5a1168c

Browse files
authored
Merge pull request #1976 from Open-Source-Legal/feature/corpus-reference-enrichment
Corpus reference enrichment: cross-corpus law linking, authority corpora, corpus-scoped analyzers
2 parents 6bc63fa + 0bf7054 commit 5a1168c

45 files changed

Lines changed: 4436 additions & 18 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -211,3 +211,9 @@ docs/superpowers/
211211

212212
# Superpowers brainstorming scratch files
213213
.superpowers/
214+
215+
# Demo corpus-import zips (large binaries, staged locally from ~/Code/EDGARx2)
216+
demo/import_zips/
217+
218+
# Local-only governance-graph demo (kept on disk, excluded from PRs)
219+
/demo/
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
- **Corpus reference enrichment agent.** A corpus-scoped agent tool pair that
2+
crawls a corpus, inventories explicit references, and persists them — proven
3+
on a real 55-document S-1 corpus (348 references created).
4+
- New deterministic engine in `opencontractserver/enrichment/`
5+
(`extractor.py`, `resolver.py`, `writer.py`, `services/`): regex/grammar
6+
extraction of law citations (`Section 145 of the Delaware General
7+
Corporation Law` → canonical key `dgcl:145`), document/exhibit references,
8+
internal section references, and defined-term definition sites
9+
(`(the "Company")` / `"Change of Control" means …``term:company` /
10+
`term:change-of-control`, opt-in and capped); resolution to in-corpus target
11+
documents and OC_SECTION annotations.
12+
- New `CorpusReference` model (`opencontractserver/annotations/models.py`,
13+
migration `annotations/0078_corpusreference`): a first-class
14+
cross-document / cross-corpus / external-law connection with an indexed
15+
`canonical_key` join key (`target_corpus` anticipates future cross-corpus
16+
linking). Within-document section links continue to use `Relationship`;
17+
every reference also gets a mention `Annotation` (law payloads stored in
18+
`Annotation.data`, resolved document links in `Annotation.link_url` as
19+
in-app site-relative paths).
20+
- Two agent tools (`opencontractserver/llms/tools/core_tools/corpus_references.py`,
21+
registered in `tool_registry.py`): `scan_corpus_references` (read-only
22+
inventory) and `apply_corpus_reference_enrichment`
23+
(`requires_approval=True`, `requires_write_permission=True`) — the
24+
CAML-style scan → approval-gated apply pattern. Enrichment is idempotent.
25+
- Read-only GraphQL `corpusReferences(corpusId)` query
26+
(`config/graphql/annotation_types.py`, `annotation_queries.py`), visibility
27+
scoped to corpus READ via `CorpusReferenceService`.
28+
- Resolved document/exhibit references additionally roll up to
29+
`DocumentRelationship` rows (`enrichment/writer.py`), feeding the corpus
30+
document graph. Mention dedup is by (document, label, span start) so a
31+
growing alias registry cannot duplicate mentions.
32+
- **Authority corpora + cross-corpus law linking.**
33+
- `opencontractserver/enrichment/authorities.py`: `AuthorityCorpusBootstrapper`
34+
materialises statute sections as keyed text documents
35+
(`Document.custom_meta.canonical_key`, e.g. `dgcl:145`), idempotently
36+
(skip / version-up on amendment / self-healing restamp after concurrent
37+
pipeline saves). `find_authority_target` resolves citations with
38+
subsection→section fallback (`dgcl:122(17)``dgcl:122`), visibility- and
39+
current-version-aware.
40+
- `EnrichmentService.link_external_references` upgrades EXTERNAL law
41+
citations to RESOLVED cross-corpus links (`target_corpus`/`target_document`
42+
+ in-app `link_url` on the mention); runs automatically inside `apply()`
43+
and is re-runnable as new authority corpora appear.
44+
- Data-driven authority alias registry: `authority_alias_registry(user)`
45+
merges static defaults with `custom_meta.authority_aliases` declared by
46+
authority corpora — adding a body of law is a bootstrap call, zero code.
47+
- **Citation grammar coverage** (`enrichment/extractor.py`): suffix form
48+
("Section 145 of the DGCL"), prefix form ("Securities Act Section 4(a)(5)"),
49+
statute-internal relative form ("§ 251 of this title", keyed via the
50+
document's own authority context), and bare SEC rules ("Rule 506(b)",
51+
"Rule 144A", "Rule 10b-5" → `sec-rule:*`).
52+
- **Analyzer framework: corpus-scoped analyzers.**
53+
- New `@corpus_analyzer_task` decorator (`shared/decorators.py`) — the
54+
corpus-scoped sibling of `@doc_analyzer_task`: runs once per Analysis,
55+
owns its own writes, wrapper manages the Analysis lifecycle
56+
(RUNNING/COMPLETED/FAILED, timestamps, result/error messages).
57+
- `get_corpus_analyzer_task_by_name` / `get_analyzer_task_by_name`
58+
(`utils/celery_tasks.py`); `run_task_name_analyzer` dispatches
59+
corpus-scoped analyzers as a single task (no per-doc fan-out); analyzer
60+
auto-sync and the `analyzer.W001` check cover both flavours.
61+
- Enrichment adapter task
62+
(`opencontractserver/tasks/corpus_analysis_tasks.py`) registers the engine
63+
as a real task-based Analyzer (dispatchable via `CorpusAction`), attaching
64+
to the framework-created Analysis instead of creating its own.
65+
- **Demo pipeline** (`demo/`, untracked zips excluded): authority seed JSONs
66+
with real statutory text (DGCL, Securities Act, Exchange Act, IRC, ICA,
67+
SEC Rules — 17 C.F.R.), bootstrap/import/export scripts, and a standalone
68+
D3 governance-graph visualization (`governance_graph.html`).
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
- **Reconciled `CorpusVote` migration state with the model** (migration
2+
`corpuses/0057`): migration `0049` created the three `CorpusVote` indexes
3+
with explicit names (`corpuses_co_corpus__vote_idx`, …) while the model
4+
declares them unnamed (Django auto-hashed names), and the `backend_lock`
5+
field's `db_index=True` was missing from migration state. The drift made
6+
`makemigrations --check` fail on every branch. Operations are three
7+
Postgres `ALTER INDEX … RENAME` calls plus one index-state alter — no data
8+
changes.
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
- **Enrichment mention links no longer 404** — the writer and
2+
`link_external_references` wrote `link_url` as `/corpus/{id}/document/{id}`,
3+
a shape no frontend route serves (it falls into the `*` catch-all → 404).
4+
They now emit the canonical slug path
5+
`/d/{corpus.creator.slug}/{corpus.slug}/{document.slug}` via the new
6+
`opencontractserver/utils/frontend_paths.py::document_in_corpus_path`
7+
helper (mirrors the frontend's `buildCanonicalPath`); cross-corpus law
8+
links point into the authority corpus. Links with missing slugs are
9+
skipped rather than written broken.
10+
- **`@corpus_analyzer_task` no longer marks a retrying Analysis FAILED**
11+
(`opencontractserver/shared/decorators.py`) — `celery.exceptions.Retry`
12+
extends `Exception`, so the wrapper's failure branch stamped
13+
`Analysis.status=FAILED` before the retry ran. `Retry` is now re-raised
14+
untouched, mirroring `doc_analyzer_task`; regression test pins the
15+
RUNNING status surviving a retry.
16+
- **Enrichment perf**: OC_SECTION lookups batched to one query per corpus
17+
(was one per document), and `link_external_references` collapses O(N)
18+
row-by-row saves into two `bulk_update` calls (refs + mentions).
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
- **`governanceGraph` GraphQL query** — the corpus-scoped reference web in
2+
node-link form, the in-app successor to `demo/export_governance_graph.py`
3+
(first integration step from #1976's callout). Nodes are documents (filing
4+
primaries, exhibits, statute sections) plus "ghost" nodes for law citations
5+
with no visible target document; edges are mention-weighted `LAW` (resolved,
6+
possibly cross-corpus), `LAW_EXTERNAL` (rolled up to the citation's section
7+
root), and `DOCUMENT` (`DocumentRelationship` rollups).
8+
- Graph assembly lives in
9+
`opencontractserver/enrichment/services/governance_graph_service.py`;
10+
the resolver (`config/graphql/annotation_queries.py`) only encodes relay
11+
ids — no inline Tier-0 (E001 green).
12+
- Visibility: corpus READ gates the query (invisible corpus → empty graph);
13+
every surfaced document is independently READ-checked — invisible source
14+
documents drop their edges, invisible target documents degrade to external
15+
ghost nodes so titles never leak, and only READ-visible target corpora are
16+
listed (pinned by `opencontractserver/tests/test_governance_graph.py`).
17+
- Node lists are degree-capped at `GOVERNANCE_GRAPH_MAX_NODES` (200,
18+
`opencontractserver/constants/stats.py`) with full-graph counts +
19+
`truncated` flag, mirroring `corpusDocumentGraph`'s contract.

config/graphql/annotation_queries.py

Lines changed: 155 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,12 +13,17 @@
1313
from graphene_django.filter import DjangoFilterConnectionField
1414
from graphql import GraphQLError
1515
from graphql_jwt.decorators import login_required
16-
from graphql_relay import from_global_id
16+
from graphql_relay import from_global_id, to_global_id
1717

1818
from config.graphql.filters import LabelFilter, LabelsetFilter, RelationshipFilter
1919
from config.graphql.graphene_types import (
2020
AnnotationLabelType,
2121
AnnotationType,
22+
CorpusReferenceType,
23+
GovernanceGraphCorpusType,
24+
GovernanceGraphEdgeType,
25+
GovernanceGraphNodeType,
26+
GovernanceGraphType,
2227
LabelSetType,
2328
NoteType,
2429
PageAwareAnnotationType,
@@ -37,7 +42,9 @@
3742
DOCUMENT_ANNOTATION_INDEX_LIMIT,
3843
MANUAL_ANNOTATION_SENTINEL,
3944
)
45+
from opencontractserver.constants.stats import GOVERNANCE_GRAPH_MAX_NODES
4046
from opencontractserver.documents.models import Document
47+
from opencontractserver.enrichment import constants as enrichment_constants
4148
from opencontractserver.shared.services.base import BaseService
4249
from opencontractserver.types.enums import LabelType
4350

@@ -47,6 +54,153 @@
4754
class AnnotationQueryMixin:
4855
"""Query fields and resolvers for annotation, relationship, label, labelset, and note queries."""
4956

57+
# CORPUS REFERENCE RESOLVERS ###############################
58+
corpus_references = DjangoConnectionField(
59+
CorpusReferenceType,
60+
corpus_id=graphene.ID(required=True),
61+
reference_type=graphene.String(),
62+
canonical_key=graphene.String(),
63+
)
64+
65+
def resolve_corpus_references(self, info, corpus_id, **kwargs) -> Any:
66+
"""List enrichment cross-references for a corpus the user can read.
67+
68+
Visibility is enforced by ``CorpusReferenceService`` (corpus-derived);
69+
no inline Tier-0 permission fusion here.
70+
"""
71+
from opencontractserver.annotations.models import CorpusReference
72+
from opencontractserver.enrichment.services import CorpusReferenceService
73+
74+
pk_str = from_global_id(corpus_id)[1]
75+
if not str(pk_str).isdigit():
76+
return CorpusReference.objects.none()
77+
pk = int(pk_str)
78+
qs = CorpusReferenceService.for_corpus(info.context.user, pk)
79+
if kwargs.get("reference_type"):
80+
qs = qs.filter(reference_type=kwargs["reference_type"])
81+
if kwargs.get("canonical_key"):
82+
qs = qs.filter(canonical_key=kwargs["canonical_key"])
83+
# Pull the FK targets the type resolves in one pass — without this each
84+
# CorpusReferenceType row fires a separate query per FK (N+1).
85+
return qs.select_related(
86+
"source_annotation",
87+
"corpus",
88+
"target_document",
89+
"target_annotation",
90+
"target_corpus",
91+
)
92+
93+
governance_graph = graphene.Field(
94+
GovernanceGraphType,
95+
corpus_id=graphene.ID(required=True),
96+
limit=graphene.Int(required=False),
97+
description=(
98+
"The corpus-scoped reference web in node-link form: documents, "
99+
"statute sections, and external-citation ghost nodes, with "
100+
"mention-weighted LAW / LAW_EXTERNAL / DOCUMENT edges. Powers the "
101+
"Governance Graph panel on the Corpus Intelligence home."
102+
),
103+
)
104+
105+
@graphql_ratelimit_dynamic(get_rate=get_user_tier_rate("READ_MEDIUM"))
106+
def resolve_governance_graph(self, info, corpus_id, limit=None) -> Any:
107+
"""Build the governance graph through ``GovernanceGraphService``.
108+
109+
All visibility decisions (corpus READ gate, per-document READ checks,
110+
ghost degradation for invisible targets) live in the service; this
111+
resolver only translates raw PKs / canonical keys into relay ids.
112+
"""
113+
# Function-local service import (services import GraphQL types
114+
# transitively — module-level import would cycle).
115+
from opencontractserver.enrichment.services import GovernanceGraphService
116+
117+
empty = GovernanceGraphType(
118+
corpora=[],
119+
nodes=[],
120+
edges=[],
121+
document_count=0,
122+
external_key_count=0,
123+
edge_count=0,
124+
mention_count=0,
125+
truncated=False,
126+
)
127+
128+
corpus_pk = from_global_id(corpus_id)[1]
129+
# Malformed/empty global ids decode to a non-numeric pk; treat as
130+
# not-found (empty graph) rather than erroring.
131+
if not str(corpus_pk).isdigit():
132+
return empty
133+
134+
node_cap = GOVERNANCE_GRAPH_MAX_NODES
135+
if limit is not None and 0 < limit < node_cap:
136+
node_cap = limit
137+
138+
data = GovernanceGraphService.build(
139+
info.context.user, int(corpus_pk), node_cap, request=info.context
140+
)
141+
if data is None:
142+
return empty
143+
144+
def _node_id(endpoint) -> str:
145+
kind, val = endpoint
146+
if kind == "doc":
147+
return to_global_id("DocumentType", val)
148+
return f"key:{val}"
149+
150+
nodes = [
151+
GovernanceGraphNodeType(
152+
id=to_global_id("DocumentType", n["doc_pk"]),
153+
document_id=to_global_id("DocumentType", n["doc_pk"]),
154+
title=n["title"],
155+
kind=n["kind"],
156+
corpus_id=(
157+
to_global_id("CorpusType", n["corpus_pk"])
158+
if n["corpus_pk"]
159+
else None
160+
),
161+
authority=n["authority"],
162+
degree=n["degree"],
163+
)
164+
for n in data["doc_nodes"]
165+
] + [
166+
GovernanceGraphNodeType(
167+
id=f"key:{g['key']}",
168+
document_id=None,
169+
title=g["key"],
170+
kind=enrichment_constants.GRAPH_NODE_EXTERNAL,
171+
corpus_id=None,
172+
authority=g["authority"],
173+
degree=g["degree"],
174+
)
175+
for g in data["ghost_nodes"]
176+
]
177+
178+
return GovernanceGraphType(
179+
corpora=[
180+
GovernanceGraphCorpusType(
181+
id=to_global_id("CorpusType", c["corpus_pk"]),
182+
title=c["title"],
183+
kind=c["kind"],
184+
)
185+
for c in data["corpora"]
186+
],
187+
nodes=nodes,
188+
edges=[
189+
GovernanceGraphEdgeType(
190+
source=_node_id(e["source"]),
191+
target=_node_id(e["target"]),
192+
edge_type=e["edge_type"],
193+
weight=e["weight"],
194+
)
195+
for e in data["edges"]
196+
],
197+
document_count=data["document_count"],
198+
external_key_count=data["external_key_count"],
199+
edge_count=data["edge_count"],
200+
mention_count=data["mention_count"],
201+
truncated=data["truncated"],
202+
)
203+
50204
# ANNOTATION RESOLVERS #####################################
51205
annotations = DjangoConnectionField(
52206
AnnotationType,

0 commit comments

Comments
 (0)