Skip to content

Commit baaf6e8

Browse files
JSv4JSv4
committed
Intelligence setup hardening, PDF inline citations, GraphQL validation + perf fixes, demo walkthroughs
Squash of six commits addressing the PR #1982 review and the defects uncovered while verifying the feature live. Intelligence setup (review fixes): - batch_run_action(allow_partial=True) queues up to BATCH_RUN_MAX_DOCS instead of refusing on large corpora; per-template remainingCount surfaces the deferred remainder in the banner toast. - Setup status gains canSetup + referenceAvailable; is_fully_set_up no longer demands deployment-unavailable pieces (zombie-CTA fix); the banner hides for viewers who cannot run setup. - setupCorpusIntelligence requires CRUD, matching AddTemplateToCorpus / CreateCorpusAction; shared CorpusActionService.install_template replaces the duplicated clone recipe; reference action dedupe via get_or_create + GovernanceGraphLive consults setup status before installing; post-create chain surfaces ok=false; lookup-only EnrichmentService.get_analyzer; redundant queries trimmed. PDF inline citations: - Enrichment mentions on PDF documents are projected onto PAWLs token bounding boxes via PlasmaPDF (shared opencontractserver/utils/ span_projection.py, also used by datacell grounding) — TOKEN_LABEL with real page numbers instead of unpaintable char-offset spans. Whitespace-insensitive ordinal-occurrence remapping absorbs txt-extract/PAWLs drift; legacy span mentions upgrade in place on re-enrichment. useReferenceMentions fixed (scoped analyzedCorpusId discovery, client.query instead of a hanging looped useLazyQuery, lean GET_REFERENCE_MENTIONS_FOR_ANALYSIS selection — ~176s -> ~0s). GraphQL spec validation + performance: - validation_rules now extend (not replace) graphql-core's spec rules — unknown-argument/field and variable-type validation had been silently disabled in every environment. All 26 invalid frontend documents repaired, including silently-broken features: new DeleteMetadataColumn and UpdateFieldset mutations (BaseService pattern, IDOR-unified messages), corpus chat history shape fix, tokenAuth unified on the WithUser payload, redirect corpus context sourced from route slugs. CI sweep: tests/architecture/test_frontend_graphql_documents.py + scripts/validate_frontend_graphql.py. - Presigned-URL cache lifetime now derives from AWS_QUERYSTRING_EXPIRE (was the 7-day CacheControl max-age) and is clamped to half the signature lifetime — cached file links could 403 for hours. - UserFeedbackQuerySet.visible_to_user inherited-visibility rewritten as a correlated Exists (was an uncorrelated IN materializing the full annotations table per evaluation): GetAnnotationsForAnalysis measured 176s -> 2.3s for a 108-mention document. Branding + docs: - Navbar wordmark [cite] -> [OpenContracts] (serif treatment kept, 19px, aria-label aligned). - README leads with two captioned demo walkthrough GIFs (docs/assets/images/gifs/demo-{1,2}-*.gif): one-click corpus intelligence setup and the explore-and-ask tour with inline statutory citations.
1 parent 6687145 commit baaf6e8

49 files changed

Lines changed: 2118 additions & 509 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 34 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -29,14 +29,39 @@ findings = await agent.structured_response(
2929
[![Sponsor](https://img.shields.io/static/v1?label=Sponsor&message=%E2%9D%A4&logo=GitHub&color=%23fe8e86)](https://github.com/sponsors/JSv4)
3030

3131
| | |
32-
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
32+
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
3333
| Backend coverage | [![backend](https://codecov.io/gh/Open-Source-Legal/OpenContracts/branch/main/graph/badge.svg?flag=backend&token=RdVsiuaTVz)](https://app.codecov.io/gh/Open-Source-Legal/OpenContracts?flags%5B0%5D=backend) |
3434
| Frontend coverage | [![frontend](https://codecov.io/gh/Open-Source-Legal/OpenContracts/branch/main/graph/badge.svg?flag=frontend&token=RdVsiuaTVz)](https://app.codecov.io/gh/Open-Source-Legal/OpenContracts?flags%5B0%5D=frontend) |
3535
| Meta | [![code style - black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![types - Mypy](https://img.shields.io/badge/types-Mypy-blue.svg)](https://github.com/python/mypy) [![imports - isort](https://img.shields.io/badge/imports-isort-ef8336.svg)](https://github.com/pycqa/isort) [![License - MIT](https://img.shields.io/badge/license-MIT-green)](https://opensource.org/licenses/MIT) |
3636

3737
---
3838

39-
![Discovery Landing Page](docs/assets/images/screenshots/auto/landing--discovery-page--anonymous.png)
39+
## From documents to a citation graph — in about a minute
40+
41+
Create a corpus, drop in your documents, and click **Set up**. That one click installs the
42+
intelligence bundle: agents describe and summarize every document, and the reference web
43+
starts weaving — every statutory citation detected, resolved, and drawn as an edge.
44+
45+
![Create a corpus and set up collection intelligence in one click](docs/assets/images/gifs/demo-1-create-and-setup.gif)
46+
47+
By the end of the clip, 36 SEC filings are a navigable graph — wired to the Delaware
48+
General Corporation Law, the Securities Act, and the SEC rules they cite, section by
49+
section. Law the library doesn't hold yet isn't dropped on the floor: it's tracked as a
50+
backlog, automatically, until you ingest it.
51+
52+
### Then explore it — and ask it questions
53+
54+
Citations are highlighted inline on the filings themselves. The References panel lists
55+
everything a document cites — click any cite to open the statute, with its own
56+
cross-references and everything that cites it back. The ask bar runs a corpus-scoped
57+
agent whose answers come back grounded and cited.
58+
59+
![Explore the citation graph — inline citations, the references panel, and grounded answers](docs/assets/images/gifs/demo-2-explore-and-ask.gif)
60+
61+
Everything in both clips is the stock product against a local install — no custom code,
62+
and every surface the UI touches is also reachable over the API and MCP server below.
63+
64+
---
4065

4166
## Build on it
4267

@@ -189,7 +214,9 @@ This is the DRY principle applied to the citation graph: annotate once, build on
189214

190215
---
191216

192-
## See it in Action
217+
## Annotation flows
218+
219+
The human side of the graph — precise, layout-faithful annotation on PDFs and text:
193220

194221
### PDF Annotation Flow
195222

@@ -240,10 +267,10 @@ docker compose -f production.yml up -d
240267

241268
The discover/landing page and the `/about` page are driven by a JSON content pack so deployers can retarget the messaging without forking the codebase. Two variants ship in the repo:
242269

243-
| Variant key | Framing | Best fit |
244-
| --------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------- |
245-
| `default` | _Open-source document intelligence you can build on._ | The OSS project's repo and most self-hosted deployments — developer-facing. |
246-
| `public-record` | _The citation layer underneath the public record._ | End-user deployments curating public-domain documents (named-incumbents pitch). |
270+
| Variant key | Framing | Best fit |
271+
| --------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------- |
272+
| `default` | _Open-source document intelligence you can build on._ | The OSS project's repo and most self-hosted deployments — developer-facing. |
273+
| `public-record` | _The citation layer underneath the public record._ | End-user deployments curating public-domain documents (named-incumbents pitch). |
247274

248275
Switch variants at runtime by setting `REACT_APP_LANDING_VARIANT` in `frontend/public/env-config.js` — no rebuild required. Unknown variant keys fall back to `default`.
249276

‎changelog.d/1982-review-fixes.fixed.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
- **Intelligence setup: large corpora no longer silently skip enrichment.**
2+
`CorpusActionService` gained `batch_run_action(user, action, allow_partial=)`
3+
(`opencontractserver/corpuses/services/corpus_actions.py`) — the trusted-caller
4+
variant the one-click setup now uses with `allow_partial=True`, queuing the
5+
first `BATCH_RUN_MAX_DOCS` documents (deterministic id order) instead of
6+
refusing outright when a corpus exceeds the per-call cap. The per-template
7+
outcome (`TemplateSetupOutcome.remaining_count`, exposed as `remainingCount`
8+
on `IntelligenceTemplateOutcomeType`) reports the deferred remainder and the
9+
banner toast surfaces it. Previously a 250-doc corpus got a success toast, a
10+
permanently hidden banner, and zero documents enriched.
11+
- **Intelligence-setup status no longer demands deployment-unavailable pieces.**
12+
`IntelligenceSetupStatus.is_fully_set_up`
13+
(`opencontractserver/corpuses/services/intelligence_setup.py`) excludes the
14+
reference action when no enrichment analyzer is registered
15+
(`reference_available`, new on the status payload) and excludes bundle
16+
templates that are unseeded/inactive deployment-wide — either condition
17+
previously made the setup CTA an undismissable zombie whose every click
18+
toasted success.
19+
- **Setup CTA hidden from viewers who can't run it.** The status payload gained
20+
`can_setup` (mirrors the mutation's permission gate);
21+
`IntelligenceSetupBanner.tsx` renders nothing unless `canSetup` — read-only
22+
and anonymous viewers of a public not-set-up corpus previously saw a
23+
guaranteed-to-fail "Set up" button.
24+
- **Permission tier harmonized to CRUD.** `setupCorpusIntelligence` (service +
25+
mutation docstrings, `config/graphql/corpus_mutations.py`) now requires CRUD
26+
on the corpus — the tier `AddTemplateToCorpus` and `CreateCorpusAction`
27+
already gate the identical writes at; it previously required only UPDATE, a
28+
weaker path to the same row installs.
29+
- **Reference action can no longer be double-installed.** The governance
30+
graph's "Map the reference web" bootstrap
31+
(`GovernanceGraphLive.tsx`) consults `corpusIntelligenceSetupStatus` and
32+
skips `createCorpusAction` when the add_document reference action already
33+
exists (a duplicate row would run the enrichment analyzer twice on every
34+
future upload); the server side switched to `get_or_create` to narrow the
35+
concurrent-race window.
36+
- **Post-create setup opt-in surfaces soft failures.** `Corpuses.tsx` now
37+
inspects the resolved `setupCorpusIntelligence.ok` and shows the
38+
"couldn't start" toast — an `ok=false` envelope was previously discarded,
39+
leaving users to believe enrichment was running.
40+
- **Setup warning toast names the actual failures.** The banner aggregates
41+
`templates[].error` into the warning instead of a generic guess.
42+
- **Dedup/cleanup.** Template installs go through a single shared
43+
`CorpusActionService.install_template` (dedupe fast-path, savepoint clone,
44+
IntegrityError recovery, CRUD grant) used by both `AddTemplateToCorpus` and
45+
the bundle; the enrichment analyzer lookup goes through the new lookup-only
46+
`EnrichmentService.get_analyzer()` next to the converge logic; setup
47+
prefetches bundle templates with `name__in` and derives
48+
`total_active_documents` from the batch summary instead of a redundant
49+
corpus-document count.

‎changelog.d/graphql-spec-validation.fixed.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
- **GraphQL spec validation restored on the served endpoint (security).**
2+
``GraphQLView(validation_rules=[DepthLimit…])`` REPLACED graphql-core's
3+
spec rule set (that is ``validate()``'s semantics for an explicit rules
4+
list), silently disabling every standard GraphQL validation — unknown
5+
arguments/fields and variable-type checks — in all environments. Invalid
6+
queries executed with the bogus parts ignored, which let ~26 invalid
7+
frontend documents ship unnoticed (several backing silently-broken
8+
features). ``config/graphql/schema.py`` now builds
9+
``[*specified_rules, DepthLimitValidationRule(, DisableIntrospection)]``,
10+
pinned by ``test_security_hardening.TestServedValidationRulesIncludeSpecRules``.
11+
Every shipped frontend document is now swept in CI by
12+
``opencontractserver/tests/architecture/test_frontend_graphql_documents.py``
13+
(ad-hoc: ``scripts/validate_frontend_graphql.py``; the sweep strips Apollo
14+
``@client`` selections and skips fragment-only/interpolated documents).
15+
- **All 26 invalid frontend documents repaired**, including features that
16+
could never have worked: ``deleteMetadataColumn`` and ``updateFieldset``
17+
were called by the UI but did not exist server-side (both now implemented
18+
in ``config/graphql/extract_mutations.py`` via the BaseService
19+
get_or_none/require_permission pattern with IDOR-unified messages);
20+
``GET_CORPUS_CHAT_MESSAGES`` used a misspelled argument + relay shape on a
21+
plain list field (corpus chat history always loaded empty objects);
22+
``tokenAuth`` was schema-conditional on ``USE_AUTH0`` (now always the
23+
``WithUser`` payload, so the login document validates everywhere); the
24+
document-by-id redirect selected the nonexistent ``DocumentType.corpus``
25+
(corpus context now sourced from the route's slug resolution where it
26+
exists — the previous mock-only field meant graph-node click-throughs
27+
always landed on standalone paths); dead ``ADD_DOCUMENT_TO_CORPUS``
28+
removed; plus variable-type (ID!/String!, JSONString/GenericScalar,
29+
String/enum) and payload-field corrections across vote, thread-moderation,
30+
research-report, TOC and corpus-list documents.
31+
- **Presigned file URLs no longer outlive their signatures.** The AWS
32+
settings branch derived the shared file-URL cache lifetime from
33+
``_AWS_EXPIRY`` (the stored objects' HTTP CacheControl max-age, 7 days)
34+
instead of the presign lifetime (``AWS_QUERYSTRING_EXPIRE``, 1 hour), so
35+
redis served dead 403 pdf/pawls/txt links for up to 5 hours.
36+
``AWS_QUERYSTRING_EXPIRE`` is now explicit, the cache TTL derives from it,
37+
and ``clamp_shared_url_cache_ttl`` (``opencontractserver/utils/files.py``)
38+
enforces TTL ≤ half the signature lifetime even against env overrides.
39+
- **3-minute analysis-annotation responses fixed.**
40+
``UserFeedbackQuerySet.visible_to_user`` expressed annotation-inherited
41+
visibility as ``commented_annotation_id__in=<visible-annotations
42+
subquery>`` — an uncorrelated ``IN`` materialized over the entire
43+
annotations table on every evaluation (~0.8s each; 216 pagination counts
44+
made ``GetAnnotationsForAnalysis`` take ~176s for a 108-mention document).
45+
Rewritten as a correlated ``Exists`` pinned to the feedback row's
46+
annotation id — identical semantics (permissioning invariant suites pass),
47+
measured 176s → 2.3s. Shape pinned by
48+
``test_feedback.TestVisibilityQueryShape``.

‎changelog.d/pdf-inline-citations.fixed.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
- **Inline citations now render on PDF documents.** The enrichment writer
2+
(`opencontractserver/enrichment/writer.py`) projected nothing visible on
3+
PDFs: it stored every reference mention as a `SPAN_LABEL` char-offset
4+
annotation, which the PDF viewer (token-indexed PAWLs renderer) cannot
5+
paint — citations showed in the References panel but never inline on the
6+
filings themselves. Mentions on PDF documents are now projected onto PAWLs
7+
token bounding boxes via PlasmaPDF (`TOKEN_LABEL`, real page numbers
8+
instead of the hardcoded `page=1`), with the char span preserved in
9+
`data.char_span` for dedupe. Projection handles real-ingest drift between
10+
`txt_extract_file` and the PAWLs text via whitespace-insensitive
11+
ordinal-occurrence remapping (covers hard line-wraps and see-quoted SECTION
12+
refs whose raw text extends left of the span start) and falls back to the
13+
span representation when the mention text genuinely is not in the PAWLs
14+
text. Re-running enrichment upgrades pre-fix span mentions **in place**
15+
(same row — `CorpusReference`/`Relationship` FKs survive), so a corpus
16+
re-enrich is also the backfill.
17+
- **Shared span→token projection utility.** The format-aware document-text
18+
loader and the PlasmaPDF span projection moved from private helpers in
19+
`opencontractserver/utils/extraction_grounding.py` to
20+
`opencontractserver/utils/span_projection.py`
21+
(`load_document_text_and_layer`, `project_span_to_token_annotation`);
22+
datacell grounding and the enrichment writer now share one implementation.
23+
- **Reference-mention merge fixed and made usable.**
24+
`useReferenceMentions` (frontend): (1) the analyses discovery query used
25+
`analyses(corpusId:)` — an argument that does not exist in the schema and
26+
was silently ignored (see validation gap below), so the hook swept every
27+
enrichment analysis platform-wide; it now uses the real
28+
`analyzedCorpusId` filter. (2) The per-analysis fetch used a `useLazyQuery`
29+
handle re-executed in a loop, whose promise was observed never settling —
30+
replaced with `client.query`. (3) The fetch now uses a lean
31+
`GET_REFERENCE_MENTIONS_FOR_ANALYSIS` selection: the previous full
32+
selection (per-annotation userFeedback / relationships / document / corpus)
33+
measured **~176s** server-side for 108 mentions vs ~0s for the lean one.
34+
Net effect: inline cites appear within seconds of opening a PDF.
35+
- **Known issues surfaced during this work (deliberately NOT fixed here):**
36+
(1) `GraphQLView(validation_rules=[DepthLimit…])` REPLACES graphql-core's
37+
spec rule set, so standard GraphQL validation (unknown arguments/fields,
38+
variable types) is disabled on the served endpoint; ~34 shipped frontend
39+
documents currently fail spec validation (`scripts/validate_frontend_graphql.py`
40+
enumerates them) — restoring `[*specified_rules, …]` must land with those
41+
query fixes (documented in `config/graphql/schema.py`). (2) Presigned file
42+
URLs are cached for `FILE_URL_SHARED_CACHE_TTL=21600`s while
43+
`AWS_QUERYSTRING_EXPIRE` defaults to 3600s — cached links 403 for hours.
44+
(3) `Document.update_summary`'s docstring claims it updates
45+
`md_summary_file` but it only writes a revision, so intelligence-panel
46+
summary coverage stays 0%. (4) `add_document`-triggered agent actions
47+
re-fire on agent-authored document writes (runaway agent loop / unbounded
48+
LLM spend).

‎config/graphql/corpus_mutations.py‎

Lines changed: 12 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
import graphene
99
from django.conf import settings
1010
from django.core.exceptions import PermissionDenied
11-
from django.db import DatabaseError, IntegrityError, transaction
11+
from django.db import DatabaseError, transaction
1212
from django.utils import timezone
1313
from graphql_jwt.decorators import login_required, user_passes_test
1414
from graphql_relay import from_global_id, to_global_id
@@ -1604,38 +1604,22 @@ def mutate(root, info, template_id: str, corpus_id: str) -> "AddTemplateToCorpus
16041604
# Get the template (templates are global, no user filter needed)
16051605
template = CorpusActionTemplate.objects.get(pk=template_pk, is_active=True)
16061606

1607-
# Fast-path duplicate check (avoids wasted clone + rollback).
1608-
# The unique constraint + IntegrityError catch below handles the
1609-
# race-condition window between this check and the insert.
1610-
if CorpusAction.objects.filter(
1611-
corpus=corpus, source_template=template
1612-
).exists():
1613-
return AddTemplateToCorpus(
1614-
ok=False,
1615-
message="This template has already been added to the corpus",
1616-
obj=None,
1617-
)
1607+
# Shared install recipe (dedupe fast-path, savepoint-wrapped
1608+
# clone, IntegrityError race recovery, CRUD grant) — the same
1609+
# method the one-click intelligence setup uses, so the two
1610+
# install paths cannot drift.
1611+
from opencontractserver.corpuses.services import CorpusActionService
16181612

1619-
# Clone the template into a CorpusAction.
1620-
# Wrap in a savepoint so that a race-condition IntegrityError
1621-
# does not abort the outer transaction (PostgreSQL requirement).
1622-
try:
1623-
with transaction.atomic():
1624-
action = template.clone_to_corpus(corpus, creator=user)
1625-
except IntegrityError:
1613+
action, created = CorpusActionService.install_template(
1614+
user, corpus, template, request=info.context
1615+
)
1616+
if not created:
16261617
return AddTemplateToCorpus(
16271618
ok=False,
16281619
message="This template has already been added to the corpus",
16291620
obj=None,
16301621
)
16311622

1632-
set_permissions_for_obj_to_user(
1633-
user,
1634-
action,
1635-
[PermissionTypes.CRUD],
1636-
request=info.context,
1637-
)
1638-
16391623
return AddTemplateToCorpus(
16401624
ok=True,
16411625
message="Template added to corpus successfully",
@@ -1664,7 +1648,8 @@ class SetupCorpusIntelligence(graphene.Mutation):
16641648
and starts the first weave (deterministic), then clones the description +
16651649
summary action templates and batch-runs each over every document already
16661650
in the corpus (LLM). Safe to repeat — every step skips work that already
1667-
exists. Requires UPDATE permission on the corpus.
1651+
exists. Requires CRUD permission on the corpus — the tier
1652+
AddTemplateToCorpus and CreateCorpusAction gate the identical writes at.
16681653
"""
16691654

16701655
class Arguments:

0 commit comments

Comments
 (0)