Skip to content

Commit 0e4af86

Browse files
authored
Merge branch 'main' into story-121/form-layout
2 parents 7a5e916 + feac43e commit 0e4af86

7 files changed

Lines changed: 264 additions & 173 deletions

File tree

README.md

Lines changed: 104 additions & 118 deletions
Original file line numberDiff line numberDiff line change
@@ -1,154 +1,140 @@
11
# Forms Lab
22

3-
LLM-assisted forms platform for government forms. Upload a PDF, extract a
4-
structured spec, deliver a form experience (static or conversational), and
5-
generate a filled PDF back out.
6-
7-
The core idea: **separate _what to collect_ from _how to present it_.** A
8-
`DataCollectionSpec` describes the fields and their semantics; a `FormSpec`
9-
describes how they are presented. Swap the presentation (static page,
10-
conversational chat, review layout) without touching the extraction pipeline,
11-
and swap the extraction strategy without touching delivery. Every LLM-powered
12-
step is a pluggable _variant_ that can be selected per user at runtime.
13-
14-
## Live demo
15-
16-
- **Dashboard**[https://forms.labs.flexion.us/](https://forms.labs.flexion.us/) (deployment overview, branch list)
17-
- **Application**[https://forms.labs.flexion.us/main/](https://forms.labs.flexion.us/main/) (main branch)
18-
- **Catalog**[https://forms.labs.flexion.us/main/catalog](https://forms.labs.flexion.us/main/catalog)
19-
(architecture, decisions, experiments, personas, stories, design system)
20-
- **Slide deck**[https://forms.labs.flexion.us/main/presentation](https://forms.labs.flexion.us/main/presentation)
21-
22-
Each active branch is deployed at `/<branch>/` alongside main.
23-
24-
## Key findings
25-
26-
Full write-ups live in the catalog. Headlines:
27-
28-
1. **Hybrid-v1 Pareto-dominates prompt-only extraction.**
29-
[`sonnet-hybrid-v1`](catalog/experiments/pdf-field-extraction/sonnet-hybrid-v1.md)
30-
(one short instruction, one inline exemplar, temperature=0) wins four of
31-
five metrics outright — precision 99.2%, recall 72.6%, sensitivity 51.1% —
32-
and ties on type accuracy, at the same cost as baseline Sonnet. It is now
33-
the production default. The prompt shape that topped the Assignment 10
34-
tool-calling leaderboard on Mistral 8B reproduces on Claude Sonnet 4 for a
35-
completely different task.
36-
37-
2. **Tool-use is the structural precision/sensitivity lever.**
38-
[`tool-use-sonnet`](catalog/experiments/pdf-field-extraction/tool-use-sonnet.md)
39-
forces typed tool calls instead of free JSON: sensitivity accuracy jumps
40-
27% → 79% (+51pp) and precision reaches 96.3%. Recall is step-limited at
41-
20 rounds, so it shines on short-to-moderate forms.
42-
43-
3. **Nova Pro marks the non-Claude capability boundary for extraction.**
44-
[`nova-pro`](catalog/experiments/pdf-field-extraction/nova-pro.md) scored
45-
97% on the homework's 10-field tool-calling task but extracts at 0.6%
46-
recall here — it summarizes sections instead of enumerating fields. Prompt
47-
engineering does not recover this. Model selection dominates prompt
48-
engineering once the task is outside the model's capability range.
49-
50-
4. **Model size is not the dominant lever for shaping.**
51-
[Shaping model comparison](catalog/experiments/shaping-model-comparison/_suite.md):
52-
Opus/Sonnet/Haiku all cluster around 67-73% command-kind precision.
53-
Three intents are at ceiling across all three models; two fail across all
54-
three. Prompt disambiguation (e.g. `renamePage` vs `renameGroup`) is the
55-
bottleneck, not parameter count.
56-
57-
Suite indexes:
3+
**Forms shouldn't be this hard.**
4+
5+
Forms Lab is an experiment in making high-quality digital forms achievable for
6+
any public sector organization. It combines practical experience from federal
7+
forms work with LLM capabilities to collapse the cost of turning paper forms
8+
into accessible, modern experiences.
9+
10+
## The problem
11+
12+
Public sector organizations struggle to deliver good digital form experiences —
13+
not because they lack ambition, but because of structural barriers.
14+
Procurement timelines stretch months. Authority-to-operate processes add
15+
overhead. Custom development requires specialized talent that's hard to hire
16+
and expensive to retain. The result: forms stay locked in PDF, and the public
17+
bears the burden.
18+
19+
## The approach
20+
21+
**Configuration over code.** Forms are defined as structured JSON specs — what
22+
data to collect, how to present it, how fields map back to source documents.
23+
The specs are readable and editable by non-technical domain experts through a
24+
no-code UI, and equally accessible to LLM agents that can generate and refine
25+
them programmatically.
26+
27+
**LLM assistance across the lifecycle.** Rather than bolting AI onto one step,
28+
Forms Lab introduces LLM capabilities at multiple stages — extracting fields
29+
from source PDFs, shaping raw extractions into usable form definitions,
30+
powering conversational form-filling, and generating accessible forms from
31+
regulatory source material. Each LLM-powered step is a pluggable _variant_:
32+
swappable at runtime, comparable in evaluation harnesses, improvable
33+
independently.
34+
35+
**Git-native collaboration.** Every project is a git repository. Branches
36+
represent draft edits; merging publishes. This gives teams version history,
37+
diff-based review, and rollback — the same workflow developers already trust,
38+
extended to form content that non-developers manage through the UI.
39+
40+
**Modern CSS with conformance testing.** The design system uses USWDS 3.x
41+
design tokens exclusively, enforced by automated stylelint rules. Cascade
42+
layers (`@layer`) provide predictable specificity. Conformance tests verify
43+
that our modern CSS implementation looks and behaves like USWDS — maintaining
44+
its well-tested patterns and accessibility guarantees even as the underlying
45+
approach evolves.
46+
47+
**Evaluation-driven development.** LLM outputs are non-deterministic, so the
48+
platform includes structured evaluation harnesses that score extraction and
49+
shaping quality across fixture sets. This replaces "does it look right?" with
50+
measurable, reproducible comparisons between strategies.
51+
52+
Experiment results and methodology:
5853
[PDF extraction](catalog/experiments/pdf-field-extraction/_suite.md) ·
5954
[Shaping](catalog/experiments/shaping-model-comparison/_suite.md) ·
60-
[Authoring pipeline](catalog/experiments/authoring-pipeline/_suite.md) ·
61-
[Roadmap](catalog/experiments/_roadmap.md)
55+
[Authoring pipeline](catalog/experiments/authoring-pipeline/_suite.md)
6256

63-
## Quick start
57+
## Try it
58+
59+
- **Live application**
60+
[https://forms.labs.flexion.us/main/](https://forms.labs.flexion.us/main/)
61+
- **Catalog**
62+
[https://forms.labs.flexion.us/main/catalog](https://forms.labs.flexion.us/main/catalog)
63+
(architecture, decisions, experiments, design system)
64+
65+
### Local development
6466

6567
**Prerequisites:** [Bun](https://bun.sh/) 1.x or later.
6668

6769
```bash
6870
bun install
69-
bun run dev # dev server at http://localhost:3000
70-
bun test # tests
71-
bun run check # lint + type check + tests (run before push)
71+
bun run dev # dev server at http://localhost:3000
72+
bun test # tests
73+
bun run check # lint + type check + tests (run before push)
7274
```
7375

7476
See [CLAUDE.md](CLAUDE.md) for the full command reference, session workflow,
75-
deployment architecture, and contribution conventions.
77+
and contribution conventions.
7678

7779
## Project layout
7880

7981
```
8082
src/
81-
├── entrypoints/ # Hono servers and CLI
82-
│ ├── app/ # Forms platform web app (routes, middleware, public)
83-
│ ├── dashboard/ # Deployment dashboard (homepage service)
84-
│ ├── webhook/ # GitHub webhook listener
85-
│ ├── notify/ # Notification delivery
86-
│ └── cli/ # CLI commands
87-
├── services/ # Domain services (one public entry per service)
88-
│ ├── data-collection/ # Core model: what to collect
89-
│ ├── forms/ # Resolution, delivery, sessions, shaping, filling
90-
│ ├── form-documents/ # PDF extraction, field mapping, filling
91-
│ ├── extraction/ # Extraction variant registry
92-
│ ├── evaluation/ # Evaluation harness and LLM-as-judge kinds
93-
│ ├── projects/ # Project service and form-project git repo
94-
│ ├── auth/ # GitHub OAuth, sessions
95-
│ ├── deployment/ # Deploy orchestration
96-
│ └── ...
97-
├── design-system/ # flex-* components (server-rendered JSX)
98-
└── shared/ # Pure utilities
99-
100-
catalog/ # Versioned catalog content (markdown/JSON)
101-
├── personas/ # Who the system serves
102-
├── stories/ # GitHub Issue copies (user stories)
103-
├── architecture/ # System docs
104-
├── decisions/ # ADRs
105-
└── experiments/ # LLM experiment suites + runs
106-
107-
projects/ # Form project directories (specs, assets)
108-
infrastructure/ # Pulumi (EC2) + NixOS (server config)
109-
test/ # Bun test suite
83+
├── entrypoints/ # Hono servers and CLI
84+
│ ├── app/ # Forms platform web app
85+
│ ├── dashboard/ # Deployment dashboard
86+
│ ├── webhook/ # GitHub webhook listener
87+
│ └── cli/ # CLI commands
88+
├── services/ # Domain services (one public entry per service)
89+
│ ├── data-collection/ # Core model: what to collect
90+
│ ├── forms/ # Resolution, delivery, sessions, shaping, filling
91+
│ ├── form-documents/ # PDF extraction, field mapping, filling
92+
│ ├── extraction/ # Extraction variant registry
93+
│ ├── evaluation/ # Evaluation harness
94+
│ ├── projects/ # Project service and git repo
95+
│ └── auth/ # GitHub OAuth, sessions
96+
├── design-system/ # flex-* components (server-rendered JSX)
97+
└── shared/ # Pure utilities
98+
99+
catalog/ # Versioned catalog (architecture, decisions, experiments)
100+
infrastructure/ # Pulumi (EC2) + NixOS (server config)
110101
```
111102

112103
Dependencies flow one way: `shared → services/design-system → entrypoints`.
113-
Each service exposes its public API through `src/services/<name>/index.ts`
114-
and is enforced by `test/architecture/dependency-rule.test.ts`. See the
115-
[architecture principles](catalog/decisions/architecture/architecture-principles.md).
104+
See the [architecture principles](catalog/decisions/architecture/architecture-principles.md).
105+
106+
## Origins
107+
108+
This work began as a final project for Flexion's
109+
[LLMs In Production](https://www.manning.com/books/llms-in-production) class
110+
([cohort repo](https://github.com/flexion/llm-class-2026-winter-cohort);
111+
class outcome preserved on the
112+
[`final-project`](https://github.com/flexion/forms-lab/tree/final-project)
113+
branch). Development continues on `main`, building toward a scalable platform
114+
for public sector forms.
115+
116+
Forms Lab builds on experience from the
117+
[10x Form Platform](https://github.com/gsa-tts/forms)
118+
([Flexion fork](https://github.com/flexion/forms)) — a GSA-funded initiative
119+
exploring reusable approaches to government forms digitization.
116120

117121
## Tech stack
118122

119123
- **Runtime:** Bun
120124
- **Framework:** Hono (server-rendered JSX, no client runtime)
121125
- **Language:** TypeScript
122-
- **Testing:** Bun test
123-
- **Linting:** Biome + Stylelint (design-token enforcement)
126+
- **LLMs:** Claude (Opus/Sonnet/Haiku) via Anthropic SDK and AWS Bedrock
124127
- **Persistence:** Git-based — specs and catalog content live in the repo
125-
- **LLMs:** Claude (Opus/Sonnet/Haiku) via Anthropic SDK and AWS Bedrock;
126-
Amazon Nova Pro for cross-provider comparison
127-
- **Deployment:** Pulumi + NixOS on EC2, branch-per-deployment via GitHub
128-
webhook
129-
130-
## Workflow
131-
132-
Session commands (installed in `.claude/commands/`):
133-
134-
```bash
135-
/create-story # New user story (GitHub issue + notes dir)
136-
/start-story <description> # Initialize a session (worktree, context)
137-
/finish-story # Run checks, code review, open PR
138-
/review-story <PR> # Review another session's PR
139-
```
140-
141-
Infrastructure changes branch from main as `infra/YYYY-MM-DD-description`;
142-
feature work branches as `story-N/name` or `docs/<description>`. See
143-
[CLAUDE.md](CLAUDE.md) for commit conventions, the stacked branch workflow,
144-
and deployment details.
128+
- **Deployment:** Pulumi + NixOS on EC2, branch-per-deployment via GitHub webhook
145129

146130
## Contributing
147131

148-
Class project for [LLM Class 2026 Winter Cohort](https://github.com/flexion/llm-class-2026-winter-cohort).
149132
Development follows vertical slicing: each user story delivers a complete,
150133
demoable capability through all layers. Tests are required for new
151-
functionality; `bun run check` must pass before push.
134+
functionality. `bun run check` must pass before push.
135+
136+
See [CLAUDE.md](CLAUDE.md) for commit conventions, the stacked branch workflow,
137+
and deployment details.
152138

153139
## License
154140

src/entrypoints/app/routes/catalog/index.tsx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,12 +66,12 @@ catalog.get('/', async (c) => {
6666

6767
<div class="l-stack" style="--stack-space: var(--flex-space-xl)">
6868
<section>
69-
<p class="catalog-group-label">Presentation</p>
69+
<p class="catalog-group-label">Presentations &amp; walkthroughs</p>
7070
<div class="l-grid" style="--grid-min: 280px">
7171
<ContentCard
7272
title="Slide Deck"
7373
href={resolveUrl('/presentation')}
74-
description="15-minute live presentation — narrative arc, key findings, and demo"
74+
description="Final project presentation for Flexion's LLMs In Production class — narrative arc, key findings, and live demo"
7575
/>
7676
<ContentCard
7777
title="Walkthrough"

0 commit comments

Comments
 (0)