|
1 | 1 | # Forms Lab |
2 | 2 |
|
3 | | -LLM-assisted forms platform for government forms. Upload a PDF, extract a |
4 | | -structured spec, deliver a form experience (static or conversational), and |
5 | | -generate a filled PDF back out. |
6 | | - |
7 | | -The core idea: **separate _what to collect_ from _how to present it_.** A |
8 | | -`DataCollectionSpec` describes the fields and their semantics; a `FormSpec` |
9 | | -describes how they are presented. Swap the presentation (static page, |
10 | | -conversational chat, review layout) without touching the extraction pipeline, |
11 | | -and swap the extraction strategy without touching delivery. Every LLM-powered |
12 | | -step is a pluggable _variant_ that can be selected per user at runtime. |
13 | | - |
14 | | -## Live demo |
15 | | - |
16 | | -- **Dashboard** — [https://forms.labs.flexion.us/](https://forms.labs.flexion.us/) (deployment overview, branch list) |
17 | | -- **Application** — [https://forms.labs.flexion.us/main/](https://forms.labs.flexion.us/main/) (main branch) |
18 | | -- **Catalog** — [https://forms.labs.flexion.us/main/catalog](https://forms.labs.flexion.us/main/catalog) |
19 | | - (architecture, decisions, experiments, personas, stories, design system) |
20 | | -- **Slide deck** — [https://forms.labs.flexion.us/main/presentation](https://forms.labs.flexion.us/main/presentation) |
21 | | - |
22 | | -Each active branch is deployed at `/<branch>/` alongside main. |
23 | | - |
24 | | -## Key findings |
25 | | - |
26 | | -Full write-ups live in the catalog. Headlines: |
27 | | - |
28 | | -1. **Hybrid-v1 Pareto-dominates prompt-only extraction.** |
29 | | - [`sonnet-hybrid-v1`](catalog/experiments/pdf-field-extraction/sonnet-hybrid-v1.md) |
30 | | - (one short instruction, one inline exemplar, temperature=0) wins four of |
31 | | - five metrics outright — precision 99.2%, recall 72.6%, sensitivity 51.1% — |
32 | | - and ties on type accuracy, at the same cost as baseline Sonnet. It is now |
33 | | - the production default. The prompt shape that topped the Assignment 10 |
34 | | - tool-calling leaderboard on Mistral 8B reproduces on Claude Sonnet 4 for a |
35 | | - completely different task. |
36 | | - |
37 | | -2. **Tool-use is the structural precision/sensitivity lever.** |
38 | | - [`tool-use-sonnet`](catalog/experiments/pdf-field-extraction/tool-use-sonnet.md) |
39 | | - forces typed tool calls instead of free JSON: sensitivity accuracy jumps |
40 | | - 27% → 79% (+51pp) and precision reaches 96.3%. Recall is step-limited at |
41 | | - 20 rounds, so it shines on short-to-moderate forms. |
42 | | - |
43 | | -3. **Nova Pro marks the non-Claude capability boundary for extraction.** |
44 | | - [`nova-pro`](catalog/experiments/pdf-field-extraction/nova-pro.md) scored |
45 | | - 97% on the homework's 10-field tool-calling task but extracts at 0.6% |
46 | | - recall here — it summarizes sections instead of enumerating fields. Prompt |
47 | | - engineering does not recover this. Model selection dominates prompt |
48 | | - engineering once the task is outside the model's capability range. |
49 | | - |
50 | | -4. **Model size is not the dominant lever for shaping.** |
51 | | - [Shaping model comparison](catalog/experiments/shaping-model-comparison/_suite.md): |
52 | | - Opus/Sonnet/Haiku all cluster around 67-73% command-kind precision. |
53 | | - Three intents are at ceiling across all three models; two fail across all |
54 | | - three. Prompt disambiguation (e.g. `renamePage` vs `renameGroup`) is the |
55 | | - bottleneck, not parameter count. |
56 | | - |
57 | | -Suite indexes: |
| 3 | +**Forms shouldn't be this hard.** |
| 4 | + |
| 5 | +Forms Lab is an experiment in making high-quality digital forms achievable for |
| 6 | +any public sector organization. It combines practical experience from federal |
| 7 | +forms work with LLM capabilities to collapse the cost of turning paper forms |
| 8 | +into accessible, modern experiences. |
| 9 | + |
| 10 | +## The problem |
| 11 | + |
| 12 | +Public sector organizations struggle to deliver good digital form experiences — |
| 13 | +not because they lack ambition, but because of structural barriers. |
| 14 | +Procurement timelines stretch months. Authority-to-operate processes add |
| 15 | +overhead. Custom development requires specialized talent that's hard to hire |
| 16 | +and expensive to retain. The result: forms stay locked in PDF, and the public |
| 17 | +bears the burden. |
| 18 | + |
| 19 | +## The approach |
| 20 | + |
| 21 | +**Configuration over code.** Forms are defined as structured JSON specs — what |
| 22 | +data to collect, how to present it, how fields map back to source documents. |
| 23 | +The specs are readable and editable by non-technical domain experts through a |
| 24 | +no-code UI, and equally accessible to LLM agents that can generate and refine |
| 25 | +them programmatically. |
| 26 | + |
| 27 | +**LLM assistance across the lifecycle.** Rather than bolting AI onto one step, |
| 28 | +Forms Lab introduces LLM capabilities at multiple stages — extracting fields |
| 29 | +from source PDFs, shaping raw extractions into usable form definitions, |
| 30 | +powering conversational form-filling, and generating accessible forms from |
| 31 | +regulatory source material. Each LLM-powered step is a pluggable _variant_: |
| 32 | +swappable at runtime, comparable in evaluation harnesses, improvable |
| 33 | +independently. |
| 34 | + |
| 35 | +**Git-native collaboration.** Every project is a git repository. Branches |
| 36 | +represent draft edits; merging publishes. This gives teams version history, |
| 37 | +diff-based review, and rollback — the same workflow developers already trust, |
| 38 | +extended to form content that non-developers manage through the UI. |
| 39 | + |
| 40 | +**Modern CSS with conformance testing.** The design system uses USWDS 3.x |
| 41 | +design tokens exclusively, enforced by automated stylelint rules. Cascade |
| 42 | +layers (`@layer`) provide predictable specificity. Conformance tests verify |
| 43 | +that our modern CSS implementation looks and behaves like USWDS — maintaining |
| 44 | +its well-tested patterns and accessibility guarantees even as the underlying |
| 45 | +approach evolves. |
| 46 | + |
| 47 | +**Evaluation-driven development.** LLM outputs are non-deterministic, so the |
| 48 | +platform includes structured evaluation harnesses that score extraction and |
| 49 | +shaping quality across fixture sets. This replaces "does it look right?" with |
| 50 | +measurable, reproducible comparisons between strategies. |
| 51 | + |
| 52 | +Experiment results and methodology: |
58 | 53 | [PDF extraction](catalog/experiments/pdf-field-extraction/_suite.md) · |
59 | 54 | [Shaping](catalog/experiments/shaping-model-comparison/_suite.md) · |
60 | | -[Authoring pipeline](catalog/experiments/authoring-pipeline/_suite.md) · |
61 | | -[Roadmap](catalog/experiments/_roadmap.md) |
| 55 | +[Authoring pipeline](catalog/experiments/authoring-pipeline/_suite.md) |
62 | 56 |
|
63 | | -## Quick start |
| 57 | +## Try it |
| 58 | + |
| 59 | +- **Live application** — |
| 60 | + [https://forms.labs.flexion.us/main/](https://forms.labs.flexion.us/main/) |
| 61 | +- **Catalog** — |
| 62 | + [https://forms.labs.flexion.us/main/catalog](https://forms.labs.flexion.us/main/catalog) |
| 63 | + (architecture, decisions, experiments, design system) |
| 64 | + |
| 65 | +### Local development |
64 | 66 |
|
65 | 67 | **Prerequisites:** [Bun](https://bun.sh/) 1.x or later. |
66 | 68 |
|
67 | 69 | ```bash |
68 | 70 | bun install |
69 | | -bun run dev # dev server at http://localhost:3000 |
70 | | -bun test # tests |
71 | | -bun run check # lint + type check + tests (run before push) |
| 71 | +bun run dev # dev server at http://localhost:3000 |
| 72 | +bun test # tests |
| 73 | +bun run check # lint + type check + tests (run before push) |
72 | 74 | ``` |
73 | 75 |
|
74 | 76 | See [CLAUDE.md](CLAUDE.md) for the full command reference, session workflow, |
75 | | -deployment architecture, and contribution conventions. |
| 77 | +and contribution conventions. |
76 | 78 |
|
77 | 79 | ## Project layout |
78 | 80 |
|
79 | 81 | ``` |
80 | 82 | src/ |
81 | | -├── entrypoints/ # Hono servers and CLI |
82 | | -│ ├── app/ # Forms platform web app (routes, middleware, public) |
83 | | -│ ├── dashboard/ # Deployment dashboard (homepage service) |
84 | | -│ ├── webhook/ # GitHub webhook listener |
85 | | -│ ├── notify/ # Notification delivery |
86 | | -│ └── cli/ # CLI commands |
87 | | -├── services/ # Domain services (one public entry per service) |
88 | | -│ ├── data-collection/ # Core model: what to collect |
89 | | -│ ├── forms/ # Resolution, delivery, sessions, shaping, filling |
90 | | -│ ├── form-documents/ # PDF extraction, field mapping, filling |
91 | | -│ ├── extraction/ # Extraction variant registry |
92 | | -│ ├── evaluation/ # Evaluation harness and LLM-as-judge kinds |
93 | | -│ ├── projects/ # Project service and form-project git repo |
94 | | -│ ├── auth/ # GitHub OAuth, sessions |
95 | | -│ ├── deployment/ # Deploy orchestration |
96 | | -│ └── ... |
97 | | -├── design-system/ # flex-* components (server-rendered JSX) |
98 | | -└── shared/ # Pure utilities |
99 | | -
|
100 | | -catalog/ # Versioned catalog content (markdown/JSON) |
101 | | -├── personas/ # Who the system serves |
102 | | -├── stories/ # GitHub Issue copies (user stories) |
103 | | -├── architecture/ # System docs |
104 | | -├── decisions/ # ADRs |
105 | | -└── experiments/ # LLM experiment suites + runs |
106 | | -
|
107 | | -projects/ # Form project directories (specs, assets) |
108 | | -infrastructure/ # Pulumi (EC2) + NixOS (server config) |
109 | | -test/ # Bun test suite |
| 83 | +├── entrypoints/ # Hono servers and CLI |
| 84 | +│ ├── app/ # Forms platform web app |
| 85 | +│ ├── dashboard/ # Deployment dashboard |
| 86 | +│ ├── webhook/ # GitHub webhook listener |
| 87 | +│ └── cli/ # CLI commands |
| 88 | +├── services/ # Domain services (one public entry per service) |
| 89 | +│ ├── data-collection/ # Core model: what to collect |
| 90 | +│ ├── forms/ # Resolution, delivery, sessions, shaping, filling |
| 91 | +│ ├── form-documents/ # PDF extraction, field mapping, filling |
| 92 | +│ ├── extraction/ # Extraction variant registry |
| 93 | +│ ├── evaluation/ # Evaluation harness |
| 94 | +│ ├── projects/ # Project service and git repo |
| 95 | +│ └── auth/ # GitHub OAuth, sessions |
| 96 | +├── design-system/ # flex-* components (server-rendered JSX) |
| 97 | +└── shared/ # Pure utilities |
| 98 | +
|
| 99 | +catalog/ # Versioned catalog (architecture, decisions, experiments) |
| 100 | +infrastructure/ # Pulumi (EC2) + NixOS (server config) |
110 | 101 | ``` |
111 | 102 |
|
112 | 103 | Dependencies flow one way: `shared → services/design-system → entrypoints`. |
113 | | -Each service exposes its public API through `src/services/<name>/index.ts` |
114 | | -and is enforced by `test/architecture/dependency-rule.test.ts`. See the |
115 | | -[architecture principles](catalog/decisions/architecture/architecture-principles.md). |
| 104 | +See the [architecture principles](catalog/decisions/architecture/architecture-principles.md). |
| 105 | + |
| 106 | +## Origins |
| 107 | + |
| 108 | +This work began as a final project for Flexion's |
| 109 | +[LLMs In Production](https://www.manning.com/books/llms-in-production) class |
| 110 | +([cohort repo](https://github.com/flexion/llm-class-2026-winter-cohort); |
| 111 | +class outcome preserved on the |
| 112 | +[`final-project`](https://github.com/flexion/forms-lab/tree/final-project) |
| 113 | +branch). Development continues on `main`, building toward a scalable platform |
| 114 | +for public sector forms. |
| 115 | + |
| 116 | +Forms Lab builds on experience from the |
| 117 | +[10x Form Platform](https://github.com/gsa-tts/forms) |
| 118 | +([Flexion fork](https://github.com/flexion/forms)) — a GSA-funded initiative |
| 119 | +exploring reusable approaches to government forms digitization. |
116 | 120 |
|
117 | 121 | ## Tech stack |
118 | 122 |
|
119 | 123 | - **Runtime:** Bun |
120 | 124 | - **Framework:** Hono (server-rendered JSX, no client runtime) |
121 | 125 | - **Language:** TypeScript |
122 | | -- **Testing:** Bun test |
123 | | -- **Linting:** Biome + Stylelint (design-token enforcement) |
| 126 | +- **LLMs:** Claude (Opus/Sonnet/Haiku) via Anthropic SDK and AWS Bedrock |
124 | 127 | - **Persistence:** Git-based — specs and catalog content live in the repo |
125 | | -- **LLMs:** Claude (Opus/Sonnet/Haiku) via Anthropic SDK and AWS Bedrock; |
126 | | - Amazon Nova Pro for cross-provider comparison |
127 | | -- **Deployment:** Pulumi + NixOS on EC2, branch-per-deployment via GitHub |
128 | | - webhook |
129 | | - |
130 | | -## Workflow |
131 | | - |
132 | | -Session commands (installed in `.claude/commands/`): |
133 | | - |
134 | | -```bash |
135 | | -/create-story # New user story (GitHub issue + notes dir) |
136 | | -/start-story <description> # Initialize a session (worktree, context) |
137 | | -/finish-story # Run checks, code review, open PR |
138 | | -/review-story <PR> # Review another session's PR |
139 | | -``` |
140 | | - |
141 | | -Infrastructure changes branch from main as `infra/YYYY-MM-DD-description`; |
142 | | -feature work branches as `story-N/name` or `docs/<description>`. See |
143 | | -[CLAUDE.md](CLAUDE.md) for commit conventions, the stacked branch workflow, |
144 | | -and deployment details. |
| 128 | +- **Deployment:** Pulumi + NixOS on EC2, branch-per-deployment via GitHub webhook |
145 | 129 |
|
146 | 130 | ## Contributing |
147 | 131 |
|
148 | | -Class project for [LLM Class 2026 Winter Cohort](https://github.com/flexion/llm-class-2026-winter-cohort). |
149 | 132 | Development follows vertical slicing: each user story delivers a complete, |
150 | 133 | demoable capability through all layers. Tests are required for new |
151 | | -functionality; `bun run check` must pass before push. |
| 134 | +functionality. `bun run check` must pass before push. |
| 135 | + |
| 136 | +See [CLAUDE.md](CLAUDE.md) for commit conventions, the stacked branch workflow, |
| 137 | +and deployment details. |
152 | 138 |
|
153 | 139 | ## License |
154 | 140 |
|
|
0 commit comments