Skip to content

Commit 9ee5bd1

Browse files
danielnaabdanielnaab
authored
Handle PDF parsing internally (#10)
1 parent 4f3fd64 commit 9ee5bd1

70 files changed

Lines changed: 3414 additions & 2998 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/_validate.yml‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -96,16 +96,16 @@ jobs:
9696
with:
9797
terraform_version: '1.10.4'
9898

99-
# - name: Initialize Terraform CDK configuration
100-
# shell: bash
101-
# working-directory: infra/cdktf
102-
# run: |
103-
# pnpm cdktf get
104-
# pnpm build:tsc
105-
106-
# - name: Typecheck source code
107-
# shell: bash
108-
# run: pnpm typecheck
99+
- name: Initialize Terraform CDK configuration
100+
shell: bash
101+
working-directory: infra/cdktf
102+
run: |
103+
pnpm cdktf get
104+
pnpm build:tsc
105+
106+
- name: Typecheck source code
107+
shell: bash
108+
run: pnpm typecheck
109109

110110
# - name: Vitest Coverage Report
111111
# if: always()

‎CLAUDE.md‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Project Overview
6+
7+
Forms Platform is a forms-as-a-service platform for government organizations, enabling non-technical staff to create user-friendly "guided interview" web experiences. The platform serves two primary personas:
8+
- **Form Builders**: Create and publish forms via a no-code browser interface
9+
- **Form Fillers**: Complete forms created by form builders
10+
11+
## Core Concepts
12+
13+
- **Blueprint**: Defines the structure of an interactive session between government and user
14+
- **Conversation**: A single instance of a blueprint (one interactive session)
15+
- **Pattern/Template**: Building blocks of a blueprint, implementing UX best-practices
16+
- **Prompt**: Produced by a pattern, defines what is presented to the user at a single point in a conversation
17+
- **Component**: UI building block of prompts
18+
19+
## Common Commands
20+
21+
### Setup
22+
```bash
23+
pnpm install # Install dependencies
24+
pnpm dlx playwright@1.51.1 install --with-deps # One-time: Install browsers for Vitest
25+
```
26+
27+
### Development
28+
```bash
29+
pnpm build # Build all packages (required before dev)
30+
pnpm dev # Start dev servers (Astro at :4321, Storybook at :61610)
31+
```
32+
33+
### Testing
34+
```bash
35+
pnpm test # Run all tests (requires Docker/Podman for PostgreSQL)
36+
pnpm vitest # Run tests in watch mode
37+
pnpm test:ci # Run tests in CI mode
38+
pnpm test:e2e:dev # Run E2E tests in dev mode
39+
pnpm test:e2e:ci # Run E2E tests in CI mode
40+
```
41+
42+
### Testing Individual Packages
43+
```bash
44+
pnpm --filter @flexion/forms-design test:watch # Watch mode for specific package
45+
```
46+
47+
### Code Quality
48+
```bash
49+
pnpm lint # Lint all packages
50+
pnpm format # Format code with Prettier
51+
pnpm typecheck # Type-check all packages
52+
```
53+
54+
### Cleanup
55+
```bash
56+
pnpm clean:dist # Remove all build artifacts recursively
57+
pnpm clean:modules # Remove all node_modules recursively
58+
```
59+
60+
### CLI Tool
61+
```bash
62+
./manage.sh --help # Access command-line operations
63+
```
64+
65+
## Architecture
66+
67+
### Monorepo Structure
68+
69+
This is a pnpm workspace managed with Turborepo for efficient builds. The codebase is organized into packages and apps:
70+
71+
**Packages** (in `/packages/`):
72+
- `forms`: Core business logic, services, patterns, repository, and document handling
73+
- `/src/services`: Public interface of Forms Platform
74+
- `/src/patterns`: Form building blocks ("patterns")
75+
- `/src/repository`: Database routines
76+
- `/src/documents`: Document ingest and creation
77+
- `/src/context`: Runtime contexts (testing, browser, server-side)
78+
- `design`: User-facing React components, USWDS theme, Storybook stories
79+
- `server`: Node.js web server built on Astro with Express adapter
80+
- `auth`: Authentication and authorization (uses deprecated Lucia Auth with Arctic for Login.gov)
81+
- `database`: PostgreSQL (production) and SQLite (testing) support with Knex migrations and Kysely queries
82+
- `common`: Shared utilities
83+
84+
**Apps** (in `/apps/`):
85+
- `spotlight`: Main Astro website (http://localhost:4321/)
86+
- `sandbox`: Testing/demo application
87+
- `server-doj`: Department of Justice specific server instance
88+
- `cli`: Command-line interface (accessed via `./manage.sh`)
89+
90+
**Infrastructure** (in `/infra/`):
91+
- `aws-cdk`: AWS CDK infrastructure code
92+
- `cdktf`: Terraform CDK infrastructure code
93+
- `core`: Shared infrastructure utilities
94+
95+
### Dependency Flow
96+
```
97+
server → auth, common, database, design, forms
98+
forms → common, database
99+
design → common, forms
100+
auth → common, database
101+
database → common
102+
common → (no dependencies)
103+
```
104+
105+
### Key Technologies
106+
107+
- **Build System**: pnpm workspaces + Turborepo for efficient caching and builds
108+
- **Frontend**: Astro (static site framework), React components, USWDS design system
109+
- **Backend**: Node.js with Express
110+
- **Database**: PostgreSQL (production), SQLite (testing)
111+
- Query builders: Kysely (type-safe queries) and Knex.js (migrations)
112+
- Testing: Testcontainers for Postgres unit tests, in-memory SQLite for integration tests
113+
- **Auth**: Lucia Auth (deprecated) with Arctic for Login.gov OIDC/PKCE
114+
- **Testing**: Vitest (unit/integration), Playwright (E2E), @vitest/browser (Storybook)
115+
- **Language**: TypeScript throughout
116+
117+
### Pattern System
118+
119+
Patterns are the platform's primary building blocks. Each pattern has:
120+
- `type`: String identifier for the pattern type
121+
- `id`: Unique identifier for the pattern instance
122+
- `data`: Configuration data specific to the pattern type
123+
124+
Patterns can be constructed manually or via `PatternBuilder` helper classes. They are stored on the form `Blueprint`'s pattern attribute.
125+
126+
## Testing Strategy
127+
128+
- **Unit tests**: Service-level with in-memory SQLite via `createInMemoryDatabaseContext()`
129+
- **Integration tests**: Database gateway logic tested against PostgreSQL Testcontainers
130+
- **E2E tests**: Playwright tests for full user flows
131+
- **Component tests**: Storybook + @vitest/browser
132+
133+
Use `describeDatabase` helper for testing database routines against both SQLite and PostgreSQL.
134+
135+
## Important Notes
136+
137+
- Node version is specified in `.nvmrc` - use `nvm install` to ensure correct version
138+
- Requires Docker or Podman for running tests (PostgreSQL container)
139+
- Playwright version must match exactly (1.51.1) across local and CI environments
140+
- Build is required before running `pnpm dev`
141+
- Pre-commit hook runs `pnpm format` automatically

‎apps/cli/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
"@flexion/forms-infra-core": "workspace:*",
2020
"@flexion/forms-auth": "workspace:^",
2121
"@flexion/forms-database": "workspace:*",
22+
"@flexion/forms-core": "workspace:*",
2223
"commander": "^11.1.0"
2324
}
2425
}

‎apps/cli/src/cli-controller/forms.ts‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
import { promises as fs } from 'fs';
2+
import { Command } from 'commander';
3+
4+
import { commands } from '@flexion/forms-infra-core';
5+
import { type Context } from './types.js';
6+
import { createFormService, createFormsRepository, defaultFormConfig, createTestPdfParser, parsePdf as parsePdfCore } from '@flexion/forms-core';
7+
import { createFilesystemDatabaseContext } from '@flexion/forms-database/context';
8+
9+
export const addFormCommands = (ctx: Context, cli: Command) => {
10+
const cmd = cli
11+
.command('forms')
12+
.description('form management commands')
13+
.option('-d, --database <string>', 'Path to the dev sqlite3 database file. (Postgres currently not wired up.)', async databasePath => {
14+
ctx.db = await createFilesystemDatabaseContext(databasePath);
15+
const repository = createFormsRepository({ db: ctx.db, formConfig: defaultFormConfig });
16+
ctx.forms = createFormService({
17+
repository,
18+
isUserLoggedIn: () => true,
19+
config: defaultFormConfig,
20+
parser: createTestPdfParser(), // Use test parser with filesystem cache for CLI
21+
});
22+
});
23+
24+
cmd
25+
.command('import-pdf')
26+
.description('Intialize a new form by importing a PDF file')
27+
.argument('<string>', 'Source PDF file for form.')
28+
.action(async inputFile => {
29+
// For standalone import-pdf command, use test parser with caching
30+
const parser = createTestPdfParser();
31+
const pdfBytes = await fs.readFile(inputFile);
32+
const maybeForm = await parsePdfCore({ parser, formConfig: defaultFormConfig }, pdfBytes);
33+
if (maybeForm === undefined) {
34+
console.error('Error parsing PDF file:', inputFile);
35+
return;
36+
}
37+
console.log(JSON.stringify(maybeForm, null, 2));
38+
});
39+
40+
cmd
41+
.command('add')
42+
.description('add a form')
43+
.argument('<string>', 'Source JSON file for form.')
44+
.action(async inputFile => {
45+
const fileContents = await fs.readFile(inputFile);
46+
const fileName = inputFile.split(/[\\/]/).pop() ?? inputFile;
47+
const result = await ctx.forms?.initializeForm({
48+
summary: {
49+
title: `Imported Form: ${fileName}`,
50+
description: 'Form imported from PDF',
51+
},
52+
document: {
53+
fileName: inputFile,
54+
data: fileContents.toString('base64')
55+
},
56+
});
57+
console.log(result);
58+
});
59+
};

‎apps/cli/src/cli-controller/index.ts‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
import { Command } from 'commander';
22

3-
import type { Context } from './types.js';
4-
import { addSecretCommands } from './secrets.js';
53
import { addE2eCommands } from './e2e.js';
4+
import { addFormCommands } from './forms.js';
5+
import { addSecretCommands } from './secrets.js';
6+
import type { Context } from './types.js';
67

78
export const CliController = (ctx: Context) => {
89
const cli = new Command().description(
@@ -16,6 +17,7 @@ export const CliController = (ctx: Context) => {
1617
ctx.console.log('Hello!');
1718
});
1819

20+
addFormCommands(ctx, cli);
1921
addSecretCommands(ctx, cli);
2022
addE2eCommands(ctx, cli);
2123

‎apps/cli/src/cli-controller/types.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
1+
import type { FormService } from "@flexion/forms-core";
2+
import type { DatabaseContext } from "@flexion/forms-database";
3+
14
export type Context = {
25
console: Console;
36
workspaceRoot: string;
47
file?: string;
8+
db?: DatabaseContext;
9+
forms?: FormService;
510
};

‎apps/spotlight/src/context.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ import {
22
type FormConfig,
33
type FormService,
44
createFormService,
5-
parsePdf,
5+
createNoopPdfParser,
66
} from '@flexion/forms-core';
77
import { defaultFormConfig } from '@flexion/forms-core';
88
import { BrowserFormRepository } from '@flexion/forms-core/context';
@@ -44,7 +44,7 @@ const createAppFormService = () => {
4444
repository,
4545
config: defaultFormConfig,
4646
isUserLoggedIn: () => true,
47-
parsePdf,
47+
parser: createNoopPdfParser(),
4848
});
4949
} else {
5050
return createTestBrowserFormService();

‎infra/core/package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,6 @@
2020
"@aws-sdk/client-ssm": "^3.624.0",
2121
"@flexion/forms-common": "workspace:*",
2222
"@flexion/forms-core": "workspace:*",
23-
"zod": "^3.23.8"
23+
"zod": "^4.1.11"
2424
}
2525
}

‎infra/core/src/lib/types.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ export type SecretKey = string;
55
export type SecretValue = string | undefined;
66
export type SecretMap = Record<SecretKey, SecretValue>;
77

8-
const secretMap = z.record(z.string());
8+
const secretMap = z.record(z.string(), z.string().optional());
99

1010
export const getSecretMapFromJsonString = (
1111
jsonString?: string

‎manage.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
#!/bin/sh
22

3-
pnpm --filter @flexion/forms-cli cli $@
3+
pnpm --filter @flexion/forms-cli cli "$@"

0 commit comments

Comments
 (0)