Skip to content

Commit c4f2ecf

Browse files
committed
Add paperworks dashboard for WG21 paper management
Paperworks is a web UI that orchestrates scrivener (PDF rendering) and an isocpp.org authenticated client. Three-source inventory correlates markdown, rendered PDFs, and isocpp.org remote state. Markdown is the source of truth for all metadata including abstract. - Flask server with SSE for live updates - File watchdogs for markdown and PDF directories - Queued upload with automatic metadata sync (title, author, abstract) - Draft/review status transitions - Prior revision links from isocpp.org - Render worker backed by scrivener - Inventory warnings for incomplete metadata
1 parent 3ab4a0f commit c4f2ecf

24 files changed

Lines changed: 2946 additions & 25 deletions

paperworks/.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
__pycache__/
2+
*.pyc

paperworks/CLAUDE.md

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# Paperworks - Agent Rules
2+
3+
## What This Is
4+
5+
Paperworks is the unified web dashboard for managing WG21 paper
6+
submissions. It orchestrates two libraries:
7+
8+
- **Scrivener** (`../scrivener/`) - markdown to PDF rendering
9+
- **Docketeer** (`lib/isocpp.py`) - isocpp.org authenticated client
10+
with internal queue
11+
12+
Paperworks itself handles the web UI, watchdogs, inventory
13+
correlation, and orchestration logic.
14+
15+
## Architecture
16+
17+
```
18+
paperworks/
19+
paperworks.py # CLI: python paperworks.py serve
20+
lib/
21+
server.py # Flask app, SSE, routes, both watchdogs, render worker
22+
server_config.py # ~/.paperworks/config.json, falls back to scrivener
23+
inventory.py # Three-source paper database (markdown + PDF + isocpp.org)
24+
isocpp.py # IsoCppSession - queued, lock-serialized, event-emitting
25+
pdf_reader.py # PDF metadata extraction via PyMuPDF
26+
templates/
27+
index.html # Dashboard page (main view)
28+
settings.html # Settings page (config, dirs, credentials)
29+
```
30+
31+
## Source of Truth
32+
33+
Markdown is the source of truth for ALL paper metadata.
34+
35+
- **Priority**: markdown > PDF > isocpp.org. If markdown and
36+
PDF disagree, markdown wins. PDF is a derived artifact.
37+
- **Metadata from markdown**: title, authors, date, audience
38+
(YAML front matter), abstract/brutal summary (`## Abstract`
39+
section in body)
40+
- **PDF fallback**: if no markdown exists, PDF metadata is used
41+
but the paper shows `orphan` status
42+
- **isocpp.org**: provides remote status (Draft/Review/Mailed)
43+
and form URLs only - never a metadata source
44+
- **D/P prefixes**: interchangeable for matching. `D4007R0` and
45+
`P4007R0` refer to the same paper. Papers start as D (draft)
46+
and become P (published) when mailed. The numeric part and
47+
revision are what identify a paper.
48+
- **Primary author**: only the first author from `reply-to` is
49+
synced to isocpp.org. Multi-author papers list all authors
50+
locally but only the first goes to the remote form.
51+
- **Upload syncs everything**: there is no separate "sync
52+
metadata" operation. UPLOAD pushes the PDF AND syncs title,
53+
author, and abstract in one action. `_do_upload` skips any
54+
field that is None/empty - it will never overwrite good
55+
remote data with nothing.
56+
- **Warnings**: if a markdown file has valid front matter but
57+
no `## Abstract` section, the paper gets a warning logged
58+
and shown in the UI. It does not block rendering or uploading.
59+
60+
## Three-Source Inventory
61+
62+
The paper inventory correlates three sources:
63+
64+
1. **Markdown** (source of truth) - YAML front matter provides
65+
doc_number, title, date, authors, audience. The `## Abstract`
66+
section in the body provides the brutal summary. These are
67+
authoritative.
68+
2. **PDF directory** - derived artifacts. If PDF mtime < markdown
69+
mtime, the PDF is stale. PDF can never be newer than markdown.
70+
3. **isocpp.org** - remote paper list with status (Draft/Review/Mailed)
71+
72+
### Derived Status
73+
74+
- `needs_render` - markdown exists, no PDF or PDF is stale
75+
- `draft` / `review` - remote status, PDF is current
76+
- `local` - has PDF, no remote entry
77+
- `orphan` - has PDF but no markdown source
78+
- `mailed` - hidden from default view
79+
80+
## IsoCppSession Queue (Critical)
81+
82+
`lib/isocpp.py` has an internal queue. ALL mutating operations
83+
(upload, transition) go through `submit(job)`.
84+
85+
- Worker thread processes one job at a time
86+
- `threading.Lock` serializes all HTTP session access
87+
- Single `on_event` callback fires for every state change
88+
- Events: job_queued, job_started, job_completed, job_failed, queue_drained
89+
- NEVER bypass the queue. NEVER call the session directly.
90+
91+
## Two Pages
92+
93+
- `/` - Dashboard: top bar (auth + summary pills), paper table, log, bottom status bar
94+
- `/settings` - Settings: output dir, watch dirs, credentials, style
95+
96+
Both pages use SSE (`/api/events`) for live updates. The SSE
97+
connection is per-page - it reconnects on navigation. State is
98+
server-side, fetched on page load via API.
99+
100+
## Pipeline
101+
102+
markdown changed -> watchdog -> render worker -> PDF created ->
103+
PDF watchdog -> dirty flag -> user clicks Upload -> IsoCppSession
104+
queue -> isocpp.org
105+
106+
## Button UX Rules (TODO - not yet fully implemented)
107+
108+
Buttons that submit work to the IsoCppSession queue (docketeer)
109+
or to the render worker follow this pattern:
110+
111+
1. **Press** -> button enters working state immediately:
112+
- Disabled (no re-click)
113+
- Animated glowing border (badge-working style)
114+
- 70% opacity
115+
2. **Log** -> IsoCppSession's on_event fires job_queued, which
116+
the server logs automatically. Render worker logs "Starting..."
117+
3. **Event** -> button stays working until an SSE event confirms
118+
completion (job_completed, job_failed, rendered, render_done)
119+
4. **Done** -> log entry for completion, button re-enables
120+
5. **Log styling** -> completion entries at 50% opacity to
121+
distinguish from active/error entries
122+
123+
Buttons that go through the queue:
124+
- Upload (IsoCppSession.submit upload - syncs title, author, abstract + PDF)
125+
- Draft/Review transition (IsoCppSession.submit transition)
126+
- Render per-paper (render worker)
127+
- Render All (render worker)
128+
- Log In (blocks on login request, not queued but same UX)
129+
130+
Buttons that do NOT go through the queue (instant, no working
131+
state needed): Log Out, Clear Log, Open Folder, Shut Down,
132+
Save (settings), tab switches.
133+
134+
## No Public Scraping
135+
136+
All remote data comes from the authenticated isocpp.org session.
137+
Do not add public scraping of open-std.org or any other site.

paperworks/assets/file-md.svg

Lines changed: 3 additions & 0 deletions
Loading

paperworks/assets/file-pdf.svg

Lines changed: 10 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)