|
| 1 | +# Paperworks - Agent Rules |
| 2 | + |
| 3 | +## What This Is |
| 4 | + |
| 5 | +Paperworks is the unified web dashboard for managing WG21 paper |
| 6 | +submissions. It orchestrates two libraries: |
| 7 | + |
| 8 | +- **Scrivener** (`../scrivener/`) - markdown to PDF rendering |
| 9 | +- **Docketeer** (`lib/isocpp.py`) - isocpp.org authenticated client |
| 10 | + with internal queue |
| 11 | + |
| 12 | +Paperworks itself handles the web UI, watchdogs, inventory |
| 13 | +correlation, and orchestration logic. |
| 14 | + |
| 15 | +## Architecture |
| 16 | + |
| 17 | +``` |
| 18 | +paperworks/ |
| 19 | + paperworks.py # CLI: python paperworks.py serve |
| 20 | + lib/ |
| 21 | + server.py # Flask app, SSE, routes, both watchdogs, render worker |
| 22 | + server_config.py # ~/.paperworks/config.json, falls back to scrivener |
| 23 | + inventory.py # Three-source paper database (markdown + PDF + isocpp.org) |
| 24 | + isocpp.py # IsoCppSession - queued, lock-serialized, event-emitting |
| 25 | + pdf_reader.py # PDF metadata extraction via PyMuPDF |
| 26 | + templates/ |
| 27 | + index.html # Dashboard page (main view) |
| 28 | + settings.html # Settings page (config, dirs, credentials) |
| 29 | +``` |
| 30 | + |
| 31 | +## Source of Truth |
| 32 | + |
| 33 | +Markdown is the source of truth for ALL paper metadata. |
| 34 | + |
| 35 | +- **Priority**: markdown > PDF > isocpp.org. If markdown and |
| 36 | + PDF disagree, markdown wins. PDF is a derived artifact. |
| 37 | +- **Metadata from markdown**: title, authors, date, audience |
| 38 | + (YAML front matter), abstract/brutal summary (`## Abstract` |
| 39 | + section in body) |
| 40 | +- **PDF fallback**: if no markdown exists, PDF metadata is used |
| 41 | + but the paper shows `orphan` status |
| 42 | +- **isocpp.org**: provides remote status (Draft/Review/Mailed) |
| 43 | + and form URLs only - never a metadata source |
| 44 | +- **D/P prefixes**: interchangeable for matching. `D4007R0` and |
| 45 | + `P4007R0` refer to the same paper. Papers start as D (draft) |
| 46 | + and become P (published) when mailed. The numeric part and |
| 47 | + revision are what identify a paper. |
| 48 | +- **Primary author**: only the first author from `reply-to` is |
| 49 | + synced to isocpp.org. Multi-author papers list all authors |
| 50 | + locally but only the first goes to the remote form. |
| 51 | +- **Upload syncs everything**: there is no separate "sync |
| 52 | + metadata" operation. UPLOAD pushes the PDF AND syncs title, |
| 53 | + author, and abstract in one action. `_do_upload` skips any |
| 54 | + field that is None/empty - it will never overwrite good |
| 55 | + remote data with nothing. |
| 56 | +- **Warnings**: if a markdown file has valid front matter but |
| 57 | + no `## Abstract` section, the paper gets a warning logged |
| 58 | + and shown in the UI. It does not block rendering or uploading. |
| 59 | + |
| 60 | +## Three-Source Inventory |
| 61 | + |
| 62 | +The paper inventory correlates three sources: |
| 63 | + |
| 64 | +1. **Markdown** (source of truth) - YAML front matter provides |
| 65 | + doc_number, title, date, authors, audience. The `## Abstract` |
| 66 | + section in the body provides the brutal summary. These are |
| 67 | + authoritative. |
| 68 | +2. **PDF directory** - derived artifacts. If PDF mtime < markdown |
| 69 | + mtime, the PDF is stale. PDF can never be newer than markdown. |
| 70 | +3. **isocpp.org** - remote paper list with status (Draft/Review/Mailed) |
| 71 | + |
| 72 | +### Derived Status |
| 73 | + |
| 74 | +- `needs_render` - markdown exists, no PDF or PDF is stale |
| 75 | +- `draft` / `review` - remote status, PDF is current |
| 76 | +- `local` - has PDF, no remote entry |
| 77 | +- `orphan` - has PDF but no markdown source |
| 78 | +- `mailed` - hidden from default view |
| 79 | + |
| 80 | +## IsoCppSession Queue (Critical) |
| 81 | + |
| 82 | +`lib/isocpp.py` has an internal queue. ALL mutating operations |
| 83 | +(upload, transition) go through `submit(job)`. |
| 84 | + |
| 85 | +- Worker thread processes one job at a time |
| 86 | +- `threading.Lock` serializes all HTTP session access |
| 87 | +- Single `on_event` callback fires for every state change |
| 88 | +- Events: job_queued, job_started, job_completed, job_failed, queue_drained |
| 89 | +- NEVER bypass the queue. NEVER call the session directly. |
| 90 | + |
| 91 | +## Two Pages |
| 92 | + |
| 93 | +- `/` - Dashboard: top bar (auth + summary pills), paper table, log, bottom status bar |
| 94 | +- `/settings` - Settings: output dir, watch dirs, credentials, style |
| 95 | + |
| 96 | +Both pages use SSE (`/api/events`) for live updates. The SSE |
| 97 | +connection is per-page - it reconnects on navigation. State is |
| 98 | +server-side, fetched on page load via API. |
| 99 | + |
| 100 | +## Pipeline |
| 101 | + |
| 102 | +markdown changed -> watchdog -> render worker -> PDF created -> |
| 103 | +PDF watchdog -> dirty flag -> user clicks Upload -> IsoCppSession |
| 104 | +queue -> isocpp.org |
| 105 | + |
| 106 | +## Button UX Rules (TODO - not yet fully implemented) |
| 107 | + |
| 108 | +Buttons that submit work to the IsoCppSession queue (docketeer) |
| 109 | +or to the render worker follow this pattern: |
| 110 | + |
| 111 | +1. **Press** -> button enters working state immediately: |
| 112 | + - Disabled (no re-click) |
| 113 | + - Animated glowing border (badge-working style) |
| 114 | + - 70% opacity |
| 115 | +2. **Log** -> IsoCppSession's on_event fires job_queued, which |
| 116 | + the server logs automatically. Render worker logs "Starting..." |
| 117 | +3. **Event** -> button stays working until an SSE event confirms |
| 118 | + completion (job_completed, job_failed, rendered, render_done) |
| 119 | +4. **Done** -> log entry for completion, button re-enables |
| 120 | +5. **Log styling** -> completion entries at 50% opacity to |
| 121 | + distinguish from active/error entries |
| 122 | + |
| 123 | +Buttons that go through the queue: |
| 124 | +- Upload (IsoCppSession.submit upload - syncs title, author, abstract + PDF) |
| 125 | +- Draft/Review transition (IsoCppSession.submit transition) |
| 126 | +- Render per-paper (render worker) |
| 127 | +- Render All (render worker) |
| 128 | +- Log In (blocks on login request, not queued but same UX) |
| 129 | + |
| 130 | +Buttons that do NOT go through the queue (instant, no working |
| 131 | +state needed): Log Out, Clear Log, Open Folder, Shut Down, |
| 132 | +Save (settings), tab switches. |
| 133 | + |
| 134 | +## No Public Scraping |
| 135 | + |
| 136 | +All remote data comes from the authenticated isocpp.org session. |
| 137 | +Do not add public scraping of open-std.org or any other site. |
0 commit comments