Skip to content

Commit ca2b611

Browse files
committed
docs: refresh extraction prompt design and history (for discussion)
Bring docs/prompt-design.md up to date with current main and reopen it as a discussion. Since the original write-up merged, the prompt moved on: - publicationYear is now platform-owned (removed from the model in 0.9.14), not the lone model-owned required field as first written. - descriptionType is set deterministically to "Other" (machine summary; "Abstract" is the author's downstream abstract), and removed from the prompt. - The resolved open questions (publicationYear, descriptionType, the grounding note) are cleared; fundingReferences is recorded as a kept model field that the prompt should request. Two open questions remain for discussion: whether to add fundingReferences to the prompt, and stating the model-ownership boundary as one rule.
1 parent e7571d5 commit ca2b611

1 file changed

Lines changed: 53 additions & 54 deletions

File tree

docs/prompt-design.md

Lines changed: 53 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -33,13 +33,12 @@ what the prompt requests. The schema requires eight top-level fields:
3333
"publisher", "conference", "formats"]
3434
```
3535

36-
Of those eight, the prompt asks the model for **four** (`creators`, `titles`,
37-
`subjects`, `descriptions`) plus `publicationYear`. It does **not** ask for
38-
`publisher`, `conference`, or `formats` at all. Those three are required by the
39-
schema but are filled by the platform and the file system, never guessed by the
40-
model. `publicationYear` is the lone required metadata field still produced by
41-
the model, and that exception is discussed under
42-
[Open questions](#open-questions-for-review).
36+
Of those eight, the prompt asks the model for **four**: `creators`, `titles`,
37+
`subjects`, and `descriptions`. It does **not** ask for `publisher`, `conference`,
38+
`formats`, or `publicationYear`. Those four are required by the schema but are
39+
filled by the platform and the file system, never guessed by the model.
40+
`publicationYear` was the last of them to move off the model; see
41+
[How we got here](#how-we-got-here).
4342

4443
This split is the result of roughly a year of moving fields off the model one at
4544
a time, each time a field turned out to be a hallucination source or was better
@@ -66,12 +65,12 @@ that change ownership are listed; routine normalization is omitted.
6665
| `researchField` | no | LLM | yes (4 domains or null) | placeholder or junk values coerced to null |
6766
| `content.sections[].sectionTitle` / `sectionContent` | no | LLM | yes | dedupe, verbatim content |
6867
| `imageCaptions[]` / `tableCaptions[]` | no | LLM | yes | normalized to caption arrays |
69-
| `publicationYear` | yes | platform (tracker) / LLM (prompt) | yes (grounded) | not overridden (see open questions) |
7068

7169
### Fields the model is deliberately not asked for
7270

7371
| Schema field | Required | Tracker source | In prompt | Post-processing |
7472
|---|---|---|---|---|
73+
| `publicationYear` | yes | platform | no | dropped from prompt, force-nulled (set at publish) |
7574
| `publisher` | yes | platform | no | forced to `null` (set at publish) |
7675
| `conference.*` | yes | platform / user | no | stripped from model output |
7776
| `formats[]` | yes | file extension | no | stripped, set from the file's MIME type |
@@ -152,9 +151,11 @@ failure we hit during the year of iteration.
152151
and is later normalized and deduped.
153152

154153
- **`descriptions`: a 3 to 4 sentence summary.** The model writes a short summary
155-
of the whole poster. Note that the prompt example tags it
156-
`"descriptionType": "Abstract"`, but post-processing now forces that type
157-
unconditionally, so the model's choice of type is inert (see open questions).
154+
of the whole poster. Only the description text is model-generated; the schema's
155+
`descriptionType` is no longer requested in the prompt and is set to `Other`
156+
deterministically in post-processing, because the summary is machine-generated.
157+
`Abstract` is reserved for the author's own formal abstract, which the platform
158+
attaches downstream, so poster2json never emits it.
158159

159160
### The fallback prompt
160161

@@ -232,11 +233,13 @@ source own a field the model had been guessing.
232233
v0.6.x series). DOIs, ORCIDs, arXiv IDs, ROR, and funder ids are extracted by
233234
regex and looked up via API rather than trusted from the model.
234235

235-
6. **Stop hallucinating `publicationYear`** (commit `6ab31af`, 2026-05-19). The
236-
prompt had a hardcoded example year and a "use the current year if not found"
237-
instruction, which fabricated years at scale. It was changed to null with a
238-
strict "only if explicitly printed, never guess" rule. Importantly,
239-
`publicationYear` stayed a model-owned field; it was grounded, not removed.
236+
6. **`publicationYear` grounded, then removed.** The prompt once had a hardcoded
237+
example year and a "use the current year if not found" instruction, which
238+
fabricated years at scale; commit `6ab31af` (2026-05-19) changed it to null
239+
with a strict "only if explicitly printed" rule. It was later removed from the
240+
model entirely (v0.9.14, commit `79f02d7`): dropped from both prompts and
241+
force-nulled in post-processing, because the platform sets the publication year
242+
at publish time.
240243

241244
7. **`formats` set from the file** (v0.6.x). Dropped from the prompt and set
242245
deterministically from the file extension and MIME type.
@@ -256,8 +259,9 @@ source own a field the model had been guessing.
256259

257260
11. **`version`, `rightsList`, `descriptionType`, conference dates removed**
258261
(v0.9.3). `version` is forced to a provenance string; `rightsList` is stripped
259-
(license is chosen by the user at publish); `descriptionType` is forced to
260-
`Abstract`; conference dates are dropped.
262+
(license is chosen by the user at publish); `descriptionType` is set
263+
deterministically in post-processing (to `Abstract` at the time, corrected to
264+
`Other` in v0.9.15); conference dates are dropped.
261265

262266
12. **The whole `conference` object removed** (v0.9.4). v0.9.3 dropped only the
263267
dates; v0.9.4 dropped the rest. Conference metadata is supplied by the
@@ -270,48 +274,43 @@ source own a field the model had been guessing.
270274
14. **`nameType` derived, not asked for** (v0.9.12). Set to Personal or
271275
Organizational from the presence of given and family names.
272276

273-
The endpoint of all fourteen moves is exactly the "Tracker source" column in the
277+
15. **`descriptionType` removed from the prompt and corrected to `Other`**
278+
(v0.9.15). The model's description is a machine-generated summary, so its type
279+
is `Other`; `Abstract` is reserved for the author's own formal abstract, which
280+
the platform attaches downstream. Post-processing had forced `Abstract` since
281+
v0.9.3; it now sets `Other`, and the prompt no longer asks for the type. (The
282+
value flip-flopped earlier: set to `Other` in `3945d81` on 2026-05-04, reverted
283+
to `Abstract` in `74536e8` on 2026-05-19.)
284+
285+
The endpoint of all these moves is exactly the "Tracker source" column in the
274286
Field Coverage sheet: the fields marked LLM are the ones the prompt still owns,
275287
and everything marked platform, regex, or an API name is a field the prompt was
276288
deliberately relieved of.
277289

278290
## Open questions for review
279291

280-
These are the inconsistencies that surfaced while reconciling the prompt, the
281-
schema, and the coverage tracker. They are the reason this is opened as a
282-
discussion rather than a finished design.
283-
284-
1. **`publicationYear` is the last required metadata field still owned by the
285-
model.** Every one of its peers (publisher, conference, version, rights,
286-
formats, language) was moved to a deterministic or platform source, and the
287-
coverage tracker already lists `publicationYear` as platform-set (current year
288-
at publish). It is grounded in the prompt but, unlike its peers, is never
289-
stripped or overridden in post-processing. If the platform sets it at publish
290-
anyway, should the prompt stop asking for it?
291-
292-
2. **`descriptionType` is asked for but ignored.** The prompt example still tags
293-
the description as `Abstract`, and older versions listed the full type enum for
294-
the model to choose from, but post-processing now forces every description's
295-
type to `Abstract`. The in-prompt instruction is dead weight. Either remove it
296-
from the prompt, or let the model's choice stand.
297-
298-
3. **`fundingReferences` is treated as LLM-sourced everywhere except the prompt.**
299-
The tracker and the post-processing both treat funder name and award number as
300-
model output, but the current prompt does not ask for them. Either add funding
301-
to the prompt explicitly, or reclassify it in the tracker.
302-
303-
4. **The GROUNDING note now names a single field.** The grounding line reads "For
304-
metadata fields (publicationYear), only extract values that appear...". The
305-
parenthetical used to list publisher, conference, version, and publicationYear;
306-
as each was removed it shrank to one. It is a small vestige, and it confirms
307-
that `publicationYear` is the only grounded metadata field left.
308-
309-
5. **The model-owned boundary is not a single rule.** Most of the year's work
310-
moved administrative fields off the model, but `subjects` and `researchField`
311-
were added to the model in the same window. The working principle is that
312-
descriptive and topical content stays with the model while administrative and
313-
provenance metadata moves to deterministic sources. That split is reasonable,
314-
but it is worth stating explicitly so future field decisions have a rule to
292+
Most of the inconsistencies that surfaced when this doc was first drafted have
293+
since been resolved: `publicationYear` was removed from the model (v0.9.14),
294+
`descriptionType` was removed from the prompt and corrected to `Other` (v0.9.15),
295+
and the grounding note no
296+
longer singles out a field. Two items remain.
297+
298+
1. **`fundingReferences` is kept as a model field, and the prompt should ask for
299+
it.** Funder name and award number are LLM-sourced by design: the coverage
300+
tracker lists them as LLM, and post-processing already normalizes funding and
301+
looks up funder ROR identifiers when funding references are present. The gap is
302+
that the current prompt does not request `fundingReferences`, so the model is
303+
only enriched on funding it happens to volunteer. The fix is to add
304+
`fundingReferences` (funderName, awardNumber, awardUri) to the prompt's JSON
305+
shape so funding is extracted on purpose rather than by accident.
306+
307+
2. **The model-ownership boundary is worth stating as one rule.** The working
308+
principle, visible across the whole history, is that descriptive and topical
309+
content (titles, creators, subjects, the summary, sections, captions,
310+
researchField) stays with the model, while administrative and provenance
311+
metadata (publisher, conference, dates, version, formats, language, license,
312+
identifiers, publicationYear) moves to deterministic sources, APIs, or the
313+
platform. Writing that rule down gives future field decisions something to
315314
follow.
316315

317316
## References

0 commit comments

Comments
 (0)