Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 10 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,7 @@ The pipeline uses:
- **Qwen2-VL-7B** for vision-based OCR of image posters
- **pdfplumber** for layout-aware PDF text extraction
- **lingua-language-detector** for ISO 639-1 language detection on body text (overrides any value the model emits — body text beats metadata-fragment guessing)
- **ROR** (`https://api.ror.org`) for affiliation and publisher canonicalisation; matched names get a ROR identifier attached
- **SPDX** matching (with integer-exact version handling) for license normalisation in `rightsList`
- **ROR** (`https://api.ror.org`) for affiliation canonicalisation; matched names get a ROR identifier attached

## Quick Start

Expand Down Expand Up @@ -124,34 +123,25 @@ Output conforms to the [poster-json-schema](https://github.com/fairdataihub/post
"name": "Stanford University",
"affiliationIdentifier": "https://ror.org/00f54p054",
"affiliationIdentifierScheme": "ROR",
"schemeUri": "https://ror.org/"
"schemeURI": "https://ror.org/"
}
]
}
],
"titles": [
{ "title": "Machine Learning Approaches to Diabetic Retinopathy Detection" }
],
"publicationYear": 2025,
"publicationYear": null,
"language": "en",
"researchField": "Health Sciences",
"subjects": [
{ "subject": "Machine Learning" },
{ "subject": "Diabetic Retinopathy" }
],
"descriptions": [
{ "description": "We present a deep learning model...", "descriptionType": "Abstract" }
],
"publisher": { "name": "Zenodo" },
"rightsList": [
{
"rights": "Creative Commons Attribution 4.0 International",
"rightsIdentifier": "CC-BY-4.0",
"rightsIdentifierScheme": "SPDX",
"schemeUri": "https://spdx.org/licenses/",
"rightsUri": "https://creativecommons.org/licenses/by/4.0/"
}
{ "description": "We present a deep learning model...", "descriptionType": "Other" }
],
"publisher": null,
"content": {
"sections": [
{ "sectionTitle": "Abstract", "sectionContent": "..." },
Expand All @@ -160,15 +150,17 @@ Output conforms to the [poster-json-schema](https://github.com/fairdataihub/post
]
},
"imageCaptions": [{ "id": "fig1", "caption": "Figure 1. ROC curves showing..." }],
"tableCaptions": [{ "id": "table1", "caption": "Table 1. Performance metrics" }]
"tableCaptions": [{ "id": "table1", "caption": "Table 1. Performance metrics" }],
"formats": ["application/pdf"]
}
```

Notes on the auto-populated fields:
- `language` is detected from the raw body text (lingua heuristic). Returns null when text is too short (<200 chars / <50 non-ASCII codepoints) or the detector is unsure.
- `researchField` must be one of the four OpenAlex top-level domains: `Health Sciences`, `Life Sciences`, `Physical Sciences`, `Social Sciences`. Null when the model can't pick one confidently.
- `affiliation` and `publisher` get ROR enrichment when the matcher returns a high-confidence chosen result. Strings without a confident match pass through unchanged. Set `POSTER2JSON_ROR=0` to disable.
- `rightsList` entries are matched against an SPDX table; the matcher is conservative on version numbers (e.g. `CC-BY-4.0` and `CC-BY-4.1` are never confused).
- `affiliation` gets ROR enrichment when the matcher returns a high-confidence chosen result. Strings without a confident match pass through unchanged. Set `POSTER2JSON_ROR=0` to disable.
- `publisher` and `publicationYear` are always emitted as `null`. They are platform-owned and set when the poster is published, not by extraction.
- `formats` is derived from the input file's extension, not the model.

## System Requirements

Expand Down
16 changes: 8 additions & 8 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,23 +127,23 @@ ORCIDs are extracted from poster text via regex, then matched to the appropriate

### ROR enrichment

Affiliation names are looked up against the [ROR API](https://ror.org). When a match is found, `affiliationIdentifier`, `affiliationIdentifierScheme`, and `schemeUri` are populated.
Affiliation names are looked up against the [ROR API](https://ror.org). When a match is found, `affiliationIdentifier`, `affiliationIdentifierScheme`, and `schemeURI` are populated.

### Publisher enrichment
### Publisher

If a publisher name is extracted, it is also looked up against ROR to populate `publisherIdentifier`, `publisherIdentifierScheme`, and `schemeURI`.
`publisher` is always emitted as `null`. It is platform-owned and assigned when the poster is published (the hosting repository), so extraction does not populate or enrich it.

### Language detection

The `language` field is detected from the raw poster text using the `lingua` language detector, overwriting any value the LLM may have produced. The result is an ISO 639-1 code (e.g., "en"), or null when the text is too short (<200 chars / <50 non-ASCII codepoints) or the detector is unsure.

### Description type

The LLM prompt instructs the model to classify `descriptionType` based on poster content. It defaults to "Abstract" for poster summaries, but the model can choose from the full set of DataCite description types (Abstract, Methods, SeriesInformation, TableOfContents, TechnicalInfo, Other).
`descriptionType` is not requested from the model; it is set deterministically to `"Other"`, since the description is a machine-generated summary. `"Abstract"` is reserved for an author-provided formal abstract, which the platform attaches downstream.

### Rights normalization
### Rights

License strings from the LLM are canonicalized to SPDX form. This includes alias matching, Creative Commons URL parsing, and fuzzy matching (Levenshtein distance 1). Junk entries like funding acknowledgments or boilerplate text are filtered out.
`rightsList` is stripped from the output. The license is chosen by the user at publish time rather than inferred from the poster.

### Funding normalization

Expand Down Expand Up @@ -185,10 +185,10 @@ Outputs conform to [poster-json-schema](https://github.com/fairdataihub/poster-j
]
},
"imageCaptions": [
{"caption": "Figure 1. Description"}
{"id": "fig1", "caption": "Figure 1. Description"}
],
"tableCaptions": [
{"captions": ["Table 1.", "Description"]}
{"id": "table1", "caption": "Table 1. Description"}
]
}
```
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[tool.poetry]

name = "poster2json"
version = "0.9.22"
version = "0.9.23"
description = "Convert scientific posters (PDF/images) to structured JSON metadata using Large Language Models"

packages = [{ include = "poster2json" }]
Expand Down
Loading