Skip to content

Commit f806f8d

Browse files
lfoppianolfoppiano
authored
Markdown output
Added markdown output, revised over two iterations
1 parent 7cd66c0 commit f806f8d

6 files changed

Lines changed: 1042 additions & 1122 deletions

File tree

‎Readme.md‎

Lines changed: 148 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,8 @@
44
[![SWH](https://archive.softwareheritage.org/badge/origin/https://github.com/kermitt2/grobid_client_python/)](https://archive.softwareheritage.org/browse/origin/https://github.com/kermitt2/grobid_client_python/)
55
[![License](http://img.shields.io/:license-apache-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0.html)
66

7-
A simple, efficient Python client for [GROBID](https://github.com/kermitt2/grobid) REST services that provides concurrent processing capabilities for PDF documents, reference strings, and patents.
7+
A simple, efficient Python client for [GROBID](https://github.com/kermitt2/grobid) REST services that provides
8+
concurrent processing capabilities for PDF documents, reference strings, and patents.
89

910
## 📋 Table of Contents
1011

@@ -13,8 +14,8 @@ A simple, efficient Python client for [GROBID](https://github.com/kermitt2/grobi
1314
- [Installation](#-installation)
1415
- [Quick Start](#-quick-start)
1516
- [Usage](#-usage)
16-
- [Command Line Interface](#command-line-interface)
17-
- [Python Library](#python-library)
17+
- [Command Line Interface](#command-line-interface)
18+
- [Python Library](#python-library)
1819
- [Configuration](#-configuration)
1920
- [Services](#-services)
2021
- [Testing](#-testing)
@@ -31,15 +32,17 @@ A simple, efficient Python client for [GROBID](https://github.com/kermitt2/grobi
3132
- **Coordinate Extraction**: Optional PDF coordinate extraction for precise element positioning
3233
- **Sentence Segmentation**: Layout-aware sentence segmentation capabilities
3334
- **JSON Output**: Convert TEI XML output to structured JSON format with CORD-19-like structure
35+
- **Markdown Output**: Convert TEI XML output to clean Markdown format with structured sections
3436

3537
## 📋 Prerequisites
3638

3739
- **Python**: 3.8 - 3.13 (tested versions)
3840
- **GROBID Server**: A running GROBID service instance
39-
- Local installation: [GROBID Documentation](http://grobid.readthedocs.io/)
40-
- Docker: `docker run -t --rm -p 8070:8070 lfoppiano/grobid:0.8.2`
41-
- Default server: `http://localhost:8070`
42-
- Online demo: https://lfoppiano-grobid.hf.space (usage limits apply), more details [here](https://grobid.readthedocs.io/en/latest/getting_started/#using-grobid-from-the-cloud).
41+
- Local installation: [GROBID Documentation](http://grobid.readthedocs.io/)
42+
- Docker: `docker run -t --rm -p 8070:8070 lfoppiano/grobid:0.8.2`
43+
- Default server: `http://localhost:8070`
44+
- Online demo: https://lfoppiano-grobid.hf.space (usage limits apply), more
45+
details [here](https://grobid.readthedocs.io/en/latest/getting_started/#using-grobid-from-the-cloud).
4346

4447

4548
> [!IMPORTANT]
@@ -51,16 +54,19 @@ A simple, efficient Python client for [GROBID](https://github.com/kermitt2/grobi
5154
Choose one of the following installation methods:
5255

5356
### PyPI (Recommended)
57+
5458
```bash
5559
pip install grobid-client-python
5660
```
5761

5862
### Development Version
63+
5964
```bash
6065
pip install git+https://github.com/kermitt2/grobid_client_python.git
6166
```
6267

6368
### Local Development
69+
6470
```bash
6571
git clone https://github.com/kermitt2/grobid_client_python
6672
cd grobid_client_python
@@ -70,6 +76,7 @@ pip install -e .
7076
## ⚡ Quick Start
7177

7278
### Command Line
79+
7380
```bash
7481
# Process PDFs in a directory
7582
grobid_client --input ./pdfs --output ./output processFulltextDocument
@@ -79,6 +86,7 @@ grobid_client --server https://your-grobid-server.com --input ./pdfs processFull
7986
```
8087

8188
### Python Library
89+
8290
```python
8391
from grobid_client.grobid_client import GrobidClient
8492

@@ -135,6 +143,7 @@ grobid_client [OPTIONS] SERVICE
135143
| `--segmentSentences` | Segment sentences with coordinates |
136144
| `--flavor` | Processing flavor for fulltext extraction |
137145
| `--json` | Convert TEI output to JSON format |
146+
| `--markdown` | Convert TEI output to Markdown format |
138147

139148

140149
#### Examples
@@ -149,6 +158,9 @@ grobid_client --input ~/pdfs --output ~/tei --n 20 --teiCoordinates processFullt
149158
# Process with JSON output
150159
grobid_client --input ~/pdfs --output ~/results --json processFulltextDocument
151160

161+
# Process with Markdown output
162+
grobid_client --input ~/pdfs --output ~/results --markdown processFulltextDocument
163+
152164
# Process citations with custom server
153165
grobid_client --server https://grobid.example.com --input ~/citations.txt processCitationList
154166

@@ -204,6 +216,14 @@ client.process(
204216
json_output=True
205217
)
206218

219+
# Process with Markdown output
220+
client.process(
221+
service="processFulltextDocument",
222+
input_path="/path/to/pdfs",
223+
output_path="/path/to/output",
224+
markdown_output=True
225+
)
226+
207227
# Process citation lists
208228
client.process(
209229
service="processCitationList",
@@ -214,17 +234,25 @@ client.process(
214234

215235
## ⚙️ Configuration
216236

217-
Configuration can be provided via a JSON file. When using the CLI, the `--server` argument overrides the config file settings.
237+
Configuration can be provided via a JSON file. When using the CLI, the `--server` argument overrides the config file
238+
settings.
218239

219240
### Default Configuration
220241

221242
```json
222243
{
223-
"grobid_server": "http://localhost:8070",
224-
"batch_size": 1000,
225-
"sleep_time": 5,
226-
"timeout": 60,
227-
"coordinates": ["persName", "figure", "ref", "biblStruct", "formula", "s"]
244+
"grobid_server": "http://localhost:8070",
245+
"batch_size": 1000,
246+
"sleep_time": 5,
247+
"timeout": 60,
248+
"coordinates": [
249+
"persName",
250+
"figure",
251+
"ref",
252+
"biblStruct",
253+
"formula",
254+
"s"
255+
]
228256
}
229257
```
230258

@@ -314,6 +342,7 @@ The config file can include logging settings:
314342
## 🔬 Services
315343

316344
### Fulltext Document Processing
345+
317346
Extracts complete document structure including headers, body text, figures, tables, and references.
318347

319348
```bash
@@ -336,11 +365,16 @@ When using the `--json` flag, the client converts TEI XML output to a structured
336365
"level": "paragraph",
337366
"biblio": {
338367
"title": "Document Title",
339-
"authors": ["Author 1", "Author 2"],
368+
"authors": [
369+
"Author 1",
370+
"Author 2"
371+
],
340372
"doi": "10.1000/example",
341373
"publication_date": "2023-01-01",
342374
"journal": "Journal Name",
343-
"abstract": [...]
375+
"abstract": [
376+
...
377+
]
344378
},
345379
"body_text": [
346380
{
@@ -365,8 +399,16 @@ When using the `--json` flag, the client converts TEI XML output to a structured
365399
"label": "Table 1",
366400
"head": "Sample Data",
367401
"content": {
368-
"headers": ["Header 1", "Header 2"],
369-
"rows": [["Value 1", "Value 2"]],
402+
"headers": [
403+
"Header 1",
404+
"Header 2"
405+
],
406+
"rows": [
407+
[
408+
"Value 1",
409+
"Value 2"
410+
]
411+
],
370412
"metadata": {
371413
"row_count": 1,
372414
"column_count": 2,
@@ -399,23 +441,107 @@ client.process(
399441
```
400442

401443
> [!NOTE]
402-
> When using `--json`, the `--force` flag only checks for existing TEI files. If a TEI file is rewritten (due to `--force`), the corresponding JSON file is automatically rewritten as well.
444+
> When using `--json`, the `--force` flag only checks for existing TEI files. If a TEI file is rewritten (due to
445+
`--force`), the corresponding JSON file is automatically rewritten as well.
446+
447+
### Markdown Output Format
448+
449+
When using the `--markdown` flag, the client converts TEI XML output to a clean, readable Markdown format. This
450+
provides:
451+
452+
- **Structured Sections**: Title, Authors, Affiliations, Publication Date, Fulltext, Annex, and References
453+
- **Clean Formatting**: Human-readable format suitable for documentation and sharing
454+
- **Preserved Content**: All text content with proper section organization
455+
- **Reference Formatting**: Bibliographic references in a readable format
456+
457+
#### Markdown Structure
458+
459+
The generated Markdown follows this structure:
460+
461+
```markdown
462+
# Document Title
463+
464+
## Authors
465+
466+
- Author Name 1
467+
- Author Name 2
468+
469+
## Affiliations
470+
471+
- Affiliation 1
472+
- Affiliation 2
473+
474+
## Publication Date
475+
476+
January 1, 2023
477+
478+
## Fulltext
479+
480+
### Introduction
481+
482+
Content of the introduction section...
483+
484+
### Methods
485+
486+
Content of the methods section...
487+
488+
## Annex
489+
490+
### Acknowledgements
491+
492+
Acknowledgement text...
493+
494+
### Competing Interests
495+
496+
Competing interests statement...
497+
498+
## References
499+
500+
**[1]** Paper Title. *Author Name*. *Journal Name* (2023).
501+
**[2]** Another Paper. *Author et al.*. *Conference* (2022).
502+
```
503+
504+
#### Usage Examples
505+
506+
```bash
507+
# Generate both TEI and Markdown outputs
508+
grobid_client --input pdfs/ --output results/ --markdown processFulltextDocument
509+
510+
# Markdown output with coordinates and sentence segmentation
511+
grobid_client --input pdfs/ --output results/ --markdown --teiCoordinates --segmentSentences processFulltextDocument
512+
```
513+
514+
```python
515+
# Python library usage
516+
client.process(
517+
service="processFulltextDocument",
518+
input_path="/path/to/pdfs",
519+
output_path="/path/to/output",
520+
markdown_output=True
521+
)
522+
```
523+
524+
> [!NOTE]
525+
> When using `--markdown`, the `--force` flag only checks for existing TEI files. If a TEI file is rewritten (due to `--force`), the corresponding Markdown file is automatically rewritten as well.
403526
404527
### Header Document Processing
528+
405529
Extracts only document metadata (title, authors, abstract, etc.).
406530

407531
```bash
408532
grobid_client --input pdfs/ --output headers/ processHeaderDocument
409533
```
410534

411535
### Reference Processing
536+
412537
Extracts and structures bibliographic references from documents.
413538

414539
```bash
415540
grobid_client --input pdfs/ --output refs/ processReferences
416541
```
417542

418543
### Citation List Processing
544+
419545
Parses raw citation strings from text files.
420546

421547
```bash
@@ -458,6 +584,7 @@ pytest -v
458584
### Continuous Integration
459585

460586
Tests are automatically run via GitHub Actions on:
587+
461588
- Push to main branch
462589
- Pull requests
463590
- Multiple Python versions (3.8-3.13)
@@ -480,7 +607,7 @@ Benchmark results for processing **136 PDFs** (3,443 pages total, ~25 pages per
480607
### Additional Benchmarks
481608

482609
- **Header processing**: 3.74s for 136 PDFs (36 PDF/s) with n=10
483-
- **Reference extraction**: 26.9s for 136 PDFs (5.1 PDF/s) with n=10
610+
- **Reference extraction**: 26.9s for 136 PDFs (5.1 PDF/s) with n=10
484611
- **Citation parsing**: 4.3s for 3,500 citations (814 citations/s) with n=10
485612

486613
## 🛠️ Development
@@ -530,7 +657,8 @@ bump-my-version bump patch
530657

531658
## 📄 License
532659

533-
Distributed under the [Apache 2.0 License](http://www.apache.org/licenses/LICENSE-2.0). See `LICENSE` for more information.
660+
Distributed under the [Apache 2.0 License](http://www.apache.org/licenses/LICENSE-2.0). See `LICENSE` for more
661+
information.
534662

535663
## 👥 Authors & Contact
536664

0 commit comments

Comments
 (0)