Skip to content

Commit facb2bc

Browse files
author
Jurij Skornik
committed
docs(document-to-markdown): update docs for unpdf default provider
Reflect the provider abstraction with unpdf as default and Mistral as opt-in. Add format support matrix, provider resolution order, REST endpoint, and per-provider capability details.
1 parent f937dd7 commit facb2bc

1 file changed

Lines changed: 95 additions & 81 deletions

File tree

Lines changed: 95 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,51 @@
11
# Document to Markdown Tool
22

3-
Convert PDF, DOCX, and PPTX documents to Markdown using configurable OCR providers.
3+
Convert PDF, DOCX, and PPTX documents to Markdown using configurable conversion providers.
44

55
## Overview
66

7-
The `document-to-markdown` MCP tool uses OCR (Optical Character Recognition) to extract text and images from documents and convert them to well-structured Markdown format. Extracted images are stored as separate blobs alongside the markdown output.
7+
The `document-to-markdown` MCP tool extracts text and images from documents and converts them to well-structured Markdown. Extracted images are stored as separate blobs alongside the markdown output.
88

9-
The tool supports a **provider abstraction** pattern, allowing you to switch between different OCR backends. The default provider is **Mistral OCR**.
9+
The tool supports a **provider abstraction** pattern, allowing you to switch between different conversion backends.
1010

11-
## Requirements
11+
## Available Providers
1212

13-
### Environment Variables
13+
| Provider | Name | Formats | Images | API Key Required |
14+
|----------|------|---------|--------|------------------|
15+
| **unpdf** (default) | `unpdf` | PDF only | No | No |
16+
| **Mistral OCR** | `mistral` | PDF, DOCX, PPTX | Yes | Yes (`MISTRAL_API_KEY`) |
17+
18+
### unpdf (Default)
19+
20+
Zero-config text extraction powered by Mozilla's `pdf.js`. Good for simple PDF text extraction without external API dependencies.
21+
22+
- **Formats**: PDF only
23+
- **Images**: Not supported (returns empty array)
24+
- **Requirements**: None — works out of the box
25+
26+
### Mistral OCR
27+
28+
Full-featured OCR with multi-format support and image extraction. Handles scanned PDFs, complex layouts, and non-PDF formats.
29+
30+
- **Formats**: PDF, DOCX, PPTX
31+
- **Images**: Full extraction support
32+
- **Requirements**: `MISTRAL_API_KEY` ([get one here](https://console.mistral.ai/))
33+
- **Timeout**: 120 seconds per request
34+
35+
## Environment Variables
1436

1537
| Variable | Required | Description |
1638
|----------|----------|-------------|
17-
| `MISTRAL_API_KEY` | **Yes** (for Mistral provider) | Your Mistral API key for OCR processing |
18-
| `DOCUMENT_CONVERSION_PROVIDER` | No | Provider name to use (default: `"mistral"`) |
19-
20-
Get your Mistral API key from [Mistral AI Console](https://console.mistral.ai/).
39+
| `DOCUMENT_CONVERSION_PROVIDER` | No | Provider to use (default: `"unpdf"`) |
40+
| `MISTRAL_API_KEY` | Only for Mistral | Your Mistral API key |
2141

2242
## Supported Formats
2343

24-
| Extension | MIME Type | Description |
25-
|-----------|-----------|-------------|
26-
| `.pdf` | `application/pdf` | PDF documents |
27-
| `.docx` | `application/vnd.openxmlformats-officedocument.wordprocessingml.document` | Microsoft Word documents |
28-
| `.pptx` | `application/vnd.openxmlformats-officedocument.presentationml.presentation` | Microsoft PowerPoint presentations |
44+
| Extension | MIME Type | unpdf | Mistral |
45+
|-----------|-----------|:-----:|:-------:|
46+
| `.pdf` | `application/pdf` | Yes | Yes |
47+
| `.docx` | `application/vnd.openxmlformats-officedocument.wordprocessingml.document` | | Yes |
48+
| `.pptx` | `application/vnd.openxmlformats-officedocument.presentationml.presentation` | | Yes |
2949

3050
## Usage
3151

@@ -42,10 +62,12 @@ Get your Mistral API key from [Mistral AI Console](https://console.mistral.ai/).
4262

4363
| Option | Type | Default | Description |
4464
|--------|------|---------|-------------|
45-
| `pageStart` | number | 1 | First page to process (1-indexed) |
46-
| `pageEnd` | number | Last page | Last page to process (inclusive) |
65+
| `pageStart` | number | 1 | First page to process (1-indexed, inclusive) |
66+
| `pageEnd` | number | Last page | Last page to process (1-indexed, inclusive) |
4767
| `includeImages` | boolean | true | Whether to extract and store images |
4868

69+
Page range values are automatically clamped to valid bounds — out-of-range values are silently adjusted.
70+
4971
### Example: Using blob ID
5072

5173
```json
@@ -75,18 +97,22 @@ Get your Mistral API key from [Mistral AI Console](https://console.mistral.ai/).
7597
}
7698
```
7799

100+
### REST Endpoint
101+
102+
`POST /document-to-markdown` accepts `multipart/form-data` with a `file` field. Returns the same response structure as the MCP tool.
103+
78104
## Output Structure
79105

80106
### Response Format
81107

82108
The tool returns a text response containing:
83109

84-
1. **Status message** - Success or failure indication
85-
2. **Output folder ID** - UUID of the folder containing all outputs
86-
3. **Markdown blob ID** - ID of the generated markdown file
87-
4. **Page count** - Number of pages processed
88-
5. **Image count** - Number of images extracted (if any)
89-
6. **Markdown content** - The full converted markdown
110+
1. **Status message** Success or failure indication
111+
2. **Output folder ID** UUID of the folder containing all outputs
112+
3. **Markdown blob ID** ID of the generated markdown file
113+
4. **Page count** Number of pages in the document
114+
5. **Image count** Number of images extracted (if any)
115+
6. **Markdown content** The full converted markdown
90116

91117
### File Organization
92118

@@ -101,10 +127,6 @@ document-conversions/
101127
└── ...
102128
```
103129

104-
- **Folder ID**: Generated UUID to group related files
105-
- **Markdown file**: Named after the original document (e.g., `report.md`)
106-
- **Images**: Named by Mistral (e.g., `img-0.jpeg`, `img-1.jpeg`), format preserved
107-
108130
### Image References
109131

110132
Images in the markdown reference their blob URLs:
@@ -115,55 +137,45 @@ Images in the markdown reference their blob URLs:
115137

116138
### Storage Location
117139

118-
All outputs (markdown files and extracted images) are stored in the DKG node's configured blob storage under the `document-conversions/` prefix for organized grouping. The physical location depends on your blob storage backend:
140+
All outputs are stored in blob storage under the `document-conversions/` prefix:
119141

120142
| Backend | Typical Location |
121143
|---------|------------------|
122144
| Filesystem (`createFsBlobStorage`) | `./data/blobs/document-conversions/{uuid}/` |
123145
| In-memory (testing only) | RAM, not persisted |
124146
| Custom implementation | As configured |
125147

126-
Files are organized in nested folders:
127-
- `document-conversions/{uuid}/{original-name}.md` — The converted markdown
128-
- `document-conversions/{uuid}/img-0.jpeg` — Extracted images (named by Mistral)
129-
130148
## Error Messages
131149

132150
| Error | Cause | Solution |
133151
|-------|-------|----------|
134-
| `MISTRAL_API_KEY environment variable is not set` | Missing API key | Set the `MISTRAL_API_KEY` environment variable |
152+
| `MISTRAL_API_KEY environment variable is not set` | Missing API key (Mistral provider) | Set the `MISTRAL_API_KEY` environment variable |
135153
| `Either 'blobId' or 'fileBase64' must be provided` | No input document | Provide either `blobId` or `fileBase64` |
136154
| `Provide either 'blobId' or 'fileBase64', not both` | Both inputs provided | Use only one input method |
137-
| `Unsupported file type: '.xyz'` | Invalid file extension | Use PDF, DOCX, or PPTX files |
155+
| `Unsupported file type: '.xyz'` | Invalid file extension | Use a supported format for your provider |
138156
| `File size (X MB) exceeds maximum of 50MB` | File too large | Use a smaller file or split the document |
139157
| `Document blob not found: {id}` | Invalid blob ID | Verify the blob ID exists |
140-
| `Mistral OCR request timed out after 120 seconds` | API timeout | The document may be too large or complex; try a smaller file |
141-
| `Document conversion failed: {reason}` | OCR processing error | Check the error details and retry |
158+
| `Mistral OCR request timed out after 120 seconds` | API timeout (Mistral) | The document may be too large or complex; try a smaller file |
159+
| `Document conversion failed: {reason}` | Processing error | Check the error details and retry |
142160

143161
## Limitations
144162

145163
- **Maximum file size**: 50 MB
146-
- **Supported formats**: PDF, DOCX, PPTX only
147-
- **API timeout**: 2 minutes (120 seconds)
148-
- **API dependency**: Requires active OCR provider API connection
149-
- **Processing time**: Large documents may take longer to process
150-
151-
## Provider Abstraction
152-
153-
The tool uses a provider abstraction pattern that allows swapping OCR backends without changing consuming code.
154-
155-
### Available Providers
164+
- **Supported formats**: PDF (all providers), DOCX/PPTX (Mistral only)
165+
- **API timeout**: 2 minutes (Mistral provider)
166+
- **Image extraction**: Mistral provider only
156167

157-
| Provider | Name | Environment Variables |
158-
|----------|------|----------------------|
159-
| Mistral OCR | `mistral` | `MISTRAL_API_KEY` |
168+
## Provider Configuration
160169

161170
### Selecting a Provider
162171

163172
**Via environment variable:**
164173

165174
```bash
166-
# Use Mistral (default)
175+
# Use unpdf (default) — no API key needed
176+
export DOCUMENT_CONVERSION_PROVIDER=unpdf
177+
178+
# Use Mistral OCR
167179
export DOCUMENT_CONVERSION_PROVIDER=mistral
168180
export MISTRAL_API_KEY=your-api-key
169181
```
@@ -185,41 +197,21 @@ const plugin = createDocumentToMarkdownPlugin({
185197
});
186198
```
187199

188-
### Exported Types and Utilities
189-
190-
The package exports types and utilities for working with providers:
191-
192-
```typescript
193-
import {
194-
// Types
195-
DocumentConversionProvider,
196-
DocumentConversionOptions,
197-
DocumentConversionOutput,
198-
ConversionResult,
199-
ExtractedImage,
200-
DocumentConversionConfig,
201-
202-
// Provider utilities
203-
createProvider,
204-
getDefaultProvider,
205-
getAvailableProviders,
206-
isProviderAvailable,
207-
208-
// Mistral-specific
209-
MistralProvider,
210-
createMistralProvider,
211-
} from "@dkg/plugin-dkg-essentials";
212-
```
200+
**Provider resolution order** (first match wins):
201+
1. Custom provider instance via `config.provider`
202+
2. Provider name from `config.providerName`
203+
3. `DOCUMENT_CONVERSION_PROVIDER` environment variable
204+
4. Default: `"unpdf"`
213205

214206
### Implementing a Custom Provider
215207

216-
To add a new OCR provider, implement the `DocumentConversionProvider` interface:
208+
Implement the `DocumentConversionProvider` interface:
217209

218210
```typescript
219-
import type {
211+
import type {
220212
DocumentConversionProvider,
221213
DocumentConversionOutput,
222-
DocumentConversionOptions
214+
DocumentConversionOptions
223215
} from "@dkg/plugin-dkg-essentials";
224216

225217
class MyCustomProvider implements DocumentConversionProvider {
@@ -230,7 +222,6 @@ class MyCustomProvider implements DocumentConversionProvider {
230222
filename: string,
231223
options?: DocumentConversionOptions,
232224
): Promise<DocumentConversionOutput> {
233-
// Your OCR implementation here
234225
return {
235226
markdown: "# Converted content",
236227
images: [],
@@ -239,12 +230,35 @@ class MyCustomProvider implements DocumentConversionProvider {
239230
}
240231
}
241232

242-
// Use with the plugin
243233
const plugin = createDocumentToMarkdownPlugin({
244234
provider: new MyCustomProvider(),
245235
});
246236
```
247237

238+
### Exported Types and Utilities
239+
240+
```typescript
241+
import {
242+
// Types
243+
DocumentConversionProvider,
244+
DocumentConversionOptions,
245+
DocumentConversionOutput,
246+
ConversionResult,
247+
ExtractedImage,
248+
DocumentConversionConfig,
249+
250+
// Provider utilities
251+
createProvider,
252+
getDefaultProvider,
253+
getAvailableProviders,
254+
isProviderAvailable,
255+
256+
// Provider-specific
257+
MistralProvider,
258+
createMistralProvider,
259+
} from "@dkg/plugin-dkg-essentials";
260+
```
261+
248262
## Security
249263

250264
- API keys are validated at runtime and never logged
@@ -253,5 +267,5 @@ const plugin = createDocumentToMarkdownPlugin({
253267

254268
## Related Tools
255269

256-
- `upload` - Upload documents to blob storage before conversion
257-
- `dkg-create` - Create Knowledge Assets from converted content
270+
- `upload` Upload documents to blob storage before conversion
271+
- `dkg-create` Create Knowledge Assets from converted content

0 commit comments

Comments
 (0)