|
| 1 | +# Serve Unlimited-OCR as a live endpoint on HF Jobs |
| 2 | + |
| 3 | +The OCR recipes in this folder run as batch jobs (dataset in → dataset out). To call a model |
| 4 | +interactively, from an agent, or with ad-hoc concurrent requests, you can instead run it as a |
| 5 | +temporary HTTP endpoint. [HF Jobs serving](https://huggingface.co/docs/hub/jobs-serving) exposes a |
| 6 | +port on a GPU Job, giving an OpenAI-compatible endpoint that runs until the job is cancelled or its |
| 7 | +`--timeout` is reached. |
| 8 | + |
| 9 | +This is a worked example for [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) |
| 10 | +(3B, MIT, based on DeepSeek-OCR; supports multi-page parsing in a single request). The model ships |
| 11 | +its own SGLang build, so it runs on the stock `lmsysorg/sglang` image with the 12 MB wheel |
| 12 | +installed at startup; no custom image is required. |
| 13 | + |
| 14 | +## 1. Start the server |
| 15 | + |
| 16 | +```bash |
| 17 | +hf jobs run --detach --expose 10000 --flavor h200 -s HF_TOKEN --timeout 30m \ |
| 18 | + lmsysorg/sglang:latest -- \ |
| 19 | + bash -lc 'pip install --no-deps https://github.com/baidu/Unlimited-OCR/raw/main/wheel/sglang-0.0.0.dev11416+g92e8bb79e-py3-none-any.whl \ |
| 20 | + && pip install -q kernels==0.11.7 \ |
| 21 | + && python -m sglang.launch_server --model baidu/Unlimited-OCR --served-model-name Unlimited-OCR \ |
| 22 | + --attention-backend fa3 --page-size 1 --mem-fraction-static 0.8 --context-length 32768 \ |
| 23 | + --enable-custom-logit-processor --disable-overlap-schedule --skip-server-warmup \ |
| 24 | + --host 0.0.0.0 --port 10000' |
| 25 | +``` |
| 26 | + |
| 27 | +Notes: |
| 28 | +- `--` before `bash` is required, or the CLI parses `-lc` as its own flags. |
| 29 | +- `--timeout` stops the endpoint (and billing) at the deadline; `hf jobs cancel <id>` stops it earlier. |
| 30 | +- `fa3` requires a Hopper GPU (e.g. `h200`). The model is small, so the attention backend, not GPU |
| 31 | + memory, determines the flavor. Run `hf jobs hardware` for available flavors. |
| 32 | +- Follow startup with `hf jobs logs -f <id>`; the server is ready at `Application startup complete` |
| 33 | + (about 3 minutes from a cold start). |
| 34 | + |
| 35 | +## 2. Call it (OpenAI client; HF token as the API key) |
| 36 | + |
| 37 | +The exposed port is at `https://<job_id>--10000.hf.jobs`; the OpenAI base URL is that plus `/v1`. |
| 38 | + |
| 39 | +```python |
| 40 | +import base64, os |
| 41 | +from openai import OpenAI |
| 42 | + |
| 43 | +client = OpenAI(base_url="https://<job_id>--10000.hf.jobs/v1", api_key=os.environ["HF_TOKEN"]) |
| 44 | +img = base64.b64encode(open("page.jpg", "rb").read()).decode() |
| 45 | + |
| 46 | +r = client.chat.completions.create( |
| 47 | + model="Unlimited-OCR", |
| 48 | + messages=[{"role": "user", "content": [ |
| 49 | + {"type": "text", "text": "document parsing."}, |
| 50 | + {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img}"}}, |
| 51 | + ]}], |
| 52 | + temperature=0, |
| 53 | + extra_body={"images_config": {"image_mode": "gundam"}}, # "gundam" (crop-tiling) or "base" |
| 54 | +) |
| 55 | +print(r.choices[0].message.content) |
| 56 | +``` |
| 57 | + |
| 58 | +Output is layout-grounded markdown: each block is tagged `<|det|>type [x1,y1,x2,y2]<|/det|> text`, |
| 59 | +with coordinates normalized to 0–1000. Remove the tags for plain text |
| 60 | +(`re.sub(r'<\|det\|>.*?<\|/det\|>', '', text)`) or keep them for structure. |
| 61 | + |
| 62 | +## 3. Multi-page / PDF |
| 63 | + |
| 64 | +Send multiple page images in one request with the `Multi page parsing.` prompt and `image_mode="base"`: |
| 65 | + |
| 66 | +```python |
| 67 | +parts = [{"type": "text", "text": "Multi page parsing."}] |
| 68 | +for page_png in page_images: # e.g. PDF pages rendered with pymupdf at ~150 dpi |
| 69 | + b64 = base64.b64encode(open(page_png, "rb").read()).decode() |
| 70 | + parts.append({"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}}) |
| 71 | + |
| 72 | +r = client.chat.completions.create( |
| 73 | + model="Unlimited-OCR", |
| 74 | + messages=[{"role": "user", "content": parts}], |
| 75 | + temperature=0, max_tokens=16384, |
| 76 | + extra_body={"images_config": {"image_mode": "base"}}, |
| 77 | +) |
| 78 | +``` |
| 79 | + |
| 80 | +Pages are separated by `<PAGE>`; tables are returned as HTML and equations as LaTeX, with reading |
| 81 | +order preserved across pages. The context length is 32k tokens, so split longer documents. |
| 82 | + |
| 83 | +## 4. Concurrency |
| 84 | + |
| 85 | +SGLang batches concurrent requests, so a client can send many requests in parallel to one endpoint; |
| 86 | +the upstream [`infer.py`](https://github.com/baidu/Unlimited-OCR/blob/main/infer.py) uses a |
| 87 | +`ThreadPoolExecutor` at `concurrency=8`. For a large corpus, a batch job that runs next to the data |
| 88 | +(resumable, no network transfer) is usually a better fit than a client-to-endpoint loop. |
| 89 | + |
| 90 | +## 5. Stop it |
| 91 | + |
| 92 | +```bash |
| 93 | +hf jobs cancel <job_id> |
| 94 | +``` |
| 95 | + |
| 96 | +Billing is per-minute for the GPU flavor plus a small flat fee for the exposed port; scheduling time |
| 97 | +is not billed. Run `hf jobs hardware` for current flavors and prices. |
0 commit comments