Skip to content

Commit f844bbe

Browse files
s-sajid-alis-sajid-ali
committed
add a page on batch inference
 new file: docs/genai/04_how_to_guides/04_batch_inference.md renamed: docs/genai/04_how_to_guides/04_llm_fine_tuning.md -> docs/genai/04_how_to_guides/05_llm_fine_tuning.md
1 parent c8796ec commit f844bbe

2 files changed

Lines changed: 140 additions & 0 deletions

File tree

‎docs/genai/04_how_to_guides/04_batch_inference.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# Batch Inference
2+
3+
When immediate results are not needed, for instance in transforming large datasets of unstructured data with LLMs, batch inference adds convenience while offering lower costs. Typical completion window are 24/48h as LLM inference providers run your workload when the load on the inference server is low. If you are interested in harnessing this feature, reach out to us at [`genai-research-support@nyu.edu`](mailto:genai-research-support@nyu.edu) and we will set up a cloud storage bucket for you.
4+
5+
:::note
6+
Batch processing is only supported for LLMs that can be accessed via the `@vertexai` provider.
7+
:::
8+
9+
## Collect the prompts
10+
You'll collect the prompts you want to send to the LLM as a newline delimited JSON ([JSONLines](https://jsonlines.org/)) where each line contains a single prompt in the OpenAI format. Here's an example we will be using in this example:
11+
```json
12+
{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gemini-3-flash-preview", "messages": [{"role": "user", "content": "Where is NYU located?"}], "max_tokens": 2048}}
13+
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gemini-3-flash-preview", "messages": [{"role": "user", "content": "What resources are available for genAI research at nyu?"}], "max_tokens": 8196}}
14+
```
15+
16+
## Upload them to the GCS bucket
17+
We will upload this via the Portkey client to the GCS ([Google Cloud Storage](https://cloud.google.com/storage)) bucket via the following script:
18+
```python
19+
from portkey_ai import Portkey
20+
21+
# Initialize the Portkey client
22+
portkey = Portkey(
23+
api_key="", # Replace with your Portkey API key
24+
provider="@vertexai",
25+
base_url="https://ai-gateway.apps.cloud.rt.nyu.edu/v1/",
26+
vertex_storage_bucket_name="", # Specify the GCS bucket name
27+
provider_file_name="test_dataset.jsonl", # Specify the file name in GCS
28+
provider_model="gemini-3-flash-preview", # Specify the model to use
29+
)
30+
31+
# Upload a file for batch inference
32+
file = portkey.files.create(file=open("test_dataset.jsonl", mode="rb"), purpose="batch")
33+
34+
print(file)
35+
```
36+
This script will print to standard output the location of the uploaded file, like:
37+
38+
```sh
39+
{
40+
"id": "...",
41+
"bytes": 739,
42+
"created_at": null,
43+
"filename": "test_dataset.jsonl",
44+
"object": "file",
45+
"purpose": "batch",
46+
"status": "processed",
47+
"status_details": null,
48+
"create_at": 1771429900875
49+
}
50+
51+
```
52+
53+
## Submit a batch inference job
54+
55+
We are now ready to submit the batch inference job. Here's a script to do so:
56+
```python
57+
from portkey_ai import Portkey
58+
59+
# Initialize the Portkey client
60+
portkey = Portkey(
61+
api_key="", # Replace with your Portkey API key
62+
provider="@vertexai",
63+
base_url="https://ai-gateway.apps.cloud.rt.nyu.edu/v1/",
64+
vertex_storage_bucket_name="", # Specify the GCS bucket name
65+
provider_model="gemini-3-flash-preview", # Specify the model to use
66+
)
67+
68+
# Create a batch inference job
69+
batch_job = portkey.batches.create(
70+
input_file_id="", # Copy this from the output of the last script
71+
endpoint="/v1/chat/completions", # API endpoint to use
72+
completion_window="24h", # Time window for completion
73+
model="gemini-3-flash-preview",
74+
)
75+
76+
print(batch_job)
77+
```
78+
79+
Upon successful submission, you'll see an `id` field that refers to the job id.
80+
81+
## Query job status
82+
83+
Using the id of the batch inference job, you can query the status by:
84+
85+
```sh
86+
curl -H "x-portkey-api-key: " -H "x-portkey-provider: @vertexai" https://ai-gateway.apps.cloud.rt.nyu.edu/v1/batches/your-batch-inference-job-id
87+
```
88+
89+
The output for a pending job looks like:
90+
```json
91+
{
92+
"id": "7284652661620604928",
93+
"object": "batch",
94+
"endpoint": "/v1/chat/completions",
95+
"input_file_id": "...",
96+
"completion_window": null,
97+
"status": "in_progress",
98+
"output_file_id": "...",
99+
"error_file_id": "...",
100+
"created_at": 1771430154933,
101+
"in_progress_at": 1771430241905,
102+
"request_counts": {
103+
"total": 0,
104+
"completed": null,
105+
"failed": null
106+
},
107+
"model": "publishers/google/models/gemini-3-flash-preview"
108+
}
109+
```
110+
111+
Once the job completes, the `status` field will change from `in_progress` to `completed`.
112+
113+
114+
## Retrieving the output
115+
116+
The output of the batch inference job can be obtained by:
117+
```sh
118+
curl -H "x-portkey-api-key: " -H "x-portkey-provider: @vertexai" https://ai-gateway.apps.cloud.rt.nyu.edu/v1/batches/7284652661620604928/output
119+
```
120+
The output is also a newline delimited json (JSONLines) file that looks like:
121+
```json
122+
{
123+
"id": "batch-7284652661620604928-aOKVaZmJGItT7OsPu-WByQc",
124+
"custom_id": "request-1",
125+
"response": {
126+
"status_code": 200,
127+
"request_id": "batch-7284652661620604928-aOKVaZmJGItT7OsPu-WByQc",
128+
"body": {
129+
"id": "chatcmpl-66109dHQ7PrBtM5Bzi37wXu97lr8m",
130+
"object": "chat.completion",
131+
"created": 1771430797,
132+
"model": "gemini-3-flash-preview",
133+
"provider": "vertex-ai",
134+
"choices": [
135+
{
136+
"message": {
137+
"role": "assistant",
138+
"content": "New York University (NYU) is primarily located in **New York City**
139+
...
140+
```

docs/genai/04_how_to_guides/04_llm_fine_tuning.md renamed to docs/genai/04_how_to_guides/05_llm_fine_tuning.md

File renamed without changes.

0 commit comments

Comments
 (0)