Skip to content

No batch inference API support #66

@djthorpe

Description

@djthorpe

Anthropic Message Batches API and OpenAI Batch API offer ~50% cost reduction for async workloads
Add BatchGenerator interface: SubmitBatch([]BatchRequest) (BatchID, error) + GetBatch(BatchID) ([]BatchResult, error)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions