Skip to content

Fix PredictInput::Batch and expect pairs as nested tuples#801

Open
alvarobartt wants to merge 4 commits into
mainfrom
allow-batch-predict
Open

Fix PredictInput::Batch and expect pairs as nested tuples#801
alvarobartt wants to merge 4 commits into
mainfrom
allow-batch-predict

Conversation

@alvarobartt

Copy link
Copy Markdown
Member

What does this PR do?

This PR allows a Vec<String> with length > 2 to be provided as PredictInput::Batch, given that it was preventing some routes as /predict on processing with more than 2 sequences as those were considered "pairs".

This PR represents a breaking change on how the /predict method works, as before this PR pairs could be provided as [a,b] or [[a,b]], but the fact that [a,b] was allowed was preventing the inputs to be a batch of sequences, given that it was limited to two sequences, and then interpreted as a pair rather than as two individual sequences.

So on, from now on, only the following are allowed:

  • Single sequences as "inputs": "a"
  • Batch of sequences as "inputs":["a", ...] (allows from 1 to N, defined by the --max-client-batch-size with a default value of 32)
  • Batch of single sequences and/or pairs as "inputs":[["a"], ...] and "inputs":[["a","b"], ...], respectively; or combined as "inputs":[["a"], ["b", "c"], ...].

Warning

It's a breaking change being introduced without a deprecation warning or prior notice, given that treating a batch of sequences as a pair, which implies that the strings there are concatenated into a single sequence, and limiting the length of batches of sequences to 2; can be misleading and/or incorrect.

Since before calling /predict with {"inputs":["I love tennis", "I hate tennis"]} was not running the forward pass on a batch of ["I love tennis", "I hate tennis"], but rather running on a single sequence formed by the concatenation of both sequences in the batch, leading then to a single classification score, rather than an independent score per sequence; and also preventing the user from running with larger batches.

Fixes #747

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline?
  • Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the documentation guidelines.
  • Did you write any new necessary tests? If applicable, did you include or update the insta snapshots?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@alvarobartt alvarobartt added this to the v1.9.0 milestone Jan 10, 2026
Comment thread router/src/http/types.rs Outdated
@alvarobartt alvarobartt modified the milestones: v1.9.0, v1.10.0 Feb 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

/predict does not work for batches for sequence classification

2 participants