Skip to content

/predict does not work for batches for sequence classification #747

@vrdn-23

Description

@vrdn-23

System Info

It seems that whenever a set of inputs as batch is being passed in to /predict for sequence classification, only the first result is returned.

Information

  • Docker
  • The CLI directly

Tasks

  • An officially supported command
  • My own modifications

Reproduction

text-embeddings-router --model-id HuggingFaceFW/fineweb-edu-classifier --port 8080
~/dev/open-source-help/text-embeddings-inference main = *4 ?8 > curl -XPOST localhost:8080/predict -H 'Content-Type: application/json' -d '{"inputs": "Butterflies cute"}' | jq                 py 3.11  12:29:07
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    70  100    40  100    30    696    522 --:--:-- --:--:-- --:--:--  1228
[
  {
    "score": 0.50506574,
    "label": "LABEL_0"
  }
]

~/dev/open-source-help/text-embeddings-inference main = *4 ?8 > curl -XPOST localhost:8080/predict -H 'Content-Type: application/json' -d '{"inputs": ["Butterflies cute", "Where can I find the one piece?"]}' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   106  100    39  100    67   1324   2274 --:--:-- --:--:-- --:--:--  3655
[
  {
    "score": 0.5077051,
    "label": "LABEL_0"
  }
]

~/dev/open-source-help/text-embeddings-inference main = *4 ?8 > curl -XPOST localhost:8080/predict -H 'Content-Type: application/json' -d '{"inputs": ["Butterflies cute", "Where can I find the one piece?", "How about now"]}'     
Failed to deserialize the JSON body into the target type: inputs: invalid length 3, expected a string, a pair of strings [string, string] or a batch of mixed strings and pairs [[string], [string, string], ...] at line 1 column 83%   

Expected behavior

TEI should either throw an error /predict endpoint when size is 2 and model does not do pair classification.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions