PoC: dynamic SUTs

[rogthefrog]

As an operator, I want to be able to pass a sensible name for a SUT that isn't registered to modelgauge or modelbench
Modelgauge and modelbench should try to create a new SUT from that name and run the requested job with it
So the code doesn't need to be updated whenever we want to add a new SUT that behaves like known SUTs.

Story

Important

This only works with huggingface proxied inference endpoints and serverless together models for now. The dynamic SUT names must be

<vendor>/<model>:<provider>:hfrelay:<date>

UID spec

e.g.

google/gemma-3-27b-it:nebius:hfrelay:20250507

How To Test Huggingface Proxied Inference Providers

deepseek-ai/DeepSeek-V3 is available on several providers (replicate, together, etc).

You can test by requesting any one of those:

poetry run modelbench benchmark -m 1 -s deepseek-ai/DeepSeek-V3:novita:hfrelay
poetry run modelbench benchmark -m 1 -s deepseek-ai/DeepSeek-V3:together:hfrelay

How To Test Together

poetry run modelbench benchmark -m1 -s Qwen/Qwen3-235B-A22B-fp8-tput:together

Additional Information

SUT IDs And Names Behave The Same Way

You can pass a dynamic SUT name the same way you can pass a known SUT ID.

poetry run modelbench benchmark -m 1 -s deepseek-ai/DeepSeek-V3:novita:hfrelay -s llama-3-70b-chat

If you pass in a dynamic SUT name that translates to a known SUT, that's fine, it'll work.

How Do I Find Models With Serverless Providers?

import huggingface_hub as hfh
name = "deepseek-ai/DeepSeek-V3"
inference_providers = hfh.model_info(name, expand="inferenceProviderMapping")
print(inference_providers.inference_provider_mapping)

Alternate Providers

If you change find_alternative to True in huggingface_sut_maker. HuggingFaceChatCompletionServerlessSUTFactory.find then the program will automatically look for the same model on a different serverless provider, if the provider you requested isn't available.

Dedicated Endpoints

Those are TBD. The code only turns one on if it already exists and is asleep. It will not create a new one. That's easy to add.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[wpietri]

I'd love to see some tests that explain what happens in a few different use cases:

1. user runs a benchmark with a fully valid sut
2. user runs a benchmark with a valid provider and sub provider, but an invalid name
3. user runs a benchmark with a valid provider but invalid subprovider
4. user runs a benchmark with a totally bogus id

I also have some questions about the way the foo/bar/baz request turns into a SUT uid of baz-foo-bar. A very common operation to is to re-run a SUT in modellab, and another thing we'll likely do is look through results, find a bunch of models, and re-run them. Will those work with this? I was thinking the first form would be a new valid SUT uid, but if we're going for something more consistent with our existing SUTs, should we just make it a postfix-with-dashes notation in all spots?

[rogthefrog]

@wpietri excellent suggestions.

user runs a benchmark with a totally bogus id

Test added.

user runs a benchmark with a fully valid sut

We already have that with a yes-no local sut and a fake sut, which have no external dependencies. Once a dynamic sut can be registered in the sut registry on the fly and instantiated in make_suts, then it's just a sut like all the others. I added tests for register_dynamic_suts and make_suts with good and bad dynamic sut names.

We don't have tests for benchmark runs with any of the remote suts in the registry, because we'd have to mock those suts functionality, and I don't think it's in the scope of the benchmark runner tests to test those.

It may be a good thing to add to a smoke test, however.

user runs a benchmark with a valid provider and sub provider, but an invalid name
user runs a benchmark with a valid provider but invalid subprovider

These are just bogus suts that can't be registered, so they will fail at that stage, and we have a test for that.

[wpietri]

These are just bogus suts that can't be registered, so they will fail at that stage, and we have a test for that.

My goal here isn't just to have a failure, but to have failures that convey the different problems, at least as far as we can detect them. That way the user has a better chance of fixing the mistake.

[bkorycki]

This looks great! One thing I'm a little concerned about is the conflation between SUT uids and names.
I think we should either a) go back to when you had a separate argument to pass in dynamic SUT names or b) request dynamic suts at the cli using a structured UID (with hyphens) instead of a name.

I also don't like that typos in SUT uids will be identified as dynamic SUT names and then result in an inaccurate error message (i.e. "couldnt make this dynamic sut" instead of "invalid SUT UID, please use one of the following..")

[bkorycki]

I think this belongs in modelgauge. That's where we keep SUT-registration code. I would also like to eventually be able to run dynamic suts with the modelgauge CLI (especially run-sut, which is a helpful debugging tool), which would also use this function.