Switch to new together annotator.#1053
Conversation
wpietri
had a problem deploying
to
Scheduled Testing
GitHub Actions
Failure
wpietri
had a problem deploying
to
Scheduled Testing
GitHub Actions
Failure
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
bkorycki
left a comment
bkorycki
left a comment
There was a problem hiding this comment.
Sorry for a couple of nits. I'm just worried about the code getting messy.
| return f"InjectSecret({self.secret_class.__name__})" | ||
|
|
||
|
|
||
| class InjectAllSecrets(Injector): |
There was a problem hiding this comment.
Is this necessary? It doesn't look like it behaves the same as a regular secret injector
There was a problem hiding this comment.
Yes. I'm adding the ability to include SUTs as annotators via the registry, and so it needs all the secrets to work. See the matching branch: https://github.com/mlcommons/modelbench-private/compare/main...annotator-experiments
There was a problem hiding this comment.
I still don't understand nor love it but given the urgency it's fine
| **options.model_dump(), | ||
| ) | ||
|
|
||
| def translate_chat_prompt(self, prompt: ChatPrompt, options: SUTOptions) -> RequestType: |
There was a problem hiding this comment.
The return type should be HuggingFaceChatCompletionRequest
3 participants
There's a matching branch in modelbench-private for the changes there.