Skip to content

evaluate multiple models on the new private sets#511

Draft
KennethEnevoldsen wants to merge 15 commits intomainfrom
private-rteb
Draft

evaluate multiple models on the new private sets#511
KennethEnevoldsen wants to merge 15 commits intomainfrom
private-rteb

Conversation

@KennethEnevoldsen
Copy link
Copy Markdown
Contributor

WIP

Checklist

  • My model has a model sheet, report, or similar
  • My model has a reference implementation in mteb/models/model_implementations/, this can be as an API. Instruction on how to add a model can be found here
    • No, but there is an existing PR ___
  • The results submitted are obtained using the reference implementation with the exception of the nvidia model
  • My model is available, either as a publicly accessible API or publicly on e.g., Huggingface
  • I solemnly swear that for all results submitted I have not trained on the evaluation dataset including training splits. If I have, I have disclosed it clearly.

Co-authored-by: Copilot <copilot@github.com>
@KennethEnevoldsen KennethEnevoldsen marked this pull request as draft May 3, 2026 16:54
Your Name and others added 4 commits May 3, 2026 20:41
…t results and leaving overnight

Co-authored-by: Copilot <copilot@github.com>
"trust_remote_code": true,
"device": "cuda"
},
"name": "nvidia/llama-nemotron-embed-1b-v2",
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, we don't have implementation of this model

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep I know

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(will not merge before we have it)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants