Skip to content

Fix Pydantic issue with cached values.#1076

Merged
wpietri merged 2 commits into
mainfrom
fix-caching-smoke-test
Jun 10, 2025
Merged

Fix Pydantic issue with cached values.#1076
wpietri merged 2 commits into
mainfrom
fix-caching-smoke-test

Conversation

@wpietri

@wpietri wpietri commented Jun 10, 2025

Copy link
Copy Markdown
Contributor

The Llama API is returning things that are invalid according to their Pydantic definitions, but this somehow doesn't show up as a problem until cache retrieval. Here I'm just fixing the data to match its typing. Maybe there's a better option?

@wpietri wpietri requested a review from a team as a code owner June 10, 2025 02:14
@wpietri wpietri temporarily deployed to Scheduled Testing June 10, 2025 02:14 — with GitHub Actions Inactive
@wpietri wpietri temporarily deployed to Scheduled Testing June 10, 2025 02:14 — with GitHub Actions Inactive
@wpietri wpietri temporarily deployed to Scheduled Testing June 10, 2025 02:14 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Jun 10, 2025

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@wpietri

wpietri commented Jun 10, 2025

Copy link
Copy Markdown
Contributor Author

Per Roger, the error:

there's an issue with a meta llama SUT that just went in

E           pydantic_core._pydantic_core.ValidationError: 1 validation error for MetaLlamaModeratedResponse
E           moderation_response.results.0.flagged_categories
E             Input should be a valid list [type=list_type, input_value=None, input_type=NoneType]



FAILED plugins/validation_tests/test_object_creation.py::test_can_cache_all_sut_responses[meta-llama-4-scout-17b-16e-instruct-fp8-moderated-llama] - pydantic_core._pydantic_core.ValidationError: 1 validation error for MetaLlamaModeratedResponse
moderation_response.results.0.flagged_categories
  Input should be a valid list [type=list_type, input_value=None, input_type=NoneType]

```

Comment thread src/modelgauge/suts/meta_llama_client.py

@dhosterman dhosterman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hate this, but it seems right (as far as what is under our control).

@wpietri wpietri temporarily deployed to Scheduled Testing June 10, 2025 14:26 — with GitHub Actions Inactive
@wpietri wpietri temporarily deployed to Scheduled Testing June 10, 2025 14:26 — with GitHub Actions Inactive
@wpietri wpietri temporarily deployed to Scheduled Testing June 10, 2025 14:26 — with GitHub Actions Inactive
@wpietri wpietri merged commit d568e1e into main Jun 10, 2025
4 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 10, 2025
@wpietri wpietri deleted the fix-caching-smoke-test branch January 16, 2026 14:08
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants