Skip to content

Lighter smoke tests#987

Merged
bkorycki merged 4 commits into
mainfrom
lighter-smoketests
Apr 17, 2025
Merged

Lighter smoke tests#987
bkorycki merged 4 commits into
mainfrom
lighter-smoketests

Conversation

@bkorycki
Copy link
Copy Markdown
Contributor

@bkorycki bkorycki commented Apr 17, 2025

This is for card 974: Refactor smoke tests to include only one SUT per provider and only serverless

  • Only include one SUT per class. SUT classes roughly correspond to providers, but some providers have multiple SUT classes (e.g. for different endpoints/apis).
  • Skip SUTs that belong to non-serverless classes ($$). Right now this just includes huggingface inference and baseten.
    • This now requires you to have specific plugins installed to run the validation tests due to imports of class names. It will break without those plugins installed. Maybe this is undesirable?

@bkorycki bkorycki requested review from rogthefrog and wpietri April 17, 2025 19:11
@bkorycki bkorycki requested a review from a team as a code owner April 17, 2025 19:11
@bkorycki bkorycki temporarily deployed to Scheduled Testing April 17, 2025 19:11 — with GitHub Actions Inactive
@bkorycki bkorycki temporarily deployed to Scheduled Testing April 17, 2025 19:11 — with GitHub Actions Inactive
@bkorycki bkorycki temporarily deployed to Scheduled Testing April 17, 2025 19:11 — with GitHub Actions Inactive
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 17, 2025

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

Copy link
Copy Markdown
Contributor

@wpietri wpietri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ideally we'd include one cheap item each for the expensive providers. But we can leave that until one of those providers breaks on us.

Comment thread plugins/validation_tests/test_object_creation.py
@bkorycki bkorycki temporarily deployed to Scheduled Testing April 17, 2025 19:30 — with GitHub Actions Inactive
@bkorycki bkorycki temporarily deployed to Scheduled Testing April 17, 2025 19:30 — with GitHub Actions Inactive
@bkorycki bkorycki temporarily deployed to Scheduled Testing April 17, 2025 19:30 — with GitHub Actions Inactive
@bkorycki bkorycki temporarily deployed to Scheduled Testing April 17, 2025 19:31 — with GitHub Actions Inactive
@bkorycki bkorycki temporarily deployed to Scheduled Testing April 17, 2025 19:31 — with GitHub Actions Inactive
@bkorycki bkorycki temporarily deployed to Scheduled Testing April 17, 2025 19:31 — with GitHub Actions Inactive
@bkorycki bkorycki merged commit 3f6e5e4 into main Apr 17, 2025
4 checks passed
@bkorycki bkorycki deleted the lighter-smoketests branch April 17, 2025 20:54
@github-actions github-actions Bot locked and limited conversation to collaborators Apr 17, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants