Skip to content

ci: Change pytest command#2475

Merged
sjrl merged 3 commits intomainfrom
ci-testing
Oct 30, 2025
Merged

ci: Change pytest command#2475
sjrl merged 3 commits intomainfrom
ci-testing

Conversation

@sjrl
Copy link
Copy Markdown
Contributor

@sjrl sjrl commented Oct 28, 2025

Related Issues

  • fixes #issue-number

Proposed Changes:

I recently noticed in some failing test logs that it looked like we weren't rerunning failing tests despite having pytest-rerunfailures set up. Additionally it appeared like the -x command was being ignored as well.

Here is an example failing test log for meta-llama where we don't appear to rerun the failed test (missing the RERUN log from pytest) and we still continue to run tests after that one failed which goes against the -x option.

It appears something goes wrong when using hatch run test:cov-retry with the configuration

all = 'pytest {args:tests}'
cov-retry = 'all --cov=haystack_integrations --reruns 3 --reruns-delay 30 -x'

when I changed it to (same command as the one we use in haystack-experimental)

cov-retry = 'pytest --cov="haystack_integrations" --reruns 3 --reruns-delay 30 -x {args:tests}'

then I was able to see RERUN logs and early exits occur. Here is an example for meta-llama.

How did you test it?

Notes for the reviewer

Checklist

@sjrl
Copy link
Copy Markdown
Contributor Author

sjrl commented Oct 28, 2025

Failing deepeval tests are unrelated to these changes. They would be fixed by this PR #2476

@sjrl sjrl marked this pull request as ready for review October 28, 2025 08:43
@sjrl sjrl requested a review from a team as a code owner October 28, 2025 08:43
Copy link
Copy Markdown
Member

@anakin87 anakin87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix!

@sjrl sjrl merged commit c287acf into main Oct 30, 2025
196 of 200 checks passed
@sjrl sjrl deleted the ci-testing branch October 30, 2025 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment