test: Add pytest-xdist to execute tests in parallel on CI (and/or locally)

[mpangrazzi]

Proposed Changes:

I've added pytest-xdist which distributes tests on N different workers for faster execution. On my Mac M3 Pro I can get tests executed ~75% faster. (35s vs 2m 15s on avg).

How did you test it?

unit tests, integration tests

Notes for the reviewer

By default, tests will be still run serially. To run them concurrently, you must add -n option. For example:

hatch run test:integration -n auto  # workers == available cores
hatch run test:all -n 4  # using 4 workers

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

[davidsbatista]

@mpangrazzi thanks for this 👍🏽

Could you quickly check if in our tests that depend on hosted instances, each (os/python version) is using a unique index name, those are the hosted doc stores:

• Astra
• AzureAISearch
• Chroma
• ElasticSearch
• OpenSearch
• PineCone
• Weaviate

[mpangrazzi]

@davidsbatista I would first use this approach on any IT tests which don't use hosted instances, since it's easier to detect issues (if any). For instance, I think it could be applied safely also to ES integration. For others, I would do that without hurry, when one has the chance to work on an integration which has relatively slow tests. WDYT?

cc @anakin87

[davidsbatista]

@davidsbatista I would first use this approach on any IT tests which don't use hosted instances, since it's easier to detect issues (if any). For instance, I think it could be applied safely also to ES integration. For others, I would do that without hurry, when one has the chance to work on an integration which has relatively slow tests. WDYT?

cc @anakin87

agree: ElasticSearch, OpenSearch and Qdrant are all good candidates - high usage and can all run locally

[anakin87]

could we simplify this? I am thinking of generating a UUID or something similar.

[mpangrazzi]

Done. I initially though that having the test name as the index name may help. But on a second though, we can also use an UUID without issues.

[anakin87]

Looks good