Skip to content

Commit e754847

Browse files
committed
DO NOT MERGE: stress-test langsmith flake fix (1000x per matrix)
Temporary experiment branch to validate that removing max_cached_workflows=0 from test_temporal_prefixed_query_not_traced fixes the Windows 3.10 deadlock flake. Changes: - tests/contrib/langsmith/test_integration.py: parametrize the test with range(1000) so each matrix entry runs it 1000x via pytest-xdist; remove max_cached_workflows=0 from the worker config. - .github/workflows/ci.yml (build-lint-test job only): - Run only the parametrized test with -p no:rerunfailures so a single failure is not hidden by retries. - Disable poe lint, poe bridge-lint, the time-skipping test step, and the openai test step for this branch. - Bump job timeout to 60 min. - Step timeout 30 min for the stress test. Success = 0 failures across 10,000 invocations (10 matrix entries x 1000). Any single failure means the cache=0 removal is not sufficient and a deeper fix is needed. This branch will be discarded once CI confirms the result. The actual fix (test change only, no CI change) will go on a separate branch/PR.
1 parent a161d18 commit e754847

2 files changed

Lines changed: 21 additions & 16 deletions

File tree

.github/workflows/ci.yml

Lines changed: 18 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ env:
1616
jobs:
1717
# Build and test the project
1818
build-lint-test:
19-
timeout-minutes: 30
19+
timeout-minutes: 60
2020
strategy:
2121
fail-fast: false
2222
matrix:
@@ -58,22 +58,25 @@ jobs:
5858
- uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8
5959
- run: uv tool install poethepoet
6060
- run: uv sync --all-extras
61-
- run: poe bridge-lint
62-
if: ${{ matrix.clippyLinter }}
61+
# [STRESS TEST BRANCH] bridge-lint disabled
62+
# - run: poe bridge-lint
63+
# if: ${{ matrix.clippyLinter }}
6364
- run: poe build-develop
64-
- run: poe lint
65+
# [STRESS TEST BRANCH] poe lint disabled
66+
# - run: poe lint
6567
- run: mkdir junit-xml
66-
- run: poe test ${{matrix.pytestExtraArgs}} -s --junit-xml=junit-xml/${{ matrix.python }}--${{ matrix.os }}.xml
67-
timeout-minutes: 15
68-
# Time skipping doesn't yet support ARM
69-
- if: ${{ !endsWith(matrix.os, '-arm') }}
70-
run: poe test ${{matrix.pytestExtraArgs}} -s --workflow-environment time-skipping --junit-xml=junit-xml/${{ matrix.python }}--${{ matrix.os }}--time-skipping.xml
71-
timeout-minutes: 10
72-
- if: ${{ matrix.openaiTestTarget && (github.event.pull_request.head.repo.full_name == '' || github.event.pull_request.head.repo.full_name == 'temporalio/sdk-python') }}
73-
run: poe test tests/contrib/openai_agents/test_openai.py ${{matrix.pytestExtraArgs}} -s --junit-xml=junit-xml/${{ matrix.python }}--${{ matrix.os }}--openai.xml
74-
timeout-minutes: 10
75-
env:
76-
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
68+
# [STRESS TEST BRANCH] Run only the flaky test 1000x with no rerun hiding
69+
- run: uv run pytest tests/contrib/langsmith/test_integration.py::TestBuiltinQueryFiltering::test_temporal_prefixed_query_not_traced -n auto --dist=worksteal -p no:rerunfailures -s --junit-xml=junit-xml/${{ matrix.python }}--${{ matrix.os }}.xml
70+
timeout-minutes: 30
71+
# [STRESS TEST BRANCH] Time-skipping and openai steps disabled for this branch
72+
# - if: ${{ !endsWith(matrix.os, '-arm') }}
73+
# run: poe test ${{matrix.pytestExtraArgs}} -s --workflow-environment time-skipping --junit-xml=junit-xml/${{ matrix.python }}--${{ matrix.os }}--time-skipping.xml
74+
# timeout-minutes: 10
75+
# - if: ${{ matrix.openaiTestTarget && (github.event.pull_request.head.repo.full_name == '' || github.event.pull_request.head.repo.full_name == 'temporalio/sdk-python') }}
76+
# run: poe test tests/contrib/openai_agents/test_openai.py ${{matrix.pytestExtraArgs}} -s --junit-xml=junit-xml/${{ matrix.python }}--${{ matrix.os }}--openai.xml
77+
# timeout-minutes: 10
78+
# env:
79+
# OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
7780
- name: "Upload junit-xml artifacts"
7881
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
7982
if: always()

tests/contrib/langsmith/test_integration.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1224,9 +1224,11 @@ def my_query(self) -> str:
12241224
class TestBuiltinQueryFiltering:
12251225
"""Verifies __temporal_ prefixed queries are not traced."""
12261226

1227+
@pytest.mark.parametrize("_iteration", range(1000))
12271228
async def test_temporal_prefixed_query_not_traced(
12281229
self,
12291230
client: Client,
1231+
_iteration: int,
12301232
) -> None:
12311233
"""__temporal_workflow_metadata query should not produce a trace,
12321234
but user-defined queries should still be traced.
@@ -1235,6 +1237,7 @@ async def test_temporal_prefixed_query_not_traced(
12351237
client-side QueryWorkflow traces, isolating the test to
12361238
worker-side HandleQuery traces only.
12371239
"""
1240+
del _iteration
12381241

12391242
task_queue = f"query-filter-{uuid.uuid4()}"
12401243
collector = InMemoryRunCollector()
@@ -1249,7 +1252,6 @@ async def test_temporal_prefixed_query_not_traced(
12491252
worker_client,
12501253
QueryFilteringWorkflow,
12511254
task_queue=task_queue,
1252-
max_cached_workflows=0,
12531255
) as worker:
12541256
handle = await query_client.start_workflow(
12551257
QueryFilteringWorkflow.run,

0 commit comments

Comments
 (0)