Skip to content

Commit 76cc73b

Browse files
authored
test: xfail a flaky app test in CI (#9361)
Many of the app composition and embedding tests are flaky in CI. This suggests either cross-test state pollution or a bug in the underlying implementation. These bugs are not high priority to fix, so the flaky test has been marked as xfail. If this ever does become a priority (in particular, if we see GitHub issues about these methods failing in real usage), these failing tests may provide a clue. This test appears to only have started failing somewhat consistently after #9257, but I don't see any relation between the two (it's possible #9257's tests may pollute the test environment, but I haven't been able to see how). <!-- This is an auto-generated description by cubic. --> --- ## Summary by cubic Marked tests/_ast/test_app.py::TestAppComposition::test_overrides_returns_multiple_defs as xfail to stabilize CI for app composition/embedding. The failure is CI-only and likely due to cross-test state or an underlying bug; we’ll revisit if it affects users. <sup>Written for commit 3046298. Summary will update on new commits.</sup> <!-- End of auto-generated description by cubic. -->
1 parent 32ce9b5 commit 76cc73b

1 file changed

Lines changed: 10 additions & 2 deletions

File tree

tests/_ast/test_app.py

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1117,6 +1117,12 @@ def __():
11171117
assert "value2" in output
11181118

11191119

1120+
# TODO(akshayka): Many of the app composition and embedding tests are flaky in
1121+
# CI. This suggests either cross-test state pollution or a bug in the
1122+
# underlying implementation. These bugs are not high priority to fix, so
1123+
# the flaky tests have been marked as xfail. If this ever does become
1124+
# a priority (in particular, if we see GitHub issues about these methods
1125+
# failing in real usage), these failing tests may provide a clue.
11201126
class TestAppComposition:
11211127
async def test_app_embed(self) -> None:
11221128
app = App()
@@ -1784,6 +1790,10 @@ def __(x: int) -> tuple[int]:
17841790
assert not k.errors
17851791
assert k.globals["overrides"] == {"x": 100}
17861792

1793+
1794+
@pytest.mark.xfail(
1795+
True, reason="Flaky in CI, can't repro locally", strict=False
1796+
)
17871797
async def test_overrides_returns_multiple_defs(
17881798
self, k: Kernel, exec_req: ExecReqProvider
17891799
) -> None:
@@ -1827,8 +1837,6 @@ def __(a: int, b: int) -> tuple[int]:
18271837
assert k.globals["overrides"] == {"a": 10, "b": 20}
18281838

18291839

1830-
1831-
18321840
class TestAppKernelRunnerRegistry:
18331841
def test_get_runner(self, k: Kernel) -> None:
18341842
# `k` fixture installs a context, needed for AppKernelRunner

0 commit comments

Comments
 (0)