Flaky Playwright tests

[rmunn]

Describe the bug
While merging #1652 I noticed that some Playwright tests had started being flaky, randomly failing or succeeding (both in CI and locally). They did eventually succeed after a few retries with no code changes, so I merged that PR. But we would like to be able to rely on our Playwright test failures being true failures, so we should track down and eliminate the source of the flakiness.

To Reproduce
Steps to reproduce the behavior:

1. Run pnpm test in the frontend directory
2. Note some test failures
3. Run it again
4. Note that some tests that failed in step 2 just passed, while other tests that passed in step 2 just failed

Expected behavior
We would like tests to be consistent, either passing every time or failing every time until the bug is fixed.

[rmunn]

Speaking of flaky tests, Playwright 1.52 added an option called testConfig.failOnFlakyTests to fail the test run if Playwright classified any tests as flaky. That might or might not be something we want, depending on the reason for test flakiness, but it's certainly something we will want to consider.

[rmunn]

Okay, just had a WTF moment. I was seeing repeated test failures because sometimes clicking on a project link from the admin dashboard would fail. So I doubled up on the click — not running locator.dblclick(), but calling await locator.click() twice in a row. It should fail, right? No! It passed! The first click did nothing, then the second click hit the link when it was actually ready to function as a link.

Don't know why that's happening yet, but I suspect once I do understand this I'll know how to fix the flaky tests. It's definitely a timing issue of some kind.

[rmunn]

It looks like the timing issue, whatever it is, is related to having a mix of stores and state. Because I'm also seeing an error that I never saw before, where the $draftProjects store can have an undefined value in it. Specifically, the AdminProjects component has a draftProject prop that is populated as follows: <AdminProjects draftProjects={$draftProjects} ... />. So in principle, the prop should be subscribed to the store, and since the store is initially populated in the load function with draftProjects: gqlResult ?? [], there should never have possibly been an undefined value in there. But it's there, consistently and repeatably.

My only theory so far is that it's related to stores having a "push" model for updates, while $derived has a "pull" model. Somehow, and this seems weak but it's the only theory I have right now, the store is "pushing" the update when the derived isn't ready for it, and then when the derived wants to "pull" the signal there's nothing in the store? That doesn't make sense. But I do know that writing let { ... draftProjects = [], ... } = $props() in the AdminProjects component fixes the error, for good AFAICT right now. There's no place where an undefined should be capable of creeping in, and yet there it is.

If I can figure out how to repro this consistently, I should try to narrow it down to a minimal reproduction. Then I'll be able to figure out if it's actually a Svelte bug, or just me misunderstanding something.

[rmunn]

I suspect the difference is that in Svelte 5, $derived values are calculated lazily, so that they only try to retrieve their value at the moment when it's needed. (The "pull" model I mentioned). Whenever their value is needed, they check if their "dirty" flag is set; if it isn't, then they return their cached value. if their "dirty" flag is set, they call their internal function which runs the expression part of the $derived and returns its value, then they clear their "dirty" flag again. (It will be set anytime one of their dependencies changes).

In the AdminProjects component, we have an allProjects array that's set to $derived[...draftProjects.map( ... ), ...projects.map( ... ) ]. And since I've been seeing the "draftProjects is undefined" error showing up on the project page, after Playwright has clicked the project link, I'm starting to think that the sequence is as follows:

1. Playwright clicks the link, causing a navigation event and changing the URL
2. The projectFilter= query is no longer in the URL
3. The allProjects derived state notices a change in one of its dependencies (the projectFilter= query) and marks itself as dirty
4. The $effect that Svelte created behind the scenes to track the HTML rendering of AdminProjects fires because one of its dependencies (allProjects) was dirty
5. By the time the allProjects derived tries to update itself again, the $draftProjects store (that the draftProjects derived was depending on) has been disposed because the page navigation has completed and you're now on the project page, where data.draftProjects is undefined.

That last point might be hard to understand, so let me show the definitions of the data:

• Admin dashboard (admin/+page.svelte) has let draftProjects = $derived(data.draftProjects), where data.draftProjects is a store. Therefore draftProjects in the admin dashboard is a state that contains a store.
• Admin dashboard calls AdminProjects component passing in draftProjects={$draftProjects}. So the store that's inside the state is subscribed to, and the value inside that store is what will be contained in the draftProjects prop of AdminProjects.
• AdminProjects has an allProjects state that's derived from the projects and draftProjects props it receives.

I don't yet understand why the allProjects state is trying to update itself after it's no longer needed because the page has navigated away, but I'm starting to be persuaded that that's what's happening and what's causing this error.

[rmunn]

Pretty much confirmed; I can reproduce this error without Playwright being involved. To reproduce:

• Check out the develop branch of lexbox
• Go to http://localhost/admin
• Type "sena" into the project filter
• Open the Javascript console so you can watch for errors
• Look closely at the console as you click on the "Sena 3" link
• Notice that the admin dashboard page went away, then an error occurred in the console saying that draftProjects is undefined
• Edit the code to add the line $inspect(draftProjects) just under where it's declared in admin/+page.svelte
• Reload the admin dashboard and watch the Javascript console
• Notice the draftProjects derived value goes undefined during nav (the console prints update undefined just before the error appears)

[rmunn]

Well, I may have been chasing a phantom. The crash is real, but the reason the draftProjects derived was being accessed was because I had put in a $inspect(shownProjects) line and shownProjects has draftProjects (among others) as a dependency. When I remove that $inspect I still get failing Playwright tests where nav doesn't happen, but I don't get that console error any more.