Playwright tests

[SmittieC]

Technical Description

This PR adds

• playwright-cli skill to the repo.
• Some core workflows that agents should follow that tests basic site functionality. This can be expanded over time, but adding a few here for scaffolding
• A "healer" agent to receive a list of broken tests and fix them (this workflow has not been tested yet, since this is on a branch and I'm not sure how useful this is)
• Playwright equivalent tests for each workflow to run them faster and cheaper

Demo

Docs and Changelog

• This PR requires docs/changelog update

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[SmittieC]

@coderabbitai full review

[coderabbitai]

✅ Actions performed

Full review triggered.

[coderabbitai]

📝 Walkthrough

Walkthrough

This pull request introduces a comprehensive Playwright-based test automation framework for Open Chat Studio. It adds a new AI agent configuration (playwright-test-healer) for debugging failing tests, multiple skill definitions for test execution and browser automation (core-workflow-testing, playwright-cli), and extensive reference documentation for Playwright CLI commands and workflows. Two GitHub Actions workflows are added: one for standard test execution on pushes/pull requests, and one for manual test invocation with AI-powered remediation. The framework includes Playwright configuration with authentication setup, three test spec files covering authentication, chatbot workflows, and evaluations, helper utilities, and test workflow documentation. Supporting changes include updates to the bootstrap data seeding command for team/feature-flag/provider setup, removal of the npm test script in favor of Playwright tests, and parametrization of the task runner for port flexibility.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

🚥 Pre-merge checks | ❌ 3

❌ Failed checks (1 warning, 2 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	❓ Inconclusive	The title 'Playwright tests' is vague and generic. While it refers to a real aspect of the changeset (Playwright test infrastructure), it does not convey meaningful information about the main changes beyond using a non-descriptive term.	Consider using a more specific title that captures the primary change, such as 'Add Playwright E2E tests and agentic test-healer infrastructure' or 'Add Playwright E2E testing infrastructure with core workflows and healer agent'.
Description check	❓ Inconclusive	The pull request description is incomplete, missing critical sections required by the template.	Add a Product Description section (user-facing impact or 'no change'), expand the Technical Description with implementation details and design decisions, explicitly state whether migrations are backwards compatible, add demo screenshots/instructions if applicable, and confirm docs/changelog requirements by checking the checkbox.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

You can enable review details to help with troubleshooting, context usage and more.

Enable the reviews.review_details setting to include review details such as the model used, the time taken for each step and more in the review comments.

[coderabbitai]

Actionable comments posted: 12

🧹 Nitpick comments (6)

.claude/agents/playwright-test-healer.md (2)

63-73: Add language specifier to status report code block.

📝 Suggested fix

-```
+```text
 Test: <test name> → FIXED | WORKFLOW_FAILED | PASSED
 Test: <test name> → FIXED | WORKFLOW_FAILED | PASSED
 ...

 STATUS: FIXED
-```
+```

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.claude/agents/playwright-test-healer.md around lines 63 - 73, Update the
markdown code fence in .claude/agents/playwright-test-healer.md for the status
report block so it includes the language specifier "text" (i.e., change the
opening fence from ``` to ```text for the block that contains the "Test: <test
name> ..." lines and "STATUS: FIXED"); leave the closing fence unchanged. This
ensures proper syntax highlighting by tagging the status report code block as
text.

---

15-20: Add language specifier to code block.

The code block showing the input format lacks a language specifier.

📝 Suggested fix

-```
+```text
 - 05-chatbot-management.spec.ts > Chatbot Management > Create a chatbot
 - 02-team-management.spec.ts > Team Management > Invite a Team Member
-```
+```

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.claude/agents/playwright-test-healer.md around lines 15 - 20, Add a
language specifier to the fenced code block that lists failing tests so syntax
highlighting is explicit; update the code block containing "-
05-chatbot-management.spec.ts > Chatbot Management > Create a chatbot ..." to
start with ```text (instead of ```), leaving the contents unchanged—locate the
fenced block in .claude/agents/playwright-test-healer.md and modify the opening
fence accordingly.

.github/workflows/playwright.yml (1)

71-71: Consider using secrets for test credentials.

The test password is hardcoded in the workflow file. While this is acceptable for test environments, consider storing it as a repository secret for better hygiene and to avoid accidental exposure if these credentials are reused elsewhere.

-      run: uv run python manage.py bootstrap_data --email tester@playwright.com --password My0riginalP@ssw0rd! --team-slug agent --team-name Agent --reset
+      run: uv run python manage.py bootstrap_data --email tester@playwright.com --password ${{ secrets.PLAYWRIGHT_TEST_PASSWORD }} --team-slug agent --team-name Agent --reset

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.github/workflows/playwright.yml at line 71, The workflow currently
hardcodes the test credentials in the run step (the uv run python manage.py
bootstrap_data invocation); replace the hardcoded password (and optionally the
email) with GitHub Actions secrets and reference them in the run step (e.g. use
the secret name PLAYWRIGHT_TEST_PASSWORD via ${{
secrets.PLAYWRIGHT_TEST_PASSWORD }} and/or PLAYWRIGHT_TEST_EMAIL) and ensure
those secrets are added to the repository/actions secrets configuration; update
the run invocation to interpolate those secrets so no plaintext credentials
remain in the workflow file.

playwright/helpers/common.ts (1)

19-26: Promote the debug-toolbar workaround to a shared helper here.

setupPage() already owns the #djDebug workaround, but both new spec files still reimplement the same page.evaluate block locally. Exporting a callable hideDebugToolbar(page) from this module would keep the selector and workaround in one place.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@playwright/helpers/common.ts` around lines 19 - 26, Extract the '#djDebug'
workaround into a new exported helper named hideDebugToolbar(page: Page) that
performs the DOM hide (e.g., via page.evaluate or page.addInitScript with the
same selector logic), export that function from this module, and update
setupPage(page: Page) to call hideDebugToolbar(page) so all tests can reuse the
single implementation; reference the existing setupPage and the new
hideDebugToolbar symbols when making the change.

.claude/skills/core-workflow-testing/SKILL.md (1)

25-28: Add a language identifier to these fenced blocks.

markdownlint-cli2 is already flagging both examples with MD040. text is enough if you do not want syntax highlighting.

Also applies to: 36-43

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.claude/skills/core-workflow-testing/SKILL.md around lines 25 - 28, The
fenced code blocks containing the examples that show "Status: <SUCCESS |
FAILED>" and "Comment: <if something failed, describe the failure>" need a
language identifier to satisfy markdownlint MD040; update each triple-backtick
fence (including the other block at the later example) to include a language
like "text" (e.g., change ``` to ```text) so the blocks are explicitly labeled
without adding syntax highlighting.

playwright/tests/02-create-and-test-chatbot.spec.ts (1)

112-114: Wait on the chat outcome instead of sleeping 5 seconds.

This step currently passes after a fixed delay even if the send action never yields a bot reply or the expected error state. Waiting on one of those concrete outcomes will keep the failure in this test instead of pushing it to the downstream sessions check.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@playwright/tests/02-create-and-test-chatbot.spec.ts` around lines 112 - 114,
Replace the fixed sleep (await page.waitForTimeout(5000);) with an explicit wait
for a concrete chat outcome: use Promise.race (or page.waitForSelector with
{timeout}) to wait for either the bot response selector or the error/alert
selector to appear on the page; for example, replace the sleep in the test where
page is used with awaiting Promise.race between
page.waitForSelector('<bot-response-selector>') and
page.waitForSelector('<error-or-alert-selector>') (both with sensible timeouts)
so the test proceeds only when a reply or an error is actually present.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.claude/skills/core-workflow-testing/SKILL.md:
- Line 13: The current heuristic that derives a workflow section from the
numeric prefix of spec filenames is incorrect; replace it with an explicit
filename→flow mapping and use that map when parsing the failing-tests string.
Add a dictionary (map) keyed by exact spec filenames (e.g.
"02-create-and-test-chatbot.spec.ts" -> the Flow 1/section for "Chatbot
Management", "03-evaluations-datasets-annotations.spec.ts" -> the Flow 2/section
for its flow, etc.), and change the code that extracts section numbers from the
failing-list to look up each spec filename in this map and collect unique flows;
if a filename is missing from the map fall back to current behavior or log a
clear warning. Ensure the mapping is referenced wherever the spec-number
heuristic was used so only the explicitly mapped workflow sections are run.

In @.claude/skills/playwright-cli/references/request-mocking.md:
- Around line 28-35: The fenced code block under the "URL Patterns" heading is
missing a language tag which triggers MD040; update the opening triple-backtick
fence for that block (the block showing patterns like "**/api/users",
"**/api/*/details", "**/*.{png,jpg,jpeg}", "**/search?q=*") to use the text
language tag (i.e., change "```" to "```text") so the docs linter no longer
flags the block.

In @.claude/skills/playwright-cli/references/running-code.md:
- Around line 83-86: The example uses page.waitForLoadState('networkidle') which
conflicts with the playright-test-healer guidance to avoid networkidle; update
the example to either replace the call with a recommended wait strategy (e.g.,
waitForLoadState('load') or an explicit waitForSelector / waitForResponse
pattern) or add a short inline note after the code block saying that networkidle
is discouraged for tests and linking to the healer guidance; target the example
that uses playwright-cli run-code and the page.waitForLoadState('networkidle')
invocation when making this change.

In @.github/workflows/playwrite_test_checker.yml:
- Around line 149-175: Playwright failures currently pass unless
claude_repair.outputs.structured_output explicitly contains "STATUS:
WORKFLOW_FAILED", which lets empty/malformed or no-repair outputs hide broken
tests; update the workflow to require an explicit success token from the healer
(e.g. check for "STATUS: WORKFLOW_FIXED" or another defined success marker) and
treat any other claude_repair output (empty, malformed, or lacking that success
marker) as a failure: change the logic around steps.playwright.outcome and
steps.claude_repair.outputs.structured_output so the auto-commit/pr branch
creation only runs when the healer indicates success, and in the "Fail if core
workflows are broken" step (or a new step) exit 1 whenever playwright failed and
the structured_output does not contain the explicit success string.
- Around line 107-117: The readiness step named "Wait for server to be ready"
currently uses curl -s which treats HTTP 5xx/4xx as success; update the step so
the curl invocation fails on HTTP errors (for example by adding the -f/--fail
flag) or point it at a dedicated health endpoint (e.g., /health or /healthz)
that returns 2xx only when Django is fully ready; modify the command in that
step to use curl -fsS (or curl -f) and/or change the URL to the health endpoint
so the loop only exits when a successful 2xx response is returned.

In @.gitignore:
- Around line 44-48: The .gitignore is missing the persisted Playwright auth
state; add an ignore entry for the Playwright auth directory (e.g., ignore
"playwright/.auth/" or "playwright/.auth/*") so files like
"playwright/.auth/user.json" written by playwright/tests/auth.setup.ts are not
committed; update the .gitignore block that lists Playwright artifacts
(playwright-report/, playwright-results*, .playwright-cli, test-results/) to
include the .auth pattern.

In `@apps/web/management/commands/bootstrap_data.py`:
- Around line 131-134: The seed command only sets user.is_superuser and
user.is_staff when the --superuser flag is present, so reruns without the flag
leave prior promotions intact; update the logic in the bootstrap command (where
the variables user and superuser are handled, e.g., in the management command's
handle/bootstrap logic) to explicitly set user.is_superuser = bool(superuser)
and user.is_staff = bool(superuser) (or otherwise assign False when superuser is
falsy) before calling user.save() so the flag is idempotent across runs.

In `@playwright/tests/03-evaluations-datasets-annotations.spec.ts`:
- Around line 159-163: The tests "run the evaluation" (where
evalRow.locator('a[href*="/runs/new/"]').click(...) is used) and "verify
evaluation results" currently use page.waitForTimeout(3000);—replace these
arbitrary sleeps with concrete waits and assertions: after clicking the run
link, wait for a run row or status indicator to appear (e.g., use
page.waitForSelector or locator.waitFor on a run list element and assert its
status text/visibility using Playwright's expect), and in the "verify evaluation
results" test wait for the results table or completion badge to be visible and
assert expected contents; target the existing locators/buttons
(evalRow.locator('a[href*="/runs/new/"]') and the details panel/run list/result
table locators used in that test) so the test fails if no job started or no
results rendered.
- Around line 236-240: The current check uses noItemsMsg.isVisible(...) which is
subject to a race and ignores timeout; replace that call with await
noItemsMsg.waitFor({ state: 'visible', timeout: 3000 }) inside a try/catch and
return when it succeeds (catch should continue normally if timeout occurs),
i.e., in the block using the Locator noItemsMsg, swap isVisible(...) ->
waitFor({ state: 'visible', timeout: 3000 }) and handle the success by returning
early (items already annotated) and handle the thrown timeout by falling through
to the subsequent assertions.
- Around line 115-136: This test block assumes an existing chatbot and will fail
when run in isolation; fix it by adding setup that creates/seeds a chatbot
before the selection logic (e.g., in a beforeEach or at the start of this spec)
and then use that created chatbot's label when operating on chatbotSelect and
versionSelect; implement the seeding via the same API/fixture used elsewhere (or
call the app API to create a chatbot/version), store the created chatbot display
name/label and use it in chatbotSelect.selectOption(...) and subsequent version
selection, and ensure any created test data is cleaned up after the test run.

In `@playwright/tests/auth.setup.ts`:
- Around line 8-9: Replace the password locator that uses
page.getByRole('textbox', { name: 'Password' }) with Playwright's recommended
labeled lookup: use page.getByLabel('Password') (e.g., await
page.getByLabel('Password').fill('My0riginalP@ssw0rd!')), leaving the email line
as-is; update the locator in auth.setup.ts where the two fill calls occur
(referencing the page.getByRole(...) call for Password) so the password input
uses page.getByLabel instead.

In `@tasks.py`:
- Around line 179-185: The ngrok URL is hardcoded to port 8000 in ngrok_url(),
so when runserver(c, public, port) starts Django on a custom port the public URL
is wrong; update ngrok_url to accept an optional port parameter (e.g.,
ngrok_url(c, port=None)) and use that port when composing the tunnel URL instead
of the hardcoded 8000, then change the call inside runserver (where ngrok_url(c)
is invoked) to pass the runserver's port variable so ngrok publishes the correct
backend port; adjust any callers of ngrok_url accordingly to preserve default
behavior when port is None.

---

Nitpick comments:
In @.claude/agents/playwright-test-healer.md:
- Around line 63-73: Update the markdown code fence in
.claude/agents/playwright-test-healer.md for the status report block so it
includes the language specifier "text" (i.e., change the opening fence from ```
to ```text for the block that contains the "Test: <test name> ..." lines and
"STATUS: FIXED"); leave the closing fence unchanged. This ensures proper syntax
highlighting by tagging the status report code block as text.
- Around line 15-20: Add a language specifier to the fenced code block that
lists failing tests so syntax highlighting is explicit; update the code block
containing "- 05-chatbot-management.spec.ts > Chatbot Management > Create a
chatbot ..." to start with ```text (instead of ```), leaving the contents
unchanged—locate the fenced block in .claude/agents/playwright-test-healer.md
and modify the opening fence accordingly.

In @.claude/skills/core-workflow-testing/SKILL.md:
- Around line 25-28: The fenced code blocks containing the examples that show
"Status: <SUCCESS | FAILED>" and "Comment: <if something failed, describe the
failure>" need a language identifier to satisfy markdownlint MD040; update each
triple-backtick fence (including the other block at the later example) to
include a language like "text" (e.g., change ``` to ```text) so the blocks are
explicitly labeled without adding syntax highlighting.

In @.github/workflows/playwright.yml:
- Line 71: The workflow currently hardcodes the test credentials in the run step
(the uv run python manage.py bootstrap_data invocation); replace the hardcoded
password (and optionally the email) with GitHub Actions secrets and reference
them in the run step (e.g. use the secret name PLAYWRIGHT_TEST_PASSWORD via ${{
secrets.PLAYWRIGHT_TEST_PASSWORD }} and/or PLAYWRIGHT_TEST_EMAIL) and ensure
those secrets are added to the repository/actions secrets configuration; update
the run invocation to interpolate those secrets so no plaintext credentials
remain in the workflow file.

In `@playwright/helpers/common.ts`:
- Around line 19-26: Extract the '#djDebug' workaround into a new exported
helper named hideDebugToolbar(page: Page) that performs the DOM hide (e.g., via
page.evaluate or page.addInitScript with the same selector logic), export that
function from this module, and update setupPage(page: Page) to call
hideDebugToolbar(page) so all tests can reuse the single implementation;
reference the existing setupPage and the new hideDebugToolbar symbols when
making the change.

In `@playwright/tests/02-create-and-test-chatbot.spec.ts`:
- Around line 112-114: Replace the fixed sleep (await
page.waitForTimeout(5000);) with an explicit wait for a concrete chat outcome:
use Promise.race (or page.waitForSelector with {timeout}) to wait for either the
bot response selector or the error/alert selector to appear on the page; for
example, replace the sleep in the test where page is used with awaiting
Promise.race between page.waitForSelector('<bot-response-selector>') and
page.waitForSelector('<error-or-alert-selector>') (both with sensible timeouts)
so the test proceeds only when a reply or an error is actually present.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b7446b2f-e16d-48ca-a0b0-6679a4ca73d7

📥 Commits

Reviewing files that changed from the base of the PR and between bba408a and d1d0356.

⛔ Files ignored due to path filters (1)

• package-lock.json is excluded by !**/package-lock.json

📒 Files selected for processing (22)

• .claude/agents/playwright-test-healer.md
• .claude/skills/core-workflow-testing/SKILL.md
• .claude/skills/playwright-cli/SKILL.md
• .claude/skills/playwright-cli/references/request-mocking.md
• .claude/skills/playwright-cli/references/running-code.md
• .claude/skills/playwright-cli/references/session-management.md
• .claude/skills/playwright-cli/references/storage-state.md
• .claude/skills/playwright-cli/references/test-generation.md
• .claude/skills/playwright-cli/references/tracing.md
• .claude/skills/playwright-cli/references/video-recording.md
• .github/workflows/playwright.yml
• .github/workflows/playwrite_test_checker.yml
• .gitignore
• apps/web/management/commands/bootstrap_data.py
• package.json
• playwright.config.ts
• playwright/helpers/common.ts
• playwright/tests/02-create-and-test-chatbot.spec.ts
• playwright/tests/03-evaluations-datasets-annotations.spec.ts
• playwright/tests/auth.setup.ts
• playwright/workflows/team_user.md
• tasks.py