ENG-1583: Automate Obsidian dev environment e2e testing#959
ENG-1583: Automate Obsidian dev environment e2e testing#959trangdoan982 wants to merge 4 commits into
Conversation
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
trangdoan982
force-pushed
the
worktree-eng-1583-automate-obsidian-dev-environment-load-and-visual-validation
branch
from
0d44589 to
1cb7a28
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
trangdoan982
force-pushed
the
worktree-eng-1583-automate-obsidian-dev-environment-load-and-visual-validation
branch
from
1cb7a28 to
c9f522a
Compare
Extract inline Obsidian interaction logic into reusable helpers (commands, vault, modal, screenshots) and split smoke.spec.ts into focused spec files for plugin-load and node-creation. Add JSON reporter to playwright config. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix vault selection by manipulating obsidian.json instead of unsupported --vault flag - Find correct workspace page via DOM query instead of assuming pages()[0] - Load .env via dotenv for OBSIDIAN_TEST_VAULT configuration - Replace blind waitForTimeout calls with proper selector/function waits - Add ensureVaultWithPlugin for safe setup on existing vaults - Add CDP connection retry logic - Scope eslint-disables to individual page.evaluate blocks - Use crypto.randomBytes for proper hex vault IDs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update pnpm-lock.yaml for @playwright/test, use port-based Obsidian teardown in afterAll, and switch multi-arg e2e helpers to named parameters. Co-authored-by: Cursor <cursoragent@cursor.com>
trangdoan982
force-pushed
the
worktree-eng-1583-automate-obsidian-dev-environment-load-and-visual-validation
branch
from
c9f522a to
21b7625
Compare
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 60025b975b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| createTestVault(vaultPath); | ||
| } | ||
|
|
||
| const launched = await launchObsidian(vaultPath); |
There was a problem hiding this comment.
When launchObsidian() throws before use is reached (for example the debug port never opens or verifyActiveVault fails after setActiveVault has rewritten ~/Library/Application Support/obsidian/obsidian.json), Playwright never executes the teardown code below, so the user's Obsidian config remains pointed at the test vault and any launched process may keep running. Wrap the launch/setup path in try/finally (or make launchObsidian roll back on failure) so cleanup happens even when fixture setup fails.
Useful? React with 👍 / 👎.
| (f: { path: string; basename: string }) => | ||
| f.path === scratchPath || f.basename.startsWith(legacyPrefix), |
There was a problem hiding this comment.
When OBSIDIAN_TEST_VAULT is set, teardown intentionally skips cleanTestVault, but this cleanup only removes e2e-scratch.md and legacy scratch-e2e-* files. The smoke scenario creates QUE - What is discourse graph testing ... notes in that same user-provided vault, so every e2e run leaves test nodes behind; track/delete the created basename or give e2e-created nodes a cleanup prefix that this filter removes.
Useful? React with 👍 / 👎.
mdroidian
left a comment
mdroidian
left a comment
There was a problem hiding this comment.
Some nits below from me, codex comments worth looking into
But the blocking change request is: clearer documentation/instructions on how to it. Right now NOTES.md is a combination of a conversation/decisions made with the LLM, but it has a lot of fluff/extra unnecessary noise. Let's drop the "options" and "phases" etc (things like that should live in linear) and only add instructions for someone who wants to use this feature.
| @@ -0,0 +1,240 @@ | |||
| # E2E Testing for Obsidian Plugin — Notes | |||
There was a problem hiding this comment.
This looks like an artifact from an LLM conversation. Should it live in the repo? If so, why?
| @@ -0,0 +1,19 @@ | |||
| import type { Page } from "@playwright/test"; | |||
| import { PLUGIN_ID } from "../constants"; | |||
| import { captureStep } from "../helpers/screenshots"; | |||
There was a problem hiding this comment.
try to remember to use our convention of ~ instead of ..
| @@ -0,0 +1,25 @@ | |||
| import path from "path"; | |||
There was a problem hiding this comment.
try to match file naming convention of fileName.ts instead of file-name.ts
2 participants
https://www.loom.com/share/7614e7da36184f109dd867621f68aa97
Summary
Adds a Playwright-based end-to-end test harness for the Obsidian plugin. Because Obsidian forks Electron on launch, tests connect via Chrome DevTools Protocol (CDP) instead of Playwright's native Electron API. The suite is organized in layers — launch/setup helpers, reusable scenarios, and a thin smoke spec — so agents and developers can run
pnpm test:e2eafter changes and get deterministic feedback.Architecture
Why CDP?
Obsidian's macOS binary is a launcher that forks a child Electron process. Playwright's
electron.launch()attaches to the launcher, which exits immediately. CDP avoids that: launch Obsidian with--remote-debugging-port=9222, connect withchromium.connectOverCDP(), and drive the renderer where Obsidian's globalappobject lives.The Obsidian CLI controls a running instance; it cannot start one with a debug port. CDP launch is required; CLI is optional afterward.
Layered layout
Fixtures own process lifecycle. Helpers are reusable primitives. Scenarios hold assertions and user flows. Specs only call scenarios — no duplicated test logic.
Launch and teardown flow
sequenceDiagram participant Fixture as Worker fixture participant Setup as obsidian-setup participant Obsidian participant PW as Playwright CDP participant Test as smoke.spec Fixture->>Setup: resolveVaultPath() alt default vault Fixture->>Setup: createTestVault() — wipe + install plugin else OBSIDIAN_TEST_VAULT set Fixture->>Setup: ensureVaultWithPlugin() — non-destructive end Setup->>Setup: killObsidianOnDebugPort(9222) Setup->>Setup: setActiveVault() in obsidian.json Setup->>Obsidian: open -na Obsidian.app --args --remote-debugging-port=9222 Setup->>PW: connectOverCDP + findWorkspacePage() Setup->>Setup: verifyActiveVault(basePath) Fixture->>Test: obsidian.page Test->>Test: run scenarios via page.evaluate(app.*) Fixture->>Setup: cleanup scratch files, browser.close() Setup->>Setup: killObsidianOnDebugPort(), restore obsidian.json alt default vault Setup->>Setup: cleanTestVault() endVault selection: Obsidian has no
--vaultflag. Tests set"open": trueon the target vault in~/Library/Application Support/obsidian/obsidian.jsonbefore launch, then restore the original config after the run.Workspace page discovery: Obsidian opens multiple CDP targets (dev console, workspace).
findWorkspacePage()scans for the page with a visible.workspaceelement.Single instance:
workers: 1and a worker-scoped fixture share one Obsidian process per run. Port 9222 is fixed, so parallel workers would conflict.How tests interact with Obsidian
Tests run code inside Obsidian's renderer via
page.evaluate():app.plugins.plugins[PLUGIN_ID]app.commands.executeCommandById(...)app.vault.getMarkdownFiles(),app.vault.read()app.vault.adapter.basePathCommand triggering has two paths:
Node creation goes through
ModifyNodeModal(.modal-container textarea, not<input>), then asserts the vault contains a file with the expectedQUE - {content}basename andnodeTypeIdfrontmatter.Configuration
Copy
e2e/.env.example→apps/obsidian/.env(loaded byplaywright.config.ts):OBSIDIAN_APP_PATH/Applications/Obsidian.app/Contents/MacOS/Obsidian)OBSIDIAN_TEST_VAULTPrerequisites:
pnpm build(copiesdist/into the test vault), Obsidian CLI enabled (Settings → General), quit other Obsidian instances before running tests.Smoke suite
pnpm test:e2erunssmoke.spec.ts:The
node-creationscenario also supportstrigger: "command"for the faster API path; add a smoke test when you want that covered in CI.Step screenshots land in
e2e/test-results/<test-name>/for agent/human visual validation.Test plan
OBSIDIAN_TEST_VAULTinapps/obsidian/.envto an existing vaultpnpm buildinapps/obsidianpnpm test:e2e— Obsidian launches the correct vault, plugin loads, node creation worksOBSIDIAN_TEST_VAULT— temp vault is created and cleaned upobsidian.jsonis restored after testsOut of scope (follow-ups)
Design notes and phased roadmap:
apps/obsidian/e2e/NOTES.md.