|
| 1 | +# Discovering and locating UI elements in IDE integration tests |
| 2 | + |
| 3 | +This document explains **how the IntelliJ Driver sees the UI**, **what to prefer when writing locators**, and **how to discover** selectors when you’re debugging a test. It complements [ui-testing.md](ui-testing.md) (API overview) and [agent-playbook.md](agent-playbook.md) (short checklist). |
| 4 | + |
| 5 | +--- |
| 6 | + |
| 7 | +## 1. Mental model: Swing → XML DOM → XPath |
| 8 | + |
| 9 | +The driver does **not** use browser-style HTML. It uses a **remote Swing hierarchy** exposed as an **XML-like DOM** string. **`SearchService`** (in `driver-sdk`) calls **`SwingHierarchyService.getSwingHierarchyAsDOM(component, onlyFrontend)`**, parses that XML, and runs **XPath** queries against it. |
| 10 | + |
| 11 | +Implications: |
| 12 | + |
| 13 | +- **“XPath” in tests** means XPath over this **Swing DOM**, not HTML. |
| 14 | +- Attributes you match on (`@accessiblename`, `@visible_text`, `@class`, `@javaclass`, `@title`, `@tooltiptext`, etc.) come from how that DOM is serialized — see **`QueryBuilder`** in **`driver-sdk-*-sources.jar`** for the exact attribute names. |
| 15 | +- **Scoped searches**: If you pass a **root `Component`** (e.g. a dialog), the hierarchy is smaller and queries are faster and less ambiguous. `Finder`/`UiComponent` implicitly search from a **search context** (often the focused window or a parent xpath). |
| 16 | + |
| 17 | +**Official background:** [Integration tests (UI)](https://plugins.jetbrains.com/docs/intellij/integration-tests-ui.html). |
| 18 | + |
| 19 | +--- |
| 20 | + |
| 21 | +## 2. What to prefer (decision order) |
| 22 | + |
| 23 | +Use this order when choosing how to find a control: |
| 24 | + |
| 25 | +| Priority | What to use | When / notes | |
| 26 | +|----------|-------------|----------------| |
| 27 | +| **1** | **Built-in `driver-sdk` UI types** (`ideFrame`, `welcomeScreen`, `projectView`, `editRunConfigurationsDialog`, …) | Default for standard IDE surfaces; maintained with platform. | |
| 28 | +| **2** | **Element APIs** (`textField { }`, `popup { }`, … in `components.elements`) | Typed controls; use **`.text`** on text fields, not `toString()`. | |
| 29 | +| **3** | **`QueryBuilder` / `xQuery { }`** (`byAccessibleName`, `byType`, `byTooltip`, `componentWithChild`, …) | Prefer over long hand-written XPath; composable and readable. | |
| 30 | +| **4** | **Raw XPath string** `x("//div[…]")` | OK for spikes or when builders don’t cover an edge case; **fragile** — scope tightly or migrate to `QueryBuilder`. | |
| 31 | +| **5** | **Project helpers** (e.g. `runConfigurationsDialog { }`) | One place for awkward dialogs; document rationale in KDoc. | |
| 32 | + |
| 33 | +**XPath is not “wrong”** — it is the **underlying query language**. The recommendation is to avoid **long, hand-maintained** XPath strings when **`xQuery { }` + `QueryBuilder`** express the same thing more clearly and reuse platform attribute helpers. |
| 34 | + |
| 35 | +--- |
| 36 | + |
| 37 | +## 3. `QueryBuilder` and `xQuery` (preferred structured locators) |
| 38 | + |
| 39 | +In **`driver-sdk`**, `QueryBuilder` builds predicate fragments that become **`//div[ … ]`**-style paths via **`xQuery { … }`**. |
| 40 | + |
| 41 | +Common builders (names may vary slightly by driver version — verify in **`QueryBuilder.kt`**): |
| 42 | + |
| 43 | +| Builder | Typical meaning / DOM attribute | |
| 44 | +|---------|-----------------------------------| |
| 45 | +| **`byAccessibleName(text)`** | Accessibility name (screen readers, many buttons/labels). | |
| 46 | +| **`byVisibleText(text)`** | **`visible_text`** — text shown on screen (good for buttons, static labels). | |
| 47 | +| **`byTitle(title)`** | Window/dialog title. | |
| 48 | +| **`byText(text)`** | **`text`** attribute where applicable. | |
| 49 | +| **`byClass(shortName)`** | Short class name (e.g. `JBList`). | |
| 50 | +| **`byJavaClass` / `byType(fullName)`** | Full JVM class name or hierarchy-aware match — e.g. **`ExpandableTextField`**, **`JDialog`**. | |
| 51 | +| **`byTooltip(text)`** | **`tooltiptext`** — hover text when accessible name is unclear. | |
| 52 | +| **`componentWithChild(parentPred, childPred)`** | Parent that **contains** a child matching another predicate (e.g. label next to field). | |
| 53 | +| **`and` / `or` / `not` / `contains`** | Boolean combination of conditions. | |
| 54 | + |
| 55 | +**Example (conceptual):** |
| 56 | + |
| 57 | +```kotlin |
| 58 | +x { |
| 59 | + or( |
| 60 | + byTooltip("Add new configuration"), |
| 61 | + byAccessibleName("Add New Configuration"), |
| 62 | + ) |
| 63 | +} |
| 64 | +``` |
| 65 | + |
| 66 | +**Example: field next to a label** (pattern for plugin forms): |
| 67 | + |
| 68 | +```kotlin |
| 69 | +textField { |
| 70 | + and( |
| 71 | + xQuery { |
| 72 | + componentWithChild( |
| 73 | + byClass("javax.swing.JLabel"), |
| 74 | + byVisibleText("Additional args:"), |
| 75 | + ) |
| 76 | + }, |
| 77 | + xQuery { byType("com.intellij.ui.components.fields.ExpandableTextField") }, |
| 78 | + ) |
| 79 | +} |
| 80 | +``` |
| 81 | + |
| 82 | +Always confirm **exact** label strings in your plugin’s **`.form`** / UI code — casing and punctuation matter. |
| 83 | + |
| 84 | +--- |
| 85 | + |
| 86 | +## 4. Raw XPath strings (`x("//div[…]")`) |
| 87 | + |
| 88 | +**When they’re used:** |
| 89 | + |
| 90 | +- Quick spikes, or matching attributes **`QueryBuilder` doesn’t wrap** yet. |
| 91 | +- Very specific structural paths (rare). |
| 92 | + |
| 93 | +**Risks:** |
| 94 | + |
| 95 | +- DOM shape can change between IDE versions or LaF. |
| 96 | +- Easy to match the **wrong** node (e.g. first `JTextField` in a dialog instead of “Additional args”). |
| 97 | +- Harder to read than `byAccessibleName` / `byType`. |
| 98 | + |
| 99 | +**Mitigation:** Scope under a known parent (`ideFrame`, dialog wrapper, or `runConfigurationsDialog` helper), and prefer migrating to **`xQuery`** once you know which attributes are stable. |
| 100 | + |
| 101 | +--- |
| 102 | + |
| 103 | +## 5. Typed components and `x(Class, init)` |
| 104 | + |
| 105 | +Use **`x(MyUiComponent::class.java) { … }`** when **`driver-sdk`** ships a subclass of **`UiComponent`** for that screen (e.g. **`EditRunConfigurationsDialogUiComponent`**). You get typed fields (`addNewConfigurationButton`, etc.) instead of ad-hoc queries. |
| 106 | + |
| 107 | +Pattern: |
| 108 | + |
| 109 | +```kotlin |
| 110 | +ideFrame { |
| 111 | + editRunConfigurationsDialog { |
| 112 | + addNewConfigurationButton.click() |
| 113 | + } |
| 114 | +} |
| 115 | +``` |
| 116 | + |
| 117 | +If the SDK has no type for your dialog, fall back to **`RunConfigurationsDialogUI`**-style wrappers in **`testSrc/integration/.../utils/`**. |
| 118 | + |
| 119 | +--- |
| 120 | + |
| 121 | +## 6. Text fields, buttons, lists |
| 122 | + |
| 123 | +| Control | Typical approach | |
| 124 | +|---------|-------------------| |
| 125 | +| **JTextField / ExpandableTextField** | **`textField { … }`** → **`JTextFieldUI`** with **`.text`** getter/setter — **do not** assert on **`UiComponent.toString()`**. | |
| 126 | +| **JButton / link** | **`x { byVisibleText("…") }`**, **`byAccessibleName`**, or XPath on `@text`. | |
| 127 | +| **JBList / tree** | **`waitOneText("…")`** on list helpers where SDK provides them; or **`byClass("JBList")`** scoped under a dialog. | |
| 128 | +| **Tool windows** | Prefer **`toolWindow("…")`**, **`projectView`**, etc., from **`IdeaFrameUI`** extensions. | |
| 129 | + |
| 130 | +--- |
| 131 | + |
| 132 | +## 7. How to *discover* what to match (workflow) |
| 133 | + |
| 134 | +### A. **`driver-sdk` sources JAR** |
| 135 | + |
| 136 | +1. Resolve version: `./gradlew dependencyInsight --dependency driver-sdk --configuration integrationRuntimeClasspath` |
| 137 | +2. Open **`driver-sdk-<ver>-sources.jar`** (same Maven coordinates as the binary JAR). |
| 138 | +3. Read **`QueryBuilder.kt`**, **`Finder.kt`**, **`JTextFieldUI.kt`**, and **`ui/components/common/...`** for patterns used by JetBrains. |
| 139 | + |
| 140 | +### B. **Swing hierarchy dump (same XML as XPath)** |
| 141 | + |
| 142 | +The IDE exposes **`SwingHierarchyService.getSwingHierarchyAsDOM`**. In this repo you can use helpers such as **`dumpSwingHierarchyToConsole`** / **`dumpXPathTreeToConsole`** (see `testSrc/integration/.../utils/SwingHierarchyDebug.kt` and **`RunConfigurationsDialogUI`**) to **print the DOM** while the UI you care about is visible. Use a **non-null root `Component`** (e.g. dialog) to avoid huge logs. |
| 143 | + |
| 144 | +**Setup:** `Setup.kt` sets **`expose.ui.hierarchy.url`** — useful for hierarchy inspection in some workflows. |
| 145 | + |
| 146 | +### C. **Plugin UI source (this repo)** |
| 147 | + |
| 148 | +For Flutter/plugin-specific labels and bindings: |
| 149 | + |
| 150 | +- **`*.form`** (GUI Designer) — exact **`text`** on **`JLabel`**s, **`binding=`** on fields. |
| 151 | +- **`TestForm.java`** / Kotlin UI — field types (**`ExpandableTextField`** vs **`JTextField`**). |
| 152 | + |
| 153 | +Match those strings in **`byVisibleText`** / **`componentWithChild`** / **`byType`**. |
| 154 | + |
| 155 | +### D. **Accessibility / hover in a running IDE** |
| 156 | + |
| 157 | +If **`byAccessibleName`** is unknown, use **`byTooltip`** with the string from **hover** (watch **casing**). Combine with **`or`** for multiple spellings. |
| 158 | + |
| 159 | +### E. **Screenshots** |
| 160 | + |
| 161 | +**`driver.takeScreenshot(...)`** — visual confirmation of what the driver session sees (see [driver-api.md](driver-api.md)). |
| 162 | + |
| 163 | +--- |
| 164 | + |
| 165 | +## 8. Scoping and ambiguity |
| 166 | + |
| 167 | +- **Prefer one strategy per dialog** — don’t mix unrelated XPaths and SDK extensions in the same flow without a clear root scope. |
| 168 | +- **“First match” bugs:** A query like “any `TextField` in the dialog” may hit **Name** before **Additional args**. Fix with **`byType`** for the concrete class, **`componentWithChild`** with a label, or a **single** `ExpandableTextField` if the form has only one. |
| 169 | +- **Template vs non-template** — Run/Debug UI differs; lists may be empty until **Add (+)** adds a type. |
| 170 | + |
| 171 | +--- |
| 172 | + |
| 173 | +## 9. Timing and visibility |
| 174 | + |
| 175 | +- **`waitFound()`** — element exists in hierarchy / visible per driver rules. |
| 176 | +- **`waitForIndicators`** — IDE not indexing / no status-bar background tasks (see [driver-api.md](driver-api.md) and SDK **`indicators.kt`**). |
| 177 | +- **`wait(duration)`** — coarse delays; use sparingly; prefer **`waitFound`** + **`waitForIndicators`**. |
| 178 | + |
| 179 | +--- |
| 180 | + |
| 181 | +## 10. Checklist before merging a new locator |
| 182 | + |
| 183 | +1. Prefer **SDK** or **`QueryBuilder`** over long raw XPath. |
| 184 | +2. Confirm strings **against real UI** or **`.form`** / hierarchy dump. |
| 185 | +3. Scope under **dialog / frame** when possible. |
| 186 | +4. Read **`.text`** (or the right property) for assertions on text fields. |
| 187 | +5. Document **project helpers** in **`utils/`** with **why** the locator exists. |
| 188 | + |
| 189 | +--- |
| 190 | + |
| 191 | +## See also |
| 192 | + |
| 193 | +- [ui-testing.md](ui-testing.md) — `Finder`, `UiComponent`, packages overview |
| 194 | +- [driver-api.md](driver-api.md) — `invokeAction`, modals, screenshots |
| 195 | +- [agent-playbook.md](agent-playbook.md) — short agent rules |
| 196 | +- [maintenance.md](maintenance.md) — refreshing docs and JAR listings |
| 197 | +- [Integration tests (UI)](https://plugins.jetbrains.com/docs/intellij/integration-tests-ui.html) — JetBrains documentation |
0 commit comments