Skip to content

Commit 3aa324a

Browse files
johnml1135johnml1135
committed
LT-22324: add FieldWorks UI automation support
1 parent 2488da1 commit 3aa324a

12 files changed

Lines changed: 828 additions & 6 deletions

File tree

‎.github/skills/fieldworks-winapp/SKILL.md‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
---
2+
name: fieldworks-winapp
3+
description: >
4+
Control and document the FieldWorks desktop application with WinForms MCP or WinApp MCP.
5+
Use this skill whenever a task requires launching FieldWorks, restoring or
6+
opening a FieldWorks project, walking WinForms UI flows, collecting manual
7+
screenshots, reproducing UI bugs, or verifying a fix inside the live FLEx
8+
application.
9+
license: MIT
10+
compatibility: >
11+
Requires a Windows/x64 FieldWorks workspace. Prefer WinForms MCP
12+
for hidden-desktop UIA2 automation when available; use WinApp MCP as the UIA3
13+
fallback and for visible desktop/window diagnostics.
14+
metadata:
15+
author: FieldWorks team
16+
version: "1.0"
17+
---
18+
19+
# FieldWorks WinForms/WinApp Automation
20+
21+
Use this skill to launch, inspect, navigate, and capture evidence from the
22+
FieldWorks Language Explorer desktop app through WinForms MCP or WinApp MCP.
23+
Keep the workflow grounded in the live UI tree: inspect first, interact second,
24+
capture evidence after the target state is visible.
25+
26+
This skill is intentionally organized as a small index plus route-specific
27+
navigation files. Read only the route files needed for the task.
28+
29+
## Core Rules
30+
31+
- Prefer WinForms MCP UIA2 tools for fresh FieldWorks runtime automation because
32+
FieldWorks is currently WinForms and the workspace config runs WinForms MCP in
33+
hidden-desktop mode.
34+
- Use WinApp MCP UIA3 tools when WinForms MCP is unavailable, when a task needs
35+
visible desktop/window diagnostics, or when UIA2 cannot see the target
36+
surface.
37+
- Prefer UI Automation IDs and names over coordinates. Use coordinates only
38+
after snapshots and element searches fail.
39+
- Always inspect the current tree with `winforms_get_element_tree`,
40+
`mcp_winapp_get_snapshot`, or targeted element search before assuming a
41+
dialog structure.
42+
- Treat FieldWorks UI automation as stateful. After opening menus, dialogs, or
43+
popups, re-query the snapshot or element list.
44+
- Capture screenshots into a task-specific evidence folder before closing the
45+
app or dialog.
46+
- Close evidence-only dialogs with `Cancel` or `Escape` unless the user asked
47+
to change project data.
48+
- Do not globally register COM or add registry hacks. FieldWorks uses reg-free
49+
COM from the build output.
50+
51+
## MCP Selection
52+
53+
- Default to WinForms MCP for launch/test/screenshot flows. Use
54+
`winforms_launch_app`, `winforms_wait_for_element`,
55+
`winforms_get_element_tree`, `winforms_find_element`,
56+
`winforms_click_element`, `winforms_type_text`, `winforms_set_value`,
57+
`winforms_select_item`, `winforms_click_menu_item`, and
58+
`winforms_take_screenshot`.
59+
- Use WinForms MCP headless-safe operations only. Avoid `winforms_send_keys`,
60+
drag/drop, and double-clicks on hidden-desktop processes.
61+
- Use WinApp MCP for visible desktop/window diagnostics, UIA3 comparison,
62+
fallback screenshots, or controls that WinForms MCP cannot operate.
63+
- Read `references/mcp-selection.md` before changing a route from one MCP driver
64+
to the other.
65+
66+
## How This Skill Is Organized
67+
68+
- `SKILL.md`: trigger metadata, global safety rules, and route index.
69+
- `navigation/*.md`: one goal-oriented navigation path per file. These are the
70+
FieldWorks equivalent of Page Objects: each file owns the menu path,
71+
automation IDs, verification cues, and safe exit path for one user-facing
72+
destination or workflow.
73+
- `references/how-to-update.md`: rules for adding or revising navigation paths.
74+
- `references/mcp-selection.md`: rules for choosing WinForms MCP or WinApp MCP.
75+
- `references/research.md`: source-backed rationale for this structure.
76+
77+
When a task names a destination, read the matching navigation file. When a task
78+
discovers a better route or a fragile selector, read and update
79+
`references/how-to-update.md` before changing the route file.
80+
81+
## Navigation Path Index
82+
83+
- Launch or attach to FieldWorks: `navigation/launch-or-attach.md`
84+
- Confirm or restore a sample project: `navigation/project-loading.md`
85+
- Writing System Properties > Font tab: `navigation/writing-system-font-options.md`
86+
- Styles dialog > Font tab: `navigation/styles-font-tab.md`
87+
- Manual screenshot evidence: `navigation/screenshot-evidence.md`
88+
- MCP driver selection: `references/mcp-selection.md`
89+
90+
## Screenshot Evidence
91+
92+
For evidence tasks, read `navigation/screenshot-evidence.md`. Prefer committed
93+
OpenSpec evidence only when the screenshot should be part of review history;
94+
otherwise use `Output/ManualEvidence/<ticket-or-change-id>/`.
95+
96+
For target detection, descriptive filenames, inline review, annotation, and
97+
retake quality gates, also use `../smart-screenshot-capture/SKILL.md`.
98+
99+
## How To Update This Skill
100+
101+
Keep improving this skill as you explore FieldWorks and discover the most
102+
efficient ways to do things. When you find a reliable automation ID, keyboard
103+
route, menu path, restore flow, modal-dialog workaround, or screenshot trick,
104+
update the relevant route file in the same change set or propose the update to
105+
the user. Prefer short, verified notes over broad guesses.
106+
107+
Use `references/how-to-update.md` for the exact update checklist. In short:
108+
109+
- add one navigation file per distinct destination or workflow;
110+
- keep each file task-focused and action-oriented;
111+
- record stable automation IDs, entry state, verification cues, and exit path;
112+
- move shared rules back to this index only when at least two route files need
113+
the same guidance;
114+
- remove or revise stale routes when WinApp snapshots prove they changed.

‎.github/skills/fieldworks-winapp/navigation/launch-or-attach.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Launch Or Attach To FieldWorks
2+
3+
Use this path when a task needs the live FieldWorks Language Explorer app.
4+
Choose WinForms MCP first for fresh FieldWorks launches, and use WinApp MCP for
5+
visible desktop diagnostics or fallback.
6+
7+
## Entry State
8+
9+
- Workspace is a Windows/x64 FieldWorks checkout.
10+
- Prefer an existing debug build at `Output/Debug/FieldWorks.exe`.
11+
12+
## Preferred Steps: WinForms MCP UIA2
13+
14+
1. Launch debug FieldWorks with `winforms_launch_app`:
15+
- `path`: `<workspace>\\Output\\Debug\\FieldWorks.exe`
16+
2. Keep the returned `pid`; pass it to later process-scoped calls when the MCP
17+
client exposes a process parameter.
18+
3. Wait for a recognizable app window or menu with `winforms_wait_for_element`.
19+
4. Inspect the tree with `winforms_get_element_tree` before interacting.
20+
5. Use `winforms_click_menu_item`, `winforms_find_element`,
21+
`winforms_click_element`, `winforms_type_text`, `winforms_set_value`, and
22+
`winforms_select_item` for headless-safe interaction.
23+
6. Capture screenshots with `winforms_take_screenshot`. Prefer passing the
24+
FieldWorks `pid` so the server can use the hidden-desktop window directly.
25+
26+
## Fallback Steps: WinApp MCP UIA3
27+
28+
1. If FieldWorks is already running, attach to the existing process with
29+
`mcp_winapp_attach_to_app` or `mcp_winapp_attach_to_pid`.
30+
2. Otherwise launch debug FieldWorks with `mcp_winapp_launch_app`:
31+
- `exePath`: `<workspace>\\Output\\Debug\\FieldWorks.exe`
32+
3. If `mcp_winapp_wait_for_input_idle` is not implemented or fails, continue
33+
with direct window discovery.
34+
4. Use `mcp_winapp_list_windows` and `mcp_winapp_list_desktop_windows` to verify
35+
the app is visible.
36+
5. Use `mcp_winapp_get_snapshot` to confirm FieldWorks has a main window and a
37+
recognizable UI tree.
38+
6. If screenshots capture VS Code or another foreground app, press `ALT+ESCAPE`
39+
with `mcp_winapp_press_key_combo` until FieldWorks is first in desktop window
40+
order or a FieldWorks element has focus.
41+
42+
## Expected Signals
43+
44+
- WinForms MCP launch returns a process ID and can operate on a hidden desktop
45+
because `.vscode/mcp.json` sets `HEADLESS=true`.
46+
- Main window title may initially appear as `The Window` in UI Automation.
47+
- A loaded project window title may include a project name such as `Sena 3`.
48+
- The side pane often contains `Lexicon`, `Texts & Words`, `Grammar`,
49+
`Notebook`, and `Lists` groups.
50+
51+
## Known Workarounds
52+
53+
- In WinForms MCP headless mode, avoid `winforms_send_keys`, drag/drop, and
54+
double-click. Use UIA pattern operations such as `winforms_type_text`,
55+
`winforms_set_value`, `winforms_select_item`, and single
56+
`winforms_click_element` calls.
57+
- `mcp_winapp_wait_for_input_idle` may return `not implemented` for
58+
FieldWorks. Use snapshots and desktop-window order instead.
59+
- `mcp_winapp_take_screenshot` captures the foreground surface. Bring
60+
FieldWorks forward before relying on screenshots.
61+
62+
## Exit
63+
64+
Close only the FieldWorks instance you launched for the task, using
65+
`winforms_close_app` or `mcp_winapp_close_app`. Do not close a user's
66+
pre-existing app instance unless the user asked for cleanup.

‎.github/skills/fieldworks-winapp/navigation/project-loading.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Confirm Or Restore A Sample Project
2+
3+
Use this path after launching FieldWorks when a task needs a repeatable project.
4+
5+
## Entry State
6+
7+
- FieldWorks is launched or attached.
8+
- You have an app ID from WinApp MCP.
9+
10+
## Steps
11+
12+
1. Get a snapshot with `mcp_winapp_get_snapshot`.
13+
2. Treat the project as loaded if the snapshot contains a root document such as
14+
`Root - Sena 3`, or if the desktop title contains the expected project name.
15+
3. For repeatable manual evidence, use the current project if `Sena 3` is
16+
already loaded.
17+
4. If no project is loaded, restore from the sample backup at the repository
18+
root:
19+
- `Sena 3 2018-09-11 1145.fwbackup`
20+
5. Use the File menu and inspect the menu tree for restore/project-management
21+
commands before clicking.
22+
6. If a file chooser opens, select the backup path and proceed through restore.
23+
7. Re-inspect the app and capture a loaded-project screenshot.
24+
25+
## Expected Signals
26+
27+
- Project loaded: `Root - Sena 3` document in the snapshot.
28+
- Common loaded side pane: `Lexicon Edit`, `Browse`, `Dictionary`,
29+
`Collect Words`, `Classified Dictionary`, `Bulk Edit Entries`.
30+
31+
## Safety
32+
33+
- Do not restore over an existing loaded project unless the task requires it.
34+
- Prefer evidence-only screenshots over changing project data.
35+
36+
## Exit
37+
38+
Continue to the requested navigation path. If project restore was only a setup
39+
step and the user did not ask to keep the app open, close the launched app at
40+
the end of the task.

‎.github/skills/fieldworks-winapp/navigation/screenshot-evidence.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Capture Manual Screenshot Evidence
2+
3+
Use this path when the task needs before/after images, Jira evidence, OpenSpec
4+
evidence, or visual proof that a FieldWorks UI flow works.
5+
6+
For general screenshot quality rules, target inference, sorted filenames,
7+
annotations, and retake criteria, read `../../smart-screenshot-capture/SKILL.md`
8+
alongside this route.
9+
10+
## Entry State
11+
12+
- FieldWorks is launched and foregrounded.
13+
- The target UI state is visible or reachable through another navigation path.
14+
15+
## Steps
16+
17+
1. Create a deterministic evidence folder:
18+
- transient: `Output/ManualEvidence/<ticket-or-change-id>/`
19+
- committed OpenSpec evidence:
20+
`openspec/changes/<change-id>/evidence/manual-winapp/`
21+
2. Confirm the target FieldWorks process, window, dialog, or tab before capture.
22+
3. Capture the initial loaded-project state when it helps reviewers orient
23+
themselves.
24+
4. Capture the broken/reproduced state from an unfixed build when available.
25+
5. Capture the fixed state from the current build.
26+
6. Inspect each saved image and retake if it is blank, wrong-window, covered,
27+
unreadable, or missing the intended state.
28+
7. Capture related canonical surfaces when the issue spans more than one UI
29+
path.
30+
8. Use an image sequence or GIF when the issue involves timing, redraw, focus,
31+
modal transitions, or steps that a single screenshot cannot communicate.
32+
33+
## MCP Choice
34+
35+
- Prefer `winforms_take_screenshot` for fresh FieldWorks launches; hidden
36+
desktop screenshots should not steal focus or require foregrounding.
37+
- Prefer passing the FieldWorks `pid` to WinForms MCP screenshots when the
38+
client exposes a process parameter.
39+
- Prefer `mcp_winapp_take_screenshot_optimized` for visible-desktop fallback
40+
captures when image size or token budget matters.
41+
- Use `mcp_winapp_take_screenshot` when WinForms MCP is unavailable, when the
42+
task is explicitly about the visible desktop, or when comparing UIA3 behavior.
43+
- Use `mcp_winapp_annotate_screenshot` when reviewers need specific controls or
44+
changed regions called out.
45+
- Use `mcp_winapp_screenshot_diff` for before/after comparisons when pixel
46+
changes are the evidence.
47+
48+
## Naming
49+
50+
Use names that tell the story in sorted order:
51+
52+
- `01-initial-<state>.png`
53+
- `02-before-<screen>.png`
54+
- `03-after-<screen>.png`
55+
- `04-after-<related-screen>.png`
56+
- `sequence-<scenario>-001.png`, `sequence-<scenario>-002.png`, ...
57+
58+
## Expected Signals
59+
60+
- Screenshots should show FieldWorks, not VS Code or another foreground app.
61+
- WinForms MCP screenshots can capture headless FieldWorks windows through
62+
`PrintWindow`; foreground-window order should not matter for that path.
63+
- If `mcp_winapp_take_screenshot` captures the wrong surface, use the launch
64+
route's `ALT+ESCAPE` foregrounding workaround and capture again.
65+
- If a true before-state is not available in the current worktree, document why
66+
and describe how to capture it from an unfixed build.
67+
68+
## Exit
69+
70+
Copy only review-worthy screenshots into committed evidence folders. Leave
71+
scratch captures under `Output/ManualEvidence` unless the user asks to commit
72+
them.

‎.github/skills/fieldworks-winapp/navigation/styles-font-tab.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Navigate To Styles Font Tab
2+
3+
Use this path to inspect or document the shared Font Features control in the
4+
Styles dialog.
5+
6+
## Entry State
7+
8+
- FieldWorks is launched and a project is loaded.
9+
- If screenshots are needed, FieldWorks is foregrounded.
10+
11+
## Steps
12+
13+
1. Click the `Format` menu item.
14+
2. Click `Styles...`.
15+
3. In `FwStylesDlg`, click the `Font` tab.
16+
4. Select the target style in `m_lstStyles` if the task requires a specific
17+
style. For general evidence, `Normal` is usually sufficient.
18+
5. Capture or inspect the Font Features state.
19+
20+
## Stable Elements Observed
21+
22+
- Dialog: `FwStylesDlg`
23+
- Styles list: `m_lstStyles`
24+
- List filter combo: `m_cboTypes`
25+
- Tab control: `m_tabControl`
26+
- Font tab pane: `m_tbFont`
27+
- Font attributes pane: `m_FontAttributes`
28+
- Writing systems list: `m_lstWritingSystems`
29+
- Font combo: `m_cboFontNames`
30+
- Font Features button: `m_btnFontFeatures`
31+
- OK button: `m_btnOk`
32+
- Cancel button: `m_btnCancel`
33+
34+
## Expected Signals
35+
36+
- The Font tab shows `Font features` in the Attributes area.
37+
- The writing-system list includes `<default settings>` and project writing
38+
systems such as `English`, `Portuguese`, `Sena`, and `Sena (Phonetic)` for
39+
the Sena 3 sample project.
40+
41+
## Known Workarounds
42+
43+
- The `Styles...` menu item appears only after the `Format` menu is open; if a
44+
first click does not expose submenu items, click `Format` again and re-query
45+
menu items.
46+
- Use `mcp_winapp_invoke_element` for `m_btnCancel` when a normal click does not
47+
close the dialog.
48+
49+
## Exit
50+
51+
Use `Cancel` for evidence-only sessions. Use `OK` only when the task requires a
52+
style change.

0 commit comments

Comments
 (0)