installer: automate checklist via AutoHotkey UI tests#696
Draft
installer: automate checklist via AutoHotkey UI tests#696
Conversation
The library is adapted from the MSYS2 runtime's ui-test-library.ahk, trimmed to only what is needed for mintty-based testing. It keeps the core logging (Info, ExitWithError), command execution (RunWaitOne), and mintty buffer capture functions (CaptureBufferFromMintty, CaptureRawHtmlFromMintty, WaitForRegExInMintty). The Windows Terminal capture functions, worktree setup/cleanup, and LaunchMintty are dropped because Git Bash launches its own mintty instance (via git-bash.exe's embedded string table resource) and the tests must work with that instance rather than a custom-launched one. Unlike the MSYS2 runtime tests which pass -o KeyFunctions=... and -o SaveFilename=... on the mintty command line, these tests rely on the user's ~/.minttyrc already having those settings configured. The checklist script skeleton validates these prerequisites at startup and exits with a clear error message if either is missing. The SaveFilename value from ~/.minttyrc may be a Unix path (e.g. /tmp/mintty-export). Since AutoHotkey cannot resolve Unix paths natively, the script detects leading-slash paths and resolves them via git's cygpath, using the well-known installed path at C:\Program Files\Git\cmd\git.exe (we cannot assume git is in PATH since the user may have chosen "Use Git from Git Bash only" during installation). This also required switching RunWaitOne() from cmd /C to cmd /S /C "..." so that quoted paths containing spaces work correctly alongside the pipe to clip.exe. New compared to the MSYS2 runtime library: CloseMinTTYWindow() handles the "close running processes?" confirmation dialog that MinTTY shows when force-closing a window with child processes still running. The function uses WinClose, then looks for a standard Windows dialog (class #32770) owned by the same MinTTY PID and clicks its OK button (Button1). A single click sometimes visually activates the button but does not dismiss the dialog, likely due to a focus race between the ControlClick and the dialog's own event handling, so the click is retried in a loop until the dialog is gone. ExitWithError uses this instead of the bare WinClose "A" so that error paths also clean up properly. CaptureBufferFromMintty replaces <br> tags with newlines before stripping remaining HTML tags (the MSYS2 version strips all tags indiscriminately, losing line breaks; this was not a problem there because those tests used simpler substring matching). Assisted-by: Claude Opus 4.6 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Add LaunchGitBashViaStartMenu() to the library, which presses the Windows key, types "Git Bash", presses Enter, and waits for a new mintty window to appear. This mirrors the way a user would actually launch Git Bash after installation, and also verifies that the installer correctly registered the Start Menu shortcut. The checklist script now performs its first real test: launch Git Bash via the Start Menu, wait for the bash prompt to appear (verified via mintty's HTML buffer export), and then cleanly exit. Assisted-by: Claude Opus 4.6 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The UI tests need a Git repository with enough history to exercise git log, gitk, and other commands that inspect commit history. Rather than shipping a binary bundle, this shell script outputs a git fast-import stream that produces a reproducible repository with 18 commits across two branches (main and feature) including a merge. The fixed committer identity and timestamps ensure the resulting repository is byte-identical on every run. The script accepts an optional --create-test-repo=<dir> argument that initializes the repository and feeds the stream into git fast-import in one step. This is designed for easy invocation from AutoHotkey, which has no built-in way to pipe the stdout of one subprocess into the stdin of another without buffering the entire output first. Without the argument, the stream is written to stdout for use with a manual pipe. Assisted-by: Claude Opus 4.6 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The subsequent checklist tests (prompt with branch, git log, gitk, git gui) all need a Git repository to work with. Rather than relying on a pre-existing repository, the script creates one in /tmp using the generate-test-repo.sh script via RunWaitOne, which runs git.exe with a shell alias outside of any UI. This happens before Git Bash is launched since repo creation is not itself a UI concern. This also hoists the gitExe path check and the scriptDirUnix conversion to the top of the script so they are available for both the repo creation and any later steps that need them. Assisted-by: Claude Opus 4.6 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
After cd-ing into a Git repository, the Git for Windows bash prompt should display the current branch in parentheses (e.g. "(main)"). This test types "cd <test-repo>" into the Git Bash window and verifies via the mintty HTML export that the prompt contains "(main)". The regex uses the AHK "s)" flag (dot-matches-newline) because the prompt string and the dollar-sign prompt marker are on separate lines in the captured buffer. Assisted-by: Claude Opus 4.6 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Run git log in the test repository and verify two things: that the output is colorful (by checking the raw HTML export for fg-color CSS classes that mintty uses for colored text), and that the pager stops after the first page (by detecting the less pager's ":" prompt in the captured text). After the checks, the pager is dismissed by sending "q", and the test proceeds to the next step. Also add a small sleep in CloseMinTTYWindow before looking for the confirmation dialog, giving MinTTY time to display it. Assisted-by: Claude Opus 4.6 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Launch gitk from the Git Bash window (the way a user would) and verify it renders the expected commit graph by comparing a downscaled screenshot against a reference image. The exit code is also checked via a shell redirect (`gitk; echo $? >gitk-exit.code`). The screenshot approach is necessary because Tk canvas widgets do not expose their content via Win32 text APIs (WinGetText, ControlGetText, and PrintWindow all return blank for the canvas area). BitBlt from the screen DC captures the actual rendered pixels. To avoid comparing against a half-rendered frame, CaptureUntilMatchesReference() repeatedly captures thumbnails and compares each one against the reference image. Once the diff drops below the threshold, the window is considered fully loaded. This replaced an earlier approach that compared consecutive captures to each other, which falsely declared stability when two identical loading frames appeared before the commit graph was fully drawn. The reference image is a heavily downscaled 80x60 PNG that captures the overall shape of the graph, commit rows, and window layout without being sensitive to individual characters. The comparison uses a per-channel tolerance of 10 (out of 255) and allows up to 15% of pixels to differ, making it robust against minor rendering variations while still catching gross failures like an empty window or a completely wrong layout. The library also gains GDI+ helper functions for the capture and comparison: StartupGdiPlus/ShutdownGdiPlus for lifecycle management, GetPngEncoderClsid (hard-coded well-known CLSID to avoid fragile ImageCodecInfo struct traversal), CaptureAndDownscaleWindow (BitBlt + GDI+ HighQualityBicubic resize), CompareImages (pixel-by-pixel with tolerance), and CaptureUntilMatchesReference. Assisted-by: Claude Opus 4.6 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Launch git gui from the Git Bash window and verify it opens without
complaining about a missing repository. The window title is checked
for error strings ("not a git repository", "Error"), and a
downscaled screenshot is compared against a reference image to
confirm the expected layout (menu bar, staging area, commit message
pane). The exit code is verified via a shell redirect, same as the
gitk test.
The same CaptureUntilMatchesReference approach is used, which
retries until the screenshot matches the reference within 15% pixel
difference, handling git gui's progressive rendering and any
blinking cursor in the commit message field.
Assisted-by: Claude Opus 4.6
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Add two checks to run-checklist.sh:
Scan all HTML files in the installed doc directory for raw linkgit:
macros that were not converted to hyperlinks. These occur when the
AsciiDoc source has the macro inside a literal block (indented text
or an explicit .... block), where Asciidoctor does not expand macros
by design. Four files are currently affected (gitformat-commit-graph,
gitformat-index, gitformat-pack, MyFirstContribution) and are
excluded by their exact file:line match. See commits b3ac6e737db8
("doc: fix accidental literal blocks") and 399694384bf9 ("doc:
patch-id: fix accidental literal blocks") in git/git for prior fixes
of the same class of issue. Any new occurrence causes the check to
fail.
Verify that `git help git` opens the correct page by temporarily
configuring a fake browser via browser.fake.cmd that writes the URL
to a temp file. The script then checks that the URL ends in
git.html.
Assisted-by: Claude Opus 4.6
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
At the end of run-checklist.sh, after all the headless checks pass, look for AutoHotkey in PATH and check whether ~/.minttyrc has the required KeyFunctions and SaveFilename settings for the MinTTY buffer export. If both prerequisites are met, run the AHK checklist script that verifies Git Bash starts, the prompt shows the branch, git log is colorful and paged, gitk shows history, and git gui runs without error. If either prerequisite is missing, print an informational message suggesting the user follow checklist.txt for manual verification. The check never fails for missing prerequisites so that the existing headless tests remain usable everywhere. Assisted-by: Claude Opus 4.6 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Add functions for capturing terminal output from Windows Terminal, needed for the Git CMD checklist tests. Windows Terminal's exportBuffer action writes the terminal buffer to a file when triggered via a configurable key binding, similar to how MinTTY's Ctrl+F5 exports to HTML. ReadWindowsTerminalExportBufferConfig() parses the user's settings.json (checking all three known locations documented at https://learn.microsoft.com/en-us/windows/terminal/install: the Microsoft Store stable and preview paths, and the unpackaged install path) to find the exportBuffer action's file path and key binding. The test will skip gracefully if this is not configured, just like the MinTTY tests skip when ~/.minttyrc is missing the required settings. WindowsTerminalHotkeyToAHK() converts Windows Terminal key binding notation (e.g. "ctrl+shift+e") to AutoHotkey Send format ("^+e"). CaptureBufferFromWindowsTerminal() and WaitForRegExInWindowsTerminal() mirror the corresponding MinTTY functions, adapted from the MSYS2 runtime's ui-test-library.ahk. Assisted-by: Claude Opus 4.6 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Launch Git CMD via the Start Menu and verify it opens in Windows
Terminal. The test tracks existing CASCADIA_HOSTING_WINDOW_CLASS
windows before typing "Git CMD" into the Start Menu search, then
waits for a new one to appear.
The Git CMD tests are only run if the user's Windows Terminal has
exportBuffer configured with a file path and key binding. If not,
the tests are skipped with an informational message, just like the
Git Bash tests are skipped when ~/.minttyrc is not configured.
The buffer is captured via the user's configured exportBuffer key
binding and verified to contain a CMD prompt ("> " at end of line).
Assisted-by: Claude Opus 4.6
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Extend the Git CMD section to cover all three remaining checklist items: git log is colorful and paged, gitk runs and shows history, and git gui runs without error. The git log color check uses a screenshot comparison against a reference image since Windows Terminal's exportBuffer output is plain text with ANSI escape sequences stripped. The pager check still uses the text buffer (looking for the ":" prompt). For gitk, the test verifies that the CMD prompt returns immediately after typing "gitk" (confirming gitk.exe is a GUI wrapper that does not block the terminal), then waits for the Tk window and compares its screenshot against the same reference image used by the Git Bash gitk test. The git gui test similarly launches and verifies via screenshot comparison, reusing the Git Bash reference image. Assisted-by: Claude Opus 4.6 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Accept --git-bash, --git-cmd, and --git-gui arguments to run only specific phases of the checklist. With no arguments all phases run, preserving backward compatibility. This makes iterating on a single failing phase much faster since the other phases can be skipped. MinTTY prerequisites are only checked when --git-bash is active, and the export file path is only resolved in that case. This allows running --git-gui or --git-cmd alone without needing ~/.minttyrc. Assisted-by: Claude Opus 4.6 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Add the reference image for the Git GUI chooser dialog, which is the
window that appears when launching Git GUI from the Start Menu
(without being inside a repository). The test uses this to verify
the chooser rendered correctly before clicking the recent repo link.
The screenshot was generated with this AutoHotkey script:
#Requires AutoHotkey v2.0
#Include D:\git-sdk-64\usr\src\build-extra\installer\ui-tests\ui-test-library.ahk
SetLogFile(EnvGet('TEMP') . '\gen-chooser-ref.log')
gitExe := 'C:\Program Files\Git\cmd\git.exe'
testRepoWin := 'C:\Users\johasc\AppData\Local\Temp\git-bash-checklist-test-repo'
testRepoUnix := StrReplace(testRepoWin, '\', '/')
; Backup and replace .gitconfig
gitconfigPath := EnvGet('USERPROFILE') . '\.gitconfig'
gitconfigBackup := gitconfigPath . '.ui-test-backup'
FileMove(gitconfigPath, gitconfigBackup)
OnExit((*) => (FileExist(gitconfigBackup) && (FileDelete(gitconfigPath), FileMove(gitconfigBackup, gitconfigPath))))
FileAppend('[gui]`n`trecentrepo = ' testRepoUnix '`n', gitconfigPath)
chooserHwnd := LaunchViaStartMenu('Git GUI', 'TkTopLevel', 'Git Gui')
WinMove(100, 100, 500, 400, 'ahk_id ' chooserHwnd)
WinActivate('ahk_id ' chooserHwnd)
Sleep 5000
outFile := 'D:\git-sdk-64\usr\src\build-extra\installer\ui-tests\git-gui-chooser-reference.png'
CaptureAndDownscaleWindow(chooserHwnd, 80, 60, outFile)
FileAppend("Saved: " FileGetSize(outFile) " bytes`n", "*")
WinClose('ahk_id ' chooserHwnd)
Sleep 500
FileDelete gitconfigPath
FileMove gitconfigBackup, gitconfigPath
The heavy downscale from the physical window size (625x500 at 125%
DPI) to 80x60 averages away pixel-level differences between DPI
settings, making the reference usable across machines. The test uses
a 20% diff threshold for this particular comparison to provide
additional margin.
Assisted-by: Claude Opus 4.6
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every Git for Windows release requires a manual checklist of UI verification steps: launch Git Bash, confirm the prompt shows the branch name, verify that
git logis colorful and paged, check thatgitkandgit guistart correctly, and so on for Git CMD and the standalone Git GUI. This has been a tedious, error-prone process for years, and one I always itched to automate.The MSYS2 runtime repository already has AutoHotkey v2-based UI tests (
background-hook.ahk,ctrl-c.ahk,keystroke-order.ahk, etc.) that exercise MinTTY and Windows Terminal interactions. Those tests demonstrated that AHK v2 is a viable tool for automated UI testing on Windows, even for applications like MinTTY that do not expose their content through standard Win32 accessibility APIs.This PR applies the same approach to the installer checklist. It automates all of the manual verification steps listed in
installer/checklist.txt:git logis colorful and paged,gitkruns,git guiruns,git help gitopens the right page, and the existingrun-checklist.shtests pass.git logis colorful and paged,gitkruns,git guiruns.The tests are integrated into
run-checklist.sh: when AutoHotkey is available and the MinTTY prerequisites are configured, the UI tests run automatically; otherwise they skip gracefully with an informational message. This keepsrun-checklist.shworking in headless/CI environments exactly as before.A few design choices worth calling out:
gitk,git gui) do not expose their content via Win32 APIs, so the tests use screenshot-based verification with heavily downscaled reference thumbnails (80x60). This keeps comparisons robust across minor rendering differences while still catching real problems (wrong window, missing content).git fast-import, so they do not depend on any pre-existing checkout.ui-test-library.ahkis adapted from the MSYS2 runtime's version for the installer's specific needs (MinTTY buffer capture, screenshot comparison, Start Menu launching).