Sort changed files before building the AI commit message prompt

[gk-bbai]

Description

When generating an AI commit message, changed files are added to the prompt in a non-deterministic order. Regenerating on an unchanged working tree produces a different file order each time.

Steps to reproduce

1. Configure Ollama as the AI provider.
2. Have a multi-file change in the working tree.
3. Click Generate message several times without changing anything.

Result

The diff content is the same between requests, but the per-file order differs each time.

Expected

A fixed file order for identical input.

Effects

• Same changeset yields differently-ordered prompts, so messages vary between regenerations.
• Reordering invalidates the local runtime's prompt cache on each regeneration.

Suggested fix

Sort the change set by file path before assembling the prompt.

Related

• AI commit messages are low quality #8755

Env: GitButler 0.20.3 (macOS), Ollama provider.

[Byron]

Thanks a lot for reporting.

I took a look and the source of the shuffle seems to be here:

gitbutler/apps/desktop/src/lib/ai/service.ts

Line 401 in 100d081

buildDiff(diffInput, diffLengthLimit),

The shuffle seems to be intentional as tests are actively working around the non-deterministic order.

Codex traced it back to commit bffcdbd:

Introduced hunk shuffling. This means that we're not going to fill up the context with 5000 characters of one particular file, and in stead take a smattering of changes from all the files present.

This commit seems to have been merged with #3741, which means that @estib-vega may have some context (it's from more than 2 years ago though 😅).

Codex

I traced this to buildDiff() in apps/desktop/src/lib/ai/service.ts.

The commit message path is:

CommitMessageEditor.svelte -> AIMacros.generateCommitMessage() -> AIService.summarizeCommit() -> buildDiff()

The nondeterministic order comes from buildDiff(), which calls shuffle(). shuffle() uses Math.random(), so identical input can produce a differently ordered prompt on each regeneration.

Upstream inputs are already generally stable/file-tree sorted via sortLikeFileTree(), for example in selectedChanges() and fetchCommitChanges(). The randomization happens later in buildDiff(), overriding that stable order.

There’s also a test that currently encodes the random behavior by accepting either ordering in service.test.ts.

So the likely fix is to remove the shuffle and sort by filePath before joining, or preserve the upstream order if file-tree order is preferred.