Benchmark runners may commit results that contain merge conflicts

[mpage]

Benchmark runners may commit results that contain merge conflicts. We ran into this recently:

• One of our weekly runs committed benchmarking results for commit 2a66dd3.
• Afterwards, another benchmark run ran for the same commit, and committed results that contain merge conflicts.

It looks like we try to handle this situation:

bench_runner/bench_runner/templates/_benchmark.src.yml

Lines 131 to 135 in d0569e7

	# Another benchmarking task may have created results for the same
	# commit while the above was running. This "magic" incantation means
	# that any local results for this commit will override anything we
	# just pulled in in that case.
	git pull -s recursive -X ours --autostash --rebase

However, the approach that we're using doesn't work. It looks like -X ours doesn't apply to resolving merge conflicts that result from applying stashed changes. I verified this locally by:

1. Creating a new repo containing a single markdown file.
2. Cloning the repo into a new directory.
3. Editing the markdown file in a way that would generate a merge conflict.
4. Ran git pull -s recursive -X ours --autostash --rebase. The merge conflicts are left for resolution.

I'm not sure if this is a behavior that changed in git recently. I've tested this using git versions 2.39.5 and 2.47.1.

[mpage]

@mdboom - How large/invasive of a change would it be to store the results for each job in a unique file?

[mdboom]

It would be pretty invasive to use a unique name for each job -- plus it's occasionally useful to re-run a job that failed for some external reason (e.g. a network outage) and replace any existing results.

I wonder if there is a magic incantation that would always replace any upstream changes (e.g. like this line but that actually works). I'll poke around and see if I can find something that works...

[mdboom]

Ugh, the solution I arrived at broke on Windows... Back to the drawing board for a bit.