sort: Fix inconsistent sort orderg under i18n-collator with equal sorting keys.

[ksgk1]

In #11980 it was noted, that the sorting behaviour was inconsistent. The cause for this was equal sorting keys, without a fallback option.

I implemented a fallback for equal keys, that fixes this issue.

[github-actions]

GNU testsuite comparison:

Skipping an intermittent issue tests/pr/bounded-memory (passes in this run but fails in the 'main' branch)
Note: The gnu test tests/cp/link-heap is now being skipped but was previously passing.
Congrats! The gnu test tests/tail/pipe-f is now passing!

[codspeed-hq]

Merging this PR will improve performance by 31.27%

⚡ 1 improved benchmark
✅ 310 untouched benchmarks
⏩ 46 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Memory	cp_recursive_deep_tree[(120, 4)]	699.2 KB	532.7 KB	+31.27%

---

_{Comparing ksgk1:sort-fix-inconsistent-sort-behaviour-i18n-collator (573eaa0) with main (5aade31)}

Footnotes

1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[cakebaker]

Can you please add a test to tests/by-util/test_sort.rs to ensure we don't regress in the future? Thanks.

[ksgk1]

I updated my code, and added different test cases.

[cakebaker]

I think this test is not correct as GNU sort returns a different result when running the command manually:

$ LC_ALL=en_US.UTF-8 sort tests/fixtures/sort/fix_i18n_collate_inconsistency_1.txt tests/fixtures/sort/fix_i18n_collate_inconsistency_2.txt 
0_1
0_1
01
01
02
02

[ksgk1]

Thank you a lot for the valuable feedback. I will fix those issues as soon as possible and verify my implementation against sort's output.