active-repositories: Don't override fully-deprecated packages

[erikd]

This PR takes over #8997 from the original author Alexander Esgen who unfortunately passed away late last year.

The PR includes tests and additions to the documentation.

When active-repositories includes a repo with :override, and that repo's preferred-versions marks all its versions of a package as deprecated, the index-combining step previously still applied full override semantics, hiding all versions of that package from earlier repos.

Fix by consulting preferred-versions when combining indexes: if lookupDependency finds no preferred version for a package in the override repo, fall back to merge semantics so earlier-repo versions remain visible.

Two changes:

• PackageIndex: add overrideOrMerge, a per-package Override/Merge strategy
• IndexUtils: add deprecationAwareStrategy, wired into getSourcePackagesAtIndexState

Fixes #8502

This code was originally authored by Alexander Esgen and sumbitted in a PR over 2 years ago. Erik manually rebased that code onto master and used Claude to add tests and do some very minor refactoring to Alexander's code to make it more testable.

Please read Github PR Conventions and then fill in one of these two templates.

---

Include the following checklist in your PR:

• Patches conform to the coding conventions.
• Any changes that could be relevant to users have been recorded in the changelog.
• Is the change significant? If so, remember to add significance: significant in the changelog file.
• The documentation has been updated, if necessary.
• Manual QA notes have been included.
• Tests have been added. (Ask for help if you don’t know how to write them! Ask for an exemption if tests are too complex for too little coverage!)

[erikd]

Tagging @andreabedini because he had some comments on Alexanders original PR.

@Mikolaj In July last year you asked if Alexander's PR could be moved forward. As I stated at the top, I have taken this over after Alexander passed.

[Mikolaj]

@Mikolaj In July last year you asked if Alexander's PR could be moved forward. As I stated at the top, I have taken this over after Alexander passed.

Oh, I didn't now about Alexander. I'm honoured to help carry onward his work. Let me also mark @geekosaur, who reviewed the original PR. @erikd, please don't forget to set the needs-review label as soon as you'd like to widen the pool of reviewers.

[Mikolaj]

An initial technical question: does the test have the property that it fails without the PR and succeeds with it? I assume the tests from #11684 don't have this property and so they just check this PR doesn't break how active-repositories should work, right?

[erikd]

@Mikolaj If I back out some of the changes in cabal-install/src/Distribution/Client/IndexUtils.hs then 3 out of the 610 tests (running cabal test cabal-install:unit-tests) fail. They all passed before I backed out the changes.

Otherwise, yes, the tests in #11684 were just to test the "active-repositories" features so that these changes did not break that functionality.

[Mikolaj]

Good to know. Are the failing tests the newly added overrideOrMergeTests and/or deprecationAwareStrategyTests?

[erikd]

Inspecting the failures more closely, they are a) intermittent and b) not related to the changes in this PR. For some reason a couple of tests in the UnitTests.Distribution.Client.UserConfig section fail intermittently with things like:

  UnitTests.Distribution.Client.UserConfig
    nullDiffOnCreate:                                                                        OK (0.02s)
    canDetectDifference:                                                                      FAIL
      Exception: /home/erikd/Git/Haskell/cabal/cabal-install/tests/fixtures/project-root/
         test-user-config.tmp: withFile: resource busy (file is locked)
      
      HasCallStack backtrace:
        ioException, called at libraries/ghc-internal/src/GHC/Internal/IO/FD.hs:331:17 in
          ghc-internal:GHC.Internal.IO.FD
      
      Use -p '/canDetectDifference/' to rerun this test only.

I did some further refactoring to expose addIndex so I could add unit tests specifically testing the Merge/Orverride/Skip logic.

There are still separate tests for addIndex and addStategy. Having separate (and cheap execution wise) tests is still worthwhile even though addIndex with an empty [Dependency] list behaves identically to applyStrategy for all three strategies.

Rebased against master as well.

[philderbeast]

Inspecting the failures more closely, they are a) intermittent and b) not related to the changes in this PR.

Yes, they're not related. These failures are reported in #11686 with a fix in flight with #11759.