Fixing `.override` interface stability

[pbsds]

Fixing override interface stability

TLDR: see the proposed "Option 3" for my personal favorite.

Note

I initially typed this up before looking for prior art, then I found #273534 which seems to have stalled aiming for perfection. Rather than post this there and risk it becoming noise or derailing progress over there I opted to open this new issue.

Currently the nixpkgs guidelines recommends preserving the .override interface.
Consider a bar by-name package which depends on a special version of foo, say foo_1_2_3: the current guidelines recommend the following pattern:

# pkgs/by-name/ba/bar/package.nix:
{
    foo
    ...
}: mkDerivation { ... foo ... }

# pkgs/top-level/all-packages.nix:
{
    ...
    bar = ../by-name/ba/bar/package.nix { foo = foo_1_2_3 };
    ...
}

In my opinion this pattern has at least two big issues:

• Poor code-locality: domain knowledge about the bar package is spread wide and becomes less discoverable.
• It directly refers to by-name paths in all-packages.nix. This makes future by-name refactoring like for example sharding pkgs/by-name/li/ into multiple longer prefixes (to account for all the lib* packages) a much larger undertaking.

I've observed more and more maintainers and committers (myself often included) disregard this guideline and just replace the package input with the special version:

# pkgs/by-name/ba/bar/package.nix:
{
    foo_1_2_3,
    ...,
}: mkDerivation { ... foo_1_2_3 ...; }

This breaks bar.override { foo = ...; } expressions in both nixpkgs and for downstream users.
It is also prone to cause extra review cycles depending on what a reviewer prefers or consider to be a blocker, making this issue prone to trigger multiple code-review antipatterns.

The problem stems from the fact that if an attribute exist in the scope, then callPackage will provide it as an input arguments even if that argument has a default value.
This also leaves us with this footgun:

# pkgs/by-name/ba/bar/package.nix:
{
    foo_1_2_3,
    foo ? foo_1_2_3, # default is not used
    ...,
}: mkDerivation { ... foo ...; } # ERROR: `foo` is `pkgs.foo`, not `pkgs.foo_1_2_3`

This risks becomes a silent error if the incorrect foo input don't result in a build failure but instead a runtime error.

Proposed options

Option 1: Don't consider .override interfaces stable.

This is the path of least resistance, and I've seen multiple contributors express this preference, partly due to #204303.
This option also avoids hammering down exactly what stability guarantees we even aim to provide.

To what degree do we actually consider using .override to be stable?
Should we support all enableX/disableX/withX flags indefinitely?
Should we try and accept (and ignore) all previously accepted package inputs?
When is it okay and not okay to change the interface?

Increasingly as package definitions that just .override some other package get migrated from all-packages.nix to by-name, we see more and more .override interfaces differ because by-name will via callPackage replace any pre-existing .override in the output with a new makeOverridable invocation.
An example if this is SDL2_mixer_2_0 whose .override interface does not match SDL2_mixer at all.

However going ahead with this option will make nixpkgs less reliable and powerful for downstream users.

Option 2: making callPackage smarter

We could extend callPackage to allow the callee to somehow inform callPackage what not to provide as initial arguments.

My dream implementation would look something like:

# pkgs/by-name/ba/bar/package.nix:
{
    lib,
    foo_1_2_3,
    foo ? lib.mkCallPackageDefault foo_1_2_3,
    ...,
}: mkDerivation { ... foo ...; }
# here ^ `foo` is `pkgs.foo_1_2_3` unless `bar` is explicitly overridden with `bar.override { foo = ...; }`

But this is to my knowledge not possible, since our only tool for the job builtins.functionArgs only informs us if a function argument has a default value but does not allow us to inspect what that default value is.
As a workaround I envision two solutions to extend callPackage:

A non-solution: Not setting arguments that have default values

While this would make

# pkgs/by-name/ba/bar/package.nix:
{
    foo_1_2_3,
    foo ? foo_1_2_3,
    ...,
}: mkDerivation { ... foo ...; }

"just work", it would also be a massive treewide breaking change.

It makes package definitions non-reusable across different scopes where some optional inputs are only available in some of the scopes.
This means the stdenv bootstrap for example would require a massive refactoring with likely high amounts of code duplication or hacky workarounds like { pkgs, baz ? pkgs.baz or null }: ....

Solution A: Extend callPackage to act on presence of magic arguments

We could make callPackage not provide certain inputs if the input has a matching __callPackage_ignore_* input.

# pkgs/by-name/ba/bar/package.nix:
{
    foo_1_2_3,
    foo ? foo_1_2_3,
    __callPackage_ignore_foo ? null,
    ...,
}: mkDerivation { ... foo ...; }

This change however affects the design space for NixOS/rfcs#169 and should be considered jointly.

Solution B: wrap package.nix expressions with a special type that callPackage acts on

This could look something like

# pkgs/by-name/ba/bar/package.nix:
{ lib }:
lib.callPackageIgnore [ "foo" ] (
    {
        lib,
        foo_1_2_3,
        foo ? foo_1_2_3,
        ...,
    }:
    mkDerivation { ... foo ... }
)

where callPackage then checks the result for some magic attribute set by lib.callPackageIgnore (like __callPackageIgnore = [ "foo" ]), and if present recursively applies a second callPackage.

However this design is prone to not play well with the formatter which puts all the git blame on you.
It also has a silent footgun where nixpkgs contributors and downstream users instead might set passthru.__callPackageIgnore inside the package instead of using lib.callPackageIgnore properly, which both increases the eval cost with multiple mkDerivation invocations, and is prone to cause infinite recursions.

(I also have great trouble in both picking a good intuitive name for this function and designing a interface for it that I like.)

Option 3: Extend the by-name interface

Rather than change callPackage we could extend the by-name interface and machinery.

Consider by-name checking for an additional optional override.nix file, whose output replaces the output of callPackage ./by-name/{shard}/{pname}/package.nix {};, and whose output is not wrapped with lib.makeOverrideable:

# pkgs/by-name/ba/bar/package.nix:
{
    foo,
    ...,
}: mkDerivation { ... foo ...; }

# pkgs/by-name/ba/bar/override.nix:
{
    foo_1_2_3,
}:
package:
package.override { foo = foo_1_2_3; }

This override.nix pattern could be made simpler using a more task-specific attrSet -> attrSet interface rather than attrSet -> nullOr overridableDrv -> any, but this is both way less powerful and already discussed in RFC140 as not a big enough of a value-add.

The attrSet -> nullOr overridableDrv -> any interface will allow us to migrate cases like the aforementioned SDL2_mixer_2_0 example to instead just define a override.nix file with no package.nix file (in which case the package input in override.nix would become null), fixing the SDL2_mixer/SDL2_mixer_2_0 override interface mismatch.
It would also enable switching to a different package set, resulting in the resulting package becoming simpler to override:

# pkgs/by-name/ba/bar/override.nix:
{ python3Packages }:
package: # if we never use/force `package` there should be no issues with its arguments not matching `pkgs` thanks to nix being lazy
python3Packages.callPackage ./package.nix { };

Since the override.nix inputs are provided by callPackage (sans makeOverrideable), we get get to override with correctly spliced packages and package sets, alleviating #204303.

(The override.nix name is actually a bit narrow in conveying its true power/affordance, perhaps apply.nix, package-apply.nix, or even just expr.nix is better?)

---

These options are not perfect, but they are actionable.
Personally I prefer option 3.
It would also allow us to eliminate all ../by-name/**/* references treewide, making by-name packages truly filename agnostic, which once achieved we may continue to enforce with CI.

We could in addition, since this option is orthogonal to option 3, also implement option 2A to support cases outside of by-name or RFC169.

[ruro]

Can you clarify your last example?

# pkgs/by-name/ba/bar/override.nix:
{ python3Packages }:
package: # if we never use/force `package` there should be no issues with its arguments not matching `pkgs`
python3Packages.callPackage ./package.nix { };

When would using python3Packages.callPackage like that be actually desirable? I might be misunderstanding something, but wouldn't this make bar only available for the latest version of python, with no way for the user to override the python3Packages itself with a different version.

Wouldn't it almost always be preferable to use /pkgs/top-level/python-packages.nix? Or is this override.nix proposal intended to also slowly replace top-level/python-packages.nix like by-name is replacing all-packages.nix?

[RossSmyth]

Also check out #273815 & #296769 for prior art

[pluiedev]

I think it would be cool if Option 3 can also generate multiple different "variants" of the same package, which is a pattern that I see often in cases where there's a lot of overlap between packages that nevertheless should remain separate out of convenience: e.g. attic-server (which is literally just an override of attic-client), or Qt themes like #367614 and #318772 that can be compiled against Qt 5 or 6 depending on which version of qtPackages they're using.

For #367614 and #318772 in particular, I currently had to add an additional entry under by-name and all it does is to override qtPackages with libsForQt5. That was a common point of contention from reviewers due to how "unnatural" it felt, but experimenting with multiple other approaches, it was essentially the cleanest method that I've found. Having an override.nix inside the original package would not only help with code locality, but it would open up room for maintainers to add more variants down the road. I could also see it being useful for package wrappers, to name some more examples.

[pbsds]

Can you clarify your last example?
[...]
When would using python3Packages.callPackage like that be actually desirable?
[...]

This is a proposed migration for a pretty common pattern, which can be observed with

• rg python3Packages.callPackage pkgs/top-level/all-packages.nix
• rg python3Packages.buildPythonApplication pkgs/by-name/

User facing applications may go in pkgs instead of python3Package, since they will never be import'ed by other python modules. Libraries on the other hand must go in python3Packages to be made available for every interpreter.

This saves compute, not having to build every user-facing python application twice

EDIT:

When would using python3Packages.callPackage like that be actually desirable?

Rather than by-name/fo/foo/package.nix importing python3Packages as an argument, it would import separate names from python3Packages like requests and typer, which then becomes a part of the .override interface.

[pbsds]

I think it would be cool if Option 3 can also generate multiple different "variants" of the same package, which is a pattern that I see often in cases where there's a lot of overlap between packages [snip]

For #367614 and #318772 in particular, I currently had to add an additional entry under by-name and all it does is to override qtPackages with libsForQt5. [snip]

By-name by design disallows defining attribute names by any other means than using the directory structure. Option 3 would let you migrate pkgs/by-name/da/darkly-qt5/package.nix to pkgs/by-name/da/darkly-qt5/override.nix and pkgs/by-name/kl/klassy-qt5/package.nix to pkgs/by-name/kl/klassy-qt5/override.nix, which avoids clobbering the .override interface of the original packages.

RFC140 considered anything requiring recurseIntoAttrs a future problem, but a possible extension may be supporting a packages.nix which would be evaluated with callPackages instead of callPackage, allowing you to migrate to using nested attribute sets, i.e. darkly.{qt5,kde} and klassy.{qt5,kde}. Until something like packages.nix is implemented and available however, override.nix would let you call callPackages yourself directly.

EDIT: this may look like:

# pkgs/by-name/da/darkly/override.nix
{ lib, callPackages }:
package:
lib.recurseIntoAttrs (callPackages ./packages.nix {});

# pkgs/by-name/da/darkly/packages.nix
{
 callPackage,
 libsForQt5,
 kdePackages,
}:
{
  qt5 = callPackage ./common.nix { qtPackages = libsForQt5; };
  kde = callPackage ./common.nix { qtPackages = kdePackages; };
}

# pkgs/by-name/da/darkly/common.nix
{ ... }: stdenv.mkDerivation { ... }

[ruro]

User facing applications may go in pkgs instead of python3Package, since they will never be import'ed by other python modules. Libraries on the other hand must go in python3Packages to be made available for every interpreter.

Hm. I am aware of this pattern, however I am not 100% sure if this is a pattern that we should encourage.

1. Due to python's general "we are all consenting adults" approach, it is (in my experience) quite common for experienced users to want to directly import code from python "applications" (rather than serializing the arguments to command line arguments and invoking the application as a subprocess). Additionally, python applications quite often offer plugin/extension mechanisms, which just end up importing user supplied code.

2. This pattern hard codes a specific instance of python. Consider the case, where the user wants to use this package with a different python instance (not even necessarily a different version of python, but possibly python with some patches applied or with enabled optimizations).

   With this pattern, you have to either overlay the whole nixpkgs instance, globally replacing python3 (and end up rebuilding half the world), or to do stuff like

   myPython3Packages.callPackage "${nixpkgs}/pkgs/by-name/ba/bar/package.nix" {}

   which is problematic for a bunch of different reasons.

Also, regarding your grep python3Packages.callPackage and buildPythonApplication samples. I am not sure that this is a fair indication of the popularity/prevalence of this pattern. For example, compare

rg python3Packages.callPackage pkgs/top-level/all-packages.nix | wc -l
134

and

rg toPythonApplication pkgs/top-level/all-packages.nix | wc -l
308

It feels to me, that (at least for packages directly available on pypi) it makes more sense to create a derivation via python-packages.nix and then expose it in all-packages.nix via toPythonApplication or similar, if it is a standalone package.

---

P.S. Is there some prior discussion where this pattern was discussed and accepted as desirable?

[pbsds]

I managed to replace a python312Packages scope with a python313Packages scope with this cursed invocation

termdown.override (lib.intersectAttrs termdown.override.__functionArgs python313Packages)

which I then realized turns out to be equivalent to

python313Packages.callPackage termdown.override {}

TIL

Preferably however callPackage would also expose the raw underlying package blueprint/lambda, which I know others have expressed interest in.

P.S. Is there some prior discussion where this pattern was discussed and accepted as desirable?

Keeping track of consensus is not quite yet nixpkgs' strong suit. Python however has a contributing guidelines section in the manual.