libexpr: specialize builtins.elem scans over attribute set needles

[bouk]

Motivation

I ran Claude Fable 5 (earlier this week before it got pulled) with the prompt 'make this nix eval 2x as fast' and let it run overnight to see what it came up with. This was one of the highest-impact changes, with a -5% improvement in runtime for a medium-complex NixOS configuration.

Context

I've human-reviewed the change and it makes sense to me, but I'm not knowledgeable enough about the nix codebase to say if there's a better way to structure this.

---

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[xokdvium]

1. would be nice to understand what's causing so many derivation equality checks and what the responsible code pattern is.

2. Maybe eqValues should be reworked in such a way that it can work with a pre-forced value if that's actually the problem here.

[xokdvium]

Hm, what's this pointer equality doing here? I'd be incredibly vary of introducing even more dubious pointer equality short-circuits, since it might change the semantics in a very annoying/hard to understand way.

[xokdvium]

I seem to recall that nixpkgs had some idiotic usage of lib.lists.unique to deduplicate derivations (primarily used in Python packages) and it has quadratic complexity. Could it be that?

[bouk]

We do have a bunch of Python packages on our system, I'll try to figure it out this week

[xokdvium]

I think @llakala reminded me of that issue and they were fixing it in nixpkgs?

[llakala]

I tried to do that last week, but was getting a change in hash for python for an unclear reason. This diff was resulting in markupsafe being removed as a dependency of python itself (transitive through mako). Still not sure why.

   requiredPythonModules =
-    drvs:
     let
-      modules = lib.filter hasPythonModule drvs;
+      withKey = pkg: {
+        key = pkg.outPath;
+        value = pkg;
+      };
+      collect = e: if e ? requiredPythonModules then map withKey e.requiredPythonModules else [ ];
     in
-    lib.unique (
-      [ python ] ++ modules ++ lib.concatLists (lib.catAttrs "requiredPythonModules" modules)
+    drvs:
+    lib.map (e: e.value) (
+      builtins.genericClosure {
+        startSet = map withKey ([ python ] ++ lib.filter hasPythonModule drvs);
+        operator = collect;
+      }
     );