Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 64 additions & 8 deletions pkgs/build-support/replace-vars/default.nix
Original file line number Diff line number Diff line change
@@ -1,34 +1,90 @@
{ lib, stdenvNoCC }:
{
lib,
stdenvNoCC,
doCheck ? true,
}:

/**
`replaceVars` is a wrapper around the [bash function `substitute`](https://nixos.org/manual/nixpkgs/stable/#fun-substitute)
`replaceVars` is a wrapper around the [bash function `substitute`](https://nixos.org/manual/nixpkgs/unstable/#fun-substitute)
in the stdenv. It allows for terse replacement of names in the specified path, while checking
for common mistakes such as naming a replacement that does nothing or forgetting a variable which
needs to be replaced.

As with the [`--subst-var-by`](https://nixos.org/manual/nixpkgs/stable/#fun-substitute-subst-var-by)
As with the [`--subst-var-by`](https://nixos.org/manual/nixpkgs/unstable/#fun-substitute-subst-var-by)
flag, names are encoded as `@name@` in the provided file at the provided path.

Any unmatched variable names in the file at the provided path will cause a build failure.

Any remaining text that matches `@[A-Za-z_][0-9A-Za-z_'-]@` in the output after replacement
has occurred will cause a build failure.
has occurred will cause a build failure. In the case this step is not desired, use
`replaceVars.withoutCheck`, which has the same semantics but doesn't have a check phase making
the assertion about remaining unreplaced variables.

# Inputs

`path` ([Store Path](https://nixos.org/manual/nix/latest/store/store-path.html#store-path) String)
: The file in which to replace variables.

`attrs` (AttrsOf String)
: Each entry in this set corresponds to a `--subst-var-by` entry in [`substitute`](https://nixos.org/manual/nixpkgs/stable/#fun-substitute).
: Each entry in this set corresponds to a `--subst-var-by` entry in [`substitute`](https://nixos.org/manual/nixpkgs/unstable/#fun-substitute).

# Example
# Examples

## `replaceVars`

Given a file `greeting.txt` that contains the following contents:

```
@greeting@! Have you heard about the way of the Jedi?
```

And a derivation that looks like this:

```nix
{ replaceVars }:

replaceVars ./greeting.txt { world = "hello"; }
replaceVars ./greeting.txt { greeting = "Hello there"; }
```

The resulting file in `$out` contains this text:

```
Hello there! Have you heard about the way of the Jedi?
```

## `replaceVars.withoutCheck`

Given a file named `template.json` that contains the following contents:

```
{
"key": "@key@",
"value": @value@
}
```

And a derivation that looks like this:

```nix
{ replaceVars }:

replaceVars.withoutCheck ./template.json { key = "master"; }
```

The resulting file in `$out` contains this text:

```
{
"key": "master",
"value": @value@
}
```

This allows chaining together `replaceVars` calls in subsequent derivations or the use of the
`substituteInPlace` stdenv function to complete the replacements, at the cost of potentially
leaving unreplaced material in the output.

# Tests

See `../../test/replace-vars/default.nix` for tests of this function.
*/
Expand All @@ -48,7 +104,7 @@ in
stdenvNoCC.mkDerivation {
name = baseNameOf (toString path);
src = path;
doCheck = true;
inherit doCheck;
dontUnpack = true;
preferLocalBuild = true;
allowSubstitutes = false;
Expand Down
25 changes: 25 additions & 0 deletions pkgs/test/replace-vars/default.nix
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,23 @@ in
'';
};

# Success case for `replaceVars.withoutCheck`.
replaceVars-without-check = testEqualContents {
assertion = "replaceVars-without-check";
actual = replaceVars.withoutCheck ./source.txt {
"equal in" = "are the same in";
brotherhood = "shared humanity";
};

expected = builtins.toFile "expected" ''
All human beings are born @free@ and are the same in dignity and rights.
They are endowed with reason and conscience and should act towards
one another in a spirit of shared humanity.

-- eroosevelt@humanrights.un.org
'';
};

# There might eventually be a usecase for this, but it's not supported at the moment.
replaceVars-fails-on-directory =
runCommand "replaceVars-fails" { failed = testBuildFailure (replaceVars emptyDirectory { }); }
Expand All @@ -43,6 +60,14 @@ in
touch $out
'';

replaceVars-without-check-fails-in-build-phase =
runCommand "replaceVars-fails"
{ failed = testBuildFailure (replaceVars.withoutCheck emptyFile { not-found = "boo~"; }); }
''
grep -e "ERROR: pattern @not-found@ doesn't match anything in file.*empty-file" $failed/testBuildFailure.log
touch $out
'';

replaceVars-fails-in-check-phase =
runCommand "replaceVars-fails"
{
Expand Down
4 changes: 3 additions & 1 deletion pkgs/top-level/all-packages.nix
Original file line number Diff line number Diff line change
Expand Up @@ -1329,7 +1329,9 @@ with pkgs;

replaceDependency = callPackage ../build-support/replace-dependency.nix { };

replaceVars = callPackage ../build-support/replace-vars { };
replaceVars = callPackage ../build-support/replace-vars { } // {
withoutCheck = callPackage ../build-support/replace-vars { doCheck = false; };
};
Comment on lines +1332 to +1334
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t understand how this works. (x: x) // { } is an error. Is this some __functor magic? I think I would prefer something like replaceVarsWith { doCheck = false; } ./file { … }, where replaceVars = replaceVarsWith { }.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this some __functor magic?

Yes, kind of. An attrset with __functor is also just an attrset.

nix-repl> replaceVars = callPackage ./pkgs/build-support/replace-vars { }

nix-repl> replaceVars
{
  __functionArgs = { ... };
  __functor = «lambda __functor @ /home/philip/Code/github.com/philiptaron/nixpkgs/lib/trivial.nix:957:19»;
  override = { ... };
}

nix-repl> replaceVars // { a = 42; }
{
  __functionArgs = { ... };
  __functor = «lambda __functor @ /home/philip/Code/github.com/philiptaron/nixpkgs/lib/trivial.nix:957:19»;
  a = 42;
  override = { ... };
}

It's used a couple places in all-packages.nix:

  1. https://github.com/NixOS/nixpkgs/blob/6cc3e274c947096556c57d2b05dad5acda395b74/pkgs/top-level/all-packages.nix#L4036-L4038
  2. https://github.com/NixOS/nixpkgs/blob/6cc3e274c947096556c57d2b05dad5acda395b74/pkgs/top-level/all-packages.nix#L15825 (where I learned this trick)
  3. https://github.com/NixOS/nixpkgs/blob/6cc3e274c947096556c57d2b05dad5acda395b74/pkgs/top-level/all-packages.nix#L33290-L33295
  4. https://github.com/NixOS/nixpkgs/blob/6cc3e274c947096556c57d2b05dad5acda395b74/pkgs/top-level/all-packages.nix#L33961-L33964

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll note that callPackage is used in cases like this where it doesn't actually resolve to a package, but rather to a factory function that will then produce packages/derivations. If there were something like a callPackageFactory function that included the ability to passthru additional contents, this would look quite a bit more sane.

The core piece of code that's creating this functor attrset is

then makeOverridable f allArgs

As it is, I'm pretty sure that this specific feature doesn't yet justify replaceVarsWith, but if you feel quite strongly, I could introduce it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I figured callPackage might be doing something like this. I just think __functor is weird and people don’t expect to find more functions under a function.

The nice thing about replaceVarsWith is that it would compose with any future extensions, like if we want to support postPatch or something. If you’re totally sure that there’ll never be another option ever again then maybe it could just be replaceVarsWithoutCheck?

Copy link
Copy Markdown
Contributor Author

@philiptaron philiptaron Sep 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

people don’t expect to find more functions under a function

One of the reasons I tilted this (odd) way was that I was able to document this variant "under" replaceVars so that it appeared in context on https://noogle.dev/f/pkgs/replaceVars. That documentation outweighs the oddity of people finding or failing to find functions under a function... I think. 😁

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @emilazy that __functor should be avoided when possible, it can be very confusing. I like the idea of replaceVarsWith more, especially because the same pattern is being used elsewhere too. Check out the docs for lib.fileset.gitTracked as an example how such a pair of functions can be crosslinked together in the docs, but nesting one under the other would also be fine imo.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All right, I'll do replaceVarsWith.


nukeReferences = callPackage ../build-support/nuke-references {
inherit (darwin) signingUtils;
Expand Down