Skip to content

Commit ae34501

Browse files
jfrochejfroche
authored
feat(darwin): add nix-darwin host and linux-builder for NixOS tests (#1999)
* feat(darwin): add nix-darwin host and linux-builder for NixOS tests Running NixOS tests on macOS requires a Linux VM capable of building NixOS VMs. This adds a nix-darwin configuration that provisions an ephemeral QEMU linux-builder with support for aarch64-linux and x86_64-linux builds. Usage: nix run github:supabase/postgres#setup-darwin-linux-builder Enables running NixOS integration tests from macOS development machines (see #1989). * docs: add guide for running NixOS tests on macOS Document the linux-builder setup for macOS users who want to run NixOS tests locally using the nix-darwin configuration. * feat: add verification script for darwin-linux-builder Add verify-darwin-linux-builder script that checks: - Launchd service status (running vs loaded-but-stopped) - Nix configuration via nix config show (substituters, trusted keys, experimental features) - Builder features from /etc/nix/machines (includes nixos-test) - Builder responsiveness (test build with 60s timeout) The script is available via: - nix run .#verify-darwin-linux-builder (standalone) - verify-darwin-linux-builder (after setup, as system package) Setup now runs verification automatically after darwin-rebuild.
1 parent cb314c1 commit ae34501

9 files changed

Lines changed: 570 additions & 0 deletions

File tree

‎flake.lock‎

Lines changed: 21 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎flake.nix‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,8 @@
1313
flake-utils.url = "github:numtide/flake-utils";
1414
git-hooks.inputs.nixpkgs.follows = "nixpkgs";
1515
git-hooks.url = "github:cachix/git-hooks.nix";
16+
nix-darwin.inputs.nixpkgs.follows = "nixpkgs";
17+
nix-darwin.url = "github:nix-darwin/nix-darwin";
1618
nix-editor.inputs.nixpkgs.follows = "nixpkgs";
1719
nix-editor.inputs.utils.follows = "flake-utils";
1820
nix-editor.url = "github:snowfallorg/nix-editor";
@@ -47,6 +49,7 @@
4749
nix/devShells.nix
4850
nix/fmt.nix
4951
nix/hooks.nix
52+
nix/hosts.nix
5053
nix/nixpkgs.nix
5154
nix/packages
5255
nix/overlays

‎nix/docs/nixos-tests-on-macos.md‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
## Prerequisites
2+
3+
Running NixOS tests on macOS requires a Linux builder VM because NixOS tests need a Linux environment.
4+
This project includes a nix-darwin configuration that sets up a linux-builder VM automatically.
5+
6+
You need:
7+
- macOS with Apple Silicon (aarch64-darwin)
8+
- Nix installed (see [Getting Started](start-here.md)) preferably a recent version (2.30+)
9+
10+
## Setup
11+
12+
Run the setup script to configure nix-darwin with the linux-builder:
13+
14+
```bash
15+
nix run .#setup-darwin-linux-builder
16+
```
17+
18+
Note that you don't have to checkout the repository to run the setup, you can run it directly from GitHub:
19+
20+
```bash
21+
nix run github:supabase/postgres#setup-darwin-linux-builder
22+
```
23+
24+
This command will:
25+
- Back up existing system files (`/etc/nix/nix.conf`, `/etc/bashrc`, `/etc/zshrc`)
26+
- Configure nix-darwin with the linux-builder VM
27+
- Install helper scripts for managing the builder
28+
29+
The linux-builder VM is configured with:
30+
- 6 CPU cores
31+
- 8GB RAM
32+
- 40GB disk
33+
- Support for both x86_64-linux and aarch64-linux builds
34+
- The `nixos-test` feature required for running NixOS tests
35+
36+
After setup completes, restart your shell to access the helper commands.
37+
38+
## Verify the setup
39+
40+
The setup script runs verification automatically after configuration.
41+
You can also run verification manually at any time:
42+
43+
```bash
44+
nix run .#verify-darwin-linux-builder
45+
```
46+
47+
Or after setup, use the installed command:
48+
49+
```bash
50+
verify-darwin-linux-builder
51+
```
52+
53+
The verification script checks:
54+
55+
1. Launchd service status (running vs loaded-but-stopped)
56+
2. Nix configuration via `nix config show` (substituters, trusted keys, experimental features)
57+
3. Builder features (`/etc/nix/machines` includes `nixos-test`)
58+
4. Builder responsiveness (test build of `nixpkgs#hello` for aarch64-linux)
59+
60+
Each check reports pass/fail with actionable guidance on failures.
61+
62+
You can also manually test that the linux-builder is working by building a simple package for Linux:
63+
64+
```bash
65+
nix build --system x86_64-linux nixpkgs#hello
66+
nix build --system aarch64-linux nixpkgs#hello
67+
```
68+
69+
If both commands succeed, the linux-builder is ready for NixOS tests.
70+
71+
## Running NixOS tests
72+
73+
NixOS tests are defined in `nix/ext/tests/` and exposed as flake checks.
74+
To run a test on macOS, use the `aarch64-darwin` system attribute:
75+
76+
```bash
77+
nix build .#checks.aarch64-darwin.ext-pgjwt -L
78+
```
79+
80+
The `-L` flag shows logs during the build, which is helpful for seeing test progress and debugging failures.
81+
82+
If the nix build exit immediately with success, it means that the result was fetched from cache and the test passed previously.
83+
To force a re-run of the test, use the `--rebuild` flag:
84+
85+
```bash
86+
nix build .#checks.aarch64-darwin.ext-pgjwt -L --rebuild
87+
```
88+
89+
### Available tests
90+
91+
List all available checks with:
92+
93+
```bash
94+
nix flake show --json 2>/dev/null | jq -r '.checks["aarch64-darwin"] | keys[]' | sort
95+
```
96+
97+
Extension tests follow the naming pattern `ext-<extension_name>`:
98+
99+
```bash
100+
nix build .#checks.aarch64-darwin.ext-pgjwt -L
101+
nix build .#checks.aarch64-darwin.ext-postgis -L
102+
nix build .#checks.aarch64-darwin.ext-vector -L
103+
nix build .#checks.aarch64-darwin.ext-pg_graphql -L
104+
```
105+
106+
## Managing the linux-builder VM
107+
108+
The setup installs two helper commands for controlling the VM:
109+
110+
```bash
111+
stop-linux-builder # Stop the VM (pauses resource usage)
112+
start-linux-builder # Start the VM again
113+
```
114+
115+
As the VM can consume significant resources, you may want to stop it when not running tests using `stop-linux-builder`.
116+
When stopped with `stop-linux-builder`, the service is unloaded to prevent automatic restart.
117+
Use `start-linux-builder` to re-enable and start the service.
118+
119+
### Checking VM status
120+
121+
```bash
122+
sudo launchctl list | grep linux-builder
123+
```
124+
125+
If the VM is running, you'll see a line containing `org.nixos.linux-builder`.
126+
127+
## Troubleshooting
128+
129+
### Tests fail with "builder not available"
130+
131+
Ensure the linux-builder is running:
132+
133+
```bash
134+
start-linux-builder
135+
```
136+
137+
Then verify with a simple build:
138+
139+
```bash
140+
nix build --system aarch64-linux nixpkgs#hello
141+
```
142+
143+
### VM won't start after reboot
144+
145+
If the VM doesn't start automatically, run:
146+
147+
```bash
148+
start-linux-builder
149+
```
150+
151+
The VM is configured as ephemeral, meaning it's recreated fresh on each start.
152+
This ensures a clean environment but requires re-downloading cached build artifacts.
153+
154+
### Slow first build
155+
156+
The first NixOS test run may download significant data.
157+
Subsequent runs benefit from the Nix store cache and the project's binary cache at `nix-postgres-artifacts.s3.amazonaws.com`.
158+
159+
## How it works
160+
161+
The linux-builder is a QEMU virtual machine managed by nix-darwin.
162+
When you run a build targeting Linux (like NixOS tests), Nix automatically delegates the build to this VM.
163+
164+
Key configuration from `nix/hosts/darwin-nixostest/darwin-configuration.nix`:
165+
166+
```nix
167+
nix.linux-builder = {
168+
enable = true;
169+
ephemeral = true;
170+
maxJobs = 4;
171+
supportedFeatures = [
172+
"kvm"
173+
"benchmark"
174+
"big-parallel"
175+
"nixos-test" # Required for NixOS integration tests
176+
];
177+
config = {
178+
virtualisation = {
179+
darwin-builder = {
180+
diskSize = 40 * 1024; # 40GB
181+
memorySize = 8 * 1024; # 8GB
182+
};
183+
cores = 6;
184+
};
185+
};
186+
};
187+
```
188+
189+
The `nixos-test` supported feature is what enables running NixOS VM tests from macOS.

‎nix/hosts.nix‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{ inputs, self, ... }:
2+
{
3+
flake = {
4+
darwinConfigurations = {
5+
darwin-nixostest = inputs.nix-darwin.lib.darwinSystem {
6+
specialArgs = { inherit self; };
7+
modules = [ ./hosts/darwin-nixostest/darwin-configuration.nix ];
8+
};
9+
};
10+
};
11+
}

‎nix/hosts/darwin-nixostest/darwin-configuration.nix‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
{
2+
lib,
3+
pkgs,
4+
self,
5+
...
6+
}:
7+
let
8+
start-linux-builder = pkgs.writeShellApplication {
9+
name = "start-linux-builder";
10+
text = ''
11+
echo "Starting linux-builder..."
12+
13+
if sudo launchctl list | grep -q org.nixos.linux-builder; then
14+
echo "linux-builder is already running"
15+
exit 0
16+
fi
17+
18+
# Use load instead of start to re-enable the service
19+
if sudo launchctl load -w /Library/LaunchDaemons/org.nixos.linux-builder.plist 2>/dev/null; then
20+
echo "linux-builder started successfully"
21+
else
22+
echo "Error: Could not start linux-builder"
23+
echo "Make sure nix-darwin is configured with linux-builder enabled"
24+
exit 1
25+
fi
26+
27+
# Check if it's running
28+
sleep 2
29+
if sudo launchctl list | grep -q org.nixos.linux-builder; then
30+
echo "linux-builder is now running"
31+
else
32+
echo "Warning: linux-builder may not have started properly"
33+
fi
34+
'';
35+
};
36+
stop-linux-builder = pkgs.writeShellApplication {
37+
name = "stop-linux-builder";
38+
text = ''
39+
echo "Stopping linux-builder..."
40+
41+
# Use unload instead of stop because KeepAlive=true will restart it
42+
if sudo launchctl unload -w /Library/LaunchDaemons/org.nixos.linux-builder.plist 2>/dev/null; then
43+
echo "linux-builder stopped successfully"
44+
else
45+
echo "Warning: Could not stop linux-builder (it may not be running)"
46+
fi
47+
48+
# Check if it's still running
49+
sleep 1
50+
if sudo launchctl list | grep -q org.nixos.linux-builder; then
51+
echo "Warning: linux-builder is still running"
52+
STATUS=$(sudo launchctl list | grep org.nixos.linux-builder || true)
53+
echo "Current status: $STATUS"
54+
else
55+
echo "linux-builder is not running"
56+
fi
57+
'';
58+
};
59+
verify-darwin-linux-builder = self.packages.aarch64-darwin.verify-darwin-linux-builder;
60+
in
61+
{
62+
nixpkgs.hostPlatform = "aarch64-darwin";
63+
64+
# Install builder control scripts
65+
environment.systemPackages = [
66+
start-linux-builder
67+
stop-linux-builder
68+
verify-darwin-linux-builder
69+
];
70+
71+
nix.settings = {
72+
experimental-features = [
73+
"nix-command"
74+
"flakes"
75+
];
76+
always-allow-substitutes = true;
77+
max-jobs = "auto";
78+
trusted-users = [ "@admin" ];
79+
extra-substituters = [ "https://nix-postgres-artifacts.s3.amazonaws.com" ];
80+
extra-trusted-substituters = [ "https://nix-postgres-artifacts.s3.amazonaws.com" ];
81+
extra-trusted-public-keys = [
82+
"nix-postgres-artifacts:dGZlQOvKcNEjvT7QEAJbcV6b6uk7VF/hWMjhYleiaLI="
83+
];
84+
};
85+
86+
nix.extraOptions = ''
87+
!include nix.custom.conf
88+
'';
89+
90+
# accept existing nix.custom.conf
91+
system.activationScripts.checks.text = lib.mkForce "";
92+
system.activationScripts.nix-daemon.text = lib.mkForce ''
93+
if ! diff /etc/nix/nix.conf /run/current-system/etc/nix/nix.conf &> /dev/null || ! diff /etc/nix/machines /run/current-system/etc/nix/machines &> /dev/null; then
94+
echo "reloading nix-daemon..." >&2
95+
launchctl kill HUP system/org.nixos.nix-daemon
96+
fi
97+
max_wait=30
98+
waited=0
99+
while ! nix-store --store daemon -q --hash ${pkgs.stdenv.shell} &>/dev/null; do
100+
if [ $waited -ge $max_wait ]; then
101+
echo "ERROR: nix-daemon failed to start after $max_wait seconds" >&2
102+
exit 1
103+
fi
104+
echo "waiting for nix-daemon" >&2
105+
launchctl kickstart system/org.nixos.nix-daemon
106+
sleep 1
107+
waited=$((waited + 1))
108+
done
109+
'';
110+
111+
nix.linux-builder = {
112+
enable = true;
113+
ephemeral = true;
114+
maxJobs = 4;
115+
supportedFeatures = [
116+
"kvm"
117+
"benchmark"
118+
"big-parallel"
119+
"nixos-test"
120+
];
121+
config = {
122+
virtualisation = {
123+
darwin-builder = {
124+
diskSize = 40 * 1024;
125+
memorySize = 8 * 1024;
126+
};
127+
cores = 6;
128+
};
129+
};
130+
};
131+
132+
nix.distributedBuilds = true;
133+
134+
system.stateVersion = 6;
135+
}

‎nix/mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ nav:
3131
- Adding Tests: adding-tests.md
3232
- Migration Tests: migration-tests.md
3333
- Testing PG Upgrade Scripts: testing-pg-upgrade-scripts.md
34+
- NixOS Tests on macOS: nixos-tests-on-macos.md
3435
- References: references.md
3536

3637
validation:

0 commit comments

Comments
 (0)