ExtensionServiceConfig environment and configFiles not injected into extension service container (v1.12.3)

[lukasgomez]

I would like to help fix this issue and contribute to the project if it's possible 😃, but first I would like to know if I'm on the right path and this is the root cause of the issue. Thank you!!

Bug Report

Description

I'm trying to install Netbird extension but ExtensionServiceConfig environment and configFiles are not injected into the extension service container process on Talos v1.12.3, despite the COSI resources being correctly created and the ExtensionServiceConfigStatus confirming the controller processed the config.

Reproduction steps:

1. Create an Image Factory schematic with siderolabs/netbird:

     customization:
       systemExtensions:
         officialExtensions:
           - siderolabs/netbird

2. Install/upgrade a Talos v1.12.3 node using the schematic installer image
3. Append an ExtensionServiceConfig document for netbird to the multi-document machine config:

   # controlplane.yaml content ... main v1alpha1 machine config ...
   ---
   apiVersion: v1alpha1
   kind: ExtensionServiceConfig
   name: netbird
   environment:
       - NB_SETUP_KEY=<setup-key-uuid>

4. Apply and reboot: talosctl apply-config --nodes <ip> --file controlplane.yaml --mode=reboot
5. Observe that ext-netbird crash-loops because NB_SETUP_KEY is never injected into the container process

Evidence the config exists but is not applied:

The ExtensionServiceConfig resource is created correctly:

$ talosctl get extensionserviceconfigs --nodes my_local_ip -o yaml
spec:
    environment:
        - NB_SETUP_KEY=<setup-key-uuid>

The ExtensionServiceConfigStatus also exists (the controller ran):

$ talosctl get extensionserviceconfigstatuses --nodes my_local_ip -o yaml
node: my_local_ip
metadata:
    namespace: runtime
    type: ExtensionServiceConfigStatuses.runtime.talos.dev
    id: netbird
    version: 1
    owner: runtime.ExtensionServiceConfigFilesController
    phase: running
    created: 2026-02-12T18:00:31Z
    updated: 2026-02-12T18:00:31Z
spec:
    specVersion: "1"

But the container's /proc/<PID>/environ does NOT contain NB_SETUP_KEY or any other var from the ExtensionServiceConfig.

I also tested configFiles — the file is not bind-mounted into the container either, which suggests the entire ExtensionServiceConfig is not being read by the Runner() method.

As a further workaround I tried machine.env with NB_SETUP_KEY — same result.

Boot sequence from talosctl dmesg shows the service correctly waits for the "extension service config" dependency (it disappears from the wait list ~3s before the runner is created), yet the vars are missing:

service[ext-netbird](Waiting): Waiting for ... extension service config
service[ext-netbird](Waiting): Waiting for service "cri" to be "up"   ← config dependency satisfied
service[ext-netbird](Preparing): Creating service runner              ← 3s later
service[ext-netbird](Running): Started task ext-netbird (PID 2637)
service[ext-netbird](Waiting): Error running Containerd(ext-netbird) ... invalid UUID length: 0

What I've tried (all unsuccessful):

Attempt	Result
NB_SETUP_KEY in ExtensionServiceConfig environment	Not injected
NB_SETUP_KEYS in ExtensionServiceConfig environment	Not injected
configFiles in ExtensionServiceConfig	Not mounted
machine.env with NB_SETUP_KEY	Not injected
Multiple talosctl apply-config --mode=reboot cycles	Same result every reboot
Upgrading talosctl from v1.11.5 → v1.12.3 to match server	No change
Verified COSI resource via talosctl get extensionserviceconfigs	Resource exists with correct data
Verified status via talosctl get extensionserviceconfigstatuses	Status exists (specVersion: "1")
Verified machine config via talosctl get mc -o yaml	Multi-doc YAML is stored correctly

Source code analysis: The Runner() in extension.go looks correct — it calls safe.StateGetByID for the config and appends spec.Environment to envVars. The resource ID (netbird) matches svc.Spec.Name. Yet neither env vars nor config file mounts reach the container.

Logs

2026-02-12T17:39:38Z service[ext-netbird](Running): Started task ext-netbird (PID 2637) for container ext-netbird
2026-02-12T17:39:38Z INFO profilemanager/config.go:584: using default MTU 1280
2026-02-12T17:39:39Z ERRO client/grpc.go:516: failed to login to Management Service: rpc error: code = PermissionDenied desc = no peer auth method provided, please use a setup key or interactive SSO login
Error: foreground login failed: login check failed: rpc error: code = InvalidArgument desc = invalid setup-key or no sso information provided, err: invalid UUID length: 0

The invalid UUID length: 0 error means netbird tried to parse an empty setup key — confirming NB_SETUP_KEY is not in the process environment.

The service then crash-loops every ~6 seconds with the same error (restart: always).

Extension is installed on the node

talosctl get extensions --nodes my_local_ip


NODE            NAMESPACE   TYPE              ID   VERSION   NAME        VERSION
my_local_ip   runtime     ExtensionStatus   0    1         netbird     0.64.4
my_local_ip   runtime     ExtensionStatus   1    1         schematic   7326f0cbca7a0e700ac1efa3f32e88df9ebe5010e6e842a8ed36fdc99ee98ead

Environment

• Talos version:

  Client:
      Tag:         v1.12.3
      SHA:         6d6471f6
  Server:
      NODE:        my_local_ip
      Tag:         v1.12.3
      SHA:         6d6471f6
      Enabled:     RBAC
  

• Kubernetes version: v1.34.1
• Platform: metal (bare-metal x86_64, single control-plane node)
• Extension: netbird 0.64.4 via Image Factory (schematic 7326f0cbca7a0e700ac1efa3f32e88df9ebe5010e6e842a8ed36fdc99ee98ead)
• Install image: factory.talos.dev/metal-installer/7326f0cbca7a0e700ac1efa3f32e88df9ebe5010e6e842a8ed36fdc99ee98ead:v1.12.3

[Br3nnabee]

Have been having the same issue!

[lukasgomez]

I'll try to debug this by myself in the upcoming days 🐛 @Br3nnabee

[whindes]

Any update on this issue? I can update the MTU by setting NB_MTU=1480 in the ExtensionServiceConfig, but I do not see where the SETUP_KEY or SETUP_KEYS is set. Maybe Netbird has a different auth now?

I can also re-target this config.json from looking at an active PR regarding persisting the file in var/lib instead of var/run
"NB_CONFIG=/var/lib/netbird/config.json",

{
"PrivateKey": "**Redacted ***",
"PreSharedKey": "",
"ManagementURL": {
"Scheme": "https",
"Opaque": "",
"User": null,
"Host": "api.netbird.io:443",
"Path": "",
"RawPath": "",
"OmitHost": false,
"ForceQuery": false,
"RawQuery": "",
"Fragment": "",
"RawFragment": ""
},
"AdminURL": {
"Scheme": "https",
"Opaque": "",
"User": null,
"Host": "app.netbird.io:443",
"Path": "",
"RawPath": "",
"OmitHost": false,
"ForceQuery": false,
"RawQuery": "",
"Fragment": "",
"RawFragment": ""
},
"WgIface": "wt0",
"WgPort": 51820,
"NetworkMonitor": null,
"IFaceBlackList": [
"wt0",
"wt",
"utun",
"tun0",
"zt",
"ZeroTier",
"wg",
"ts",
"Tailscale",
"tailscale",
"docker",
"veth",
"br-",
"lo"
],
"DisableIPv6Discovery": false,
"RosenpassEnabled": false,
"RosenpassPermissive": false,
"ServerSSHAllowed": false,
"EnableSSHRoot": null,
"EnableSSHSFTP": null,
"EnableSSHLocalPortForwarding": null,
"EnableSSHRemotePortForwarding": null,
"DisableSSHAuth": null,
"SSHJWTCacheTTL": null,
"DisableClientRoutes": false,
"DisableServerRoutes": false,
"DisableDNS": false,
"DisableFirewall": false,
"BlockLANAccess": false,
"BlockInbound": false,
"DisableNotifications": true,
"DNSLabels": null,
"SSHKey": "-----BEGIN PRIVATE KEY-----redacted-----END PRIVATE KEY-----\n",
"NATExternalIPs": null,
"CustomDNSAddress": "",
"DisableAutoConnect": false,
"DNSRouteInterval": 60000000000,
"ClientCertPath": "",
"ClientCertKeyPath": "",
"LazyConnectionEnabled": false,
"MTU": 1480
}

[verscheures]

I am having the same issue deploying a new cluster using OpenTofu. The setup works with 1.12.0 (factory.talos.dev/metal-installer/7326f0cbca7a0e700ac1efa3f32e88df9ebe5010e6e842a8ed36fdc99ee98ead:v1.12.0) images but not 1.12.4 (factory.talos.dev/metal-installer/7326f0cbca7a0e700ac1efa3f32e88df9ebe5010e6e842a8ed36fdc99ee98ead:v1.12.4)
As a workaround Installing 1.12.0 and updating to 1.12.4 works is the netbird setup is made persistent.

[apatrida]

big problem, ouch. regression

[frezbo]

We believe this is only related to netbird, other extensions work fine, it's probably netbird changed something upstream, netird is a community contrib extension so it's only looked at at best effort basis

[rawkode]

This has been fixed in netbird:

netbirdio/netbird#5295

[apatrida]

@rawkode great, that was 0.64.6, and the extension is 0.64.5 still.

[whindes]

Do we have to wait for the next version bump for talos 1.12.5 to get the latest netbird version >=0.64.6 ? I am curious in case there was an urgent security patch needed for extensions in general (netbird in this case), how do we update?

[smira]

Do we have to wait for the next version bump for talos 1.12.5 to get the latest netbird version >=0.64.6 ? I am curious in case there was an urgent security patch needed for extensions in general (netbird in this case), how do we update?

netbird is a community extension, so if you depend on it, you should get some time to test newer versions, submit patches, etc.

the bump will not happen on its own (and the regression will not be noticed).