Skip to content

Sync with upstream 2.33.4#435

Merged
cole-h merged 28 commits intomainfrom
sync-2.33.4
Apr 23, 2026
Merged

Sync with upstream 2.33.4#435
cole-h merged 28 commits intomainfrom
sync-2.33.4

Conversation

@edolstra
Copy link
Copy Markdown
Collaborator

@edolstra edolstra commented Apr 23, 2026

Motivation

Context

Summary by CodeRabbit

  • Bug Fixes

    • Restored STS WebIdentity and ECS container credential providers in the S3 credential chain; IMDS is skipped when ECS container credentials are present.
  • Documentation

    • Added a manual describing the S3 credential-chain behavior and provider evaluation order.
  • Chores / Packaging

    • Made archive creation deterministic, adjusted debug-info handling to emit separate debug symbols, and ensured debug uploads include binaries when separate debug info is unavailable.
    • Bumped project version.
  • Refactor

    • Improved logging writes, terminal initialization, and a remote filesystem root lookup optimization.

lovesegfault and others added 26 commits February 13, 2026 07:47
…-2.33-maintenance

[Backport 2.33-maintenance] upload-release: disable containerd image store to preserve gzip layer compression
Generalize writeToStderr into writeFullLogging, with similar behavior
but taking an arbitrary fd, and reimplement writeToStderr with it as a
convenient wrapper.

(cherry picked from commit 473d54e)
Logging should not check for interrupts. Use the new writeFullLogging
function to write JSON output to get similar behavior to SimpleLogger.

This avoids the logger itself throwing Interrupted exceptions in
handleExceptions.

(cherry picked from commit 1210100)
…tenance

[Backport 2.33-maintenance] Don't throw Interrupted from JSONLogger::write
…tenance

[Backport 2.33-maintenance] Fix the nix-community/patsh/0.2.1 flake regression test (again)
RemoteFSAccessor::fetch() crashes when called with the root path.

  $ nix build nixpkgs#hello --store ssh-ng://host
  error: path '/nix/store/' is not in the Nix store

Add path.isRoot() guard like LocalStoreAccessor::maybeLstat

Fixes: eb643d0 ("`Store::getFSAccessor`: Do not include the store dir")
(cherry picked from commit b523564)
Resolves NixOS#15420

Signed-off-by: Lisanna Dettwyler <lisanna.dettwyler@gmail.com>
(cherry picked from commit 342faaa)
…tenance

[Backport 2.33-maintenance] libstore: handle root path in RemoteFSAccessor::maybeLstat
…tenance

[Backport 2.33-maintenance] Fix compatibility with lowdown 3
…hain

Nix 2.33 replaced aws-sdk-cpp with aws-crt-cpp for S3 credentials. The
custom credential chain (Environment → SSO → Profile → IMDS) omitted the
STS WebIdentity provider that was part of the old
DefaultAWSCredentialsProviderChain, breaking EKS IRSA, GitHub Actions
OIDC, and any other sts:AssumeRoleWithWebIdentity flow.

The observed failure mode on EKS pods using IRSA was a misleading
'Valid credentials could not be sourced by the IMDS provider' error —
IMDS is merely the last provider tried after WebIdentity was skipped.

aws-c-auth (already linked) exposes aws_credentials_provider_new_sts_web_identity,
which reads AWS_WEB_IDENTITY_TOKEN_FILE, AWS_ROLE_ARN,
AWS_ROLE_SESSION_NAME, and AWS_REGION from the environment (falling back
to the profile config) and returns nullptr at construction time when
they aren't set. This is wired into the chain after Profile and before
IMDS, matching the pre-2.33 ordering.

(cherry picked from commit 9ae4b50)
Same regression class as the preceding commit. The pre-2.33 aws-sdk-cpp
credential chain included the ECS container metadata provider, which
reads AWS_CONTAINER_CREDENTIALS_RELATIVE_URI (or _FULL_URI) and is used
by ECS tasks and EKS Pod Identity.

aws_credentials_provider_new_ecs_from_environment reads the relevant env
vars and returns nullptr at construction time when none are set, so this
slots into the chain with zero runtime cost when unused. Wired after STS
WebIdentity and before IMDS, matching the pre-2.33 ordering.

The full credential chain is now:
Environment → SSO → Profile → STS WebIdentity → ECS → IMDS

(cherry picked from commit eec6d92)
…tenance

[Backport 2.33-maintenance] fix(libstore/aws-creds): add STS WebIdentity and ECR metadata
This fixes a a destructor ordering issue where during shutdown, ~ProgressBar() calls
getWindowSize() after the windowSize object (and thus its mutex) has been destroyed.
This happens when a draw() call happens to be active during destruction.

Fixes #361, NixOS#14300, NixOS#14361.

(cherry picked from commit cf44b7c)
…tenance

[Backport 2.33-maintenance] Don't destroy windowSize mutex
While debugging NixOS#15564 I noticed that
our release tarball don't seem to be reproducible. This doesn't explain
why hydra calculated the different hash of the tarball, which should only
be built once (???).

Regardless, I noticed that checking the build reproducibility against hydra
built tarballs fails with e.g.

nix build .\#hydraJobs.binaryTarball.x86_64-linux -L --keep-failed --rebuild

Looking with diffoscope, it was the entry ordering issue. Let's nip this in the bud.

(cherry picked from commit 4c41c5f)
…tenance

[Backport 2.33-maintenance] packaging: Make binaryTarball more reproducible
This error no longer seems to occur on Linux 6.19+ anymore. Skip in that
case to fix build.

(cherry picked from commit 2e6a03e)
…tenance

[Backport 2.33-maintenance] tests/functional/stale-file-handle: Skip if the error doesn't happen
…e output in a temporary directory in the store

Puts the temporary FOD output copies in a temporary directory inside the
store instead of the (for Linux sandboxed builds) chroot.
This prevents file overwrite due to symlink following that std::filesystem::copy_file does.

Also applies the same output copying approach for impure derivations that don't
have network sandboxing and thus are subject to FD smuggling.

Fixes GHSA-g3g9-5vj6-r3gj.

(cherry picked from commit a760af8)
…ew enough kernels

This partially fixes the issue with cooperating processes being able
to communicate via abstract sockets. The fix is partial, because processes
outside the landlock domain of the sandboxed process can still connect to
a socket created by the FOD. There's no equivalent way of restricting inbound
connections. This closes the gap when there's no cooperating process on the host
(i.e. 2 separate FODs).

>= 6.12 kernel is widespread enough (NixOS 25.11 ships it by
default) that we have no reason not to apply this hardening, even though
it's incomplete.

ca-fd-leak test exercises this exact code path and now the smuggling
process fails with (on new enough kernels that have landlock support enabled):

vm-test-run-ca-fd-leak> machine # sandbox setup: applied landlock sandboxing
vm-test-run-ca-fd-leak> machine # building '/nix/store/s7brgi6pdr5f3n8yqlgmdlz8blb89njc-smuggled.drv'...
vm-test-run-ca-fd-leak> machine # building derivation '/nix/store/s7brgi6pdr5f3n8yqlgmdlz8blb89njc-smuggled.drv': woken up
vm-test-run-ca-fd-leak> machine # connect: Operation not permitted
vm-test-run-ca-fd-leak> machine # sendmsg: Socket not connected

(cherry picked from commit 44017ca)
Tagging release 2.33.4
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 23, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a1843f6b-e36b-4602-ab4a-1fb701690282

📥 Commits

Reviewing files that changed from the base of the PR and between 7d8efbc and dd0920f.

📒 Files selected for processing (1)
  • maintainers/upload-debug-info-to-sentry.py

📝 Walkthrough

Walkthrough

Updates version to 2.33.4; documents S3 credential-chain changes; makes tarball creation deterministic; adds lowdown v3 detection and fixes markdown flags; extends AWS credential provider chain to include STS WebIdentity and ECS with conditional IMDS fallback; refactors logging writes; minor terminal and remote FS accessor tweaks; build change for sentry-native debug info.

Changes

Cohort / File(s) Summary
Version & Docs
\.version, doc/manual/rl-next/s3-credential-chain-web-identity.md
Bump to 2.33.4; add docs describing S3 credential-chain change (Environment → SSO → Profile → STS WebIdentity → (ECS
AWS Credentials Chain
src/libstore/aws-creds.cc
Add STS WebIdentity and ECS providers, new addProviderToChain helper, conditional IMDS append, expanded TLS-failure messaging, and two helpers for building STS/ECS providers.
Build & Packaging
packaging/binary-tarball.nix, packaging/sentry-native.nix, src/libcmd/meson.build
Make tar creation deterministic (--sort=name + explicit xz); enable separateDebugInfo = true in sentry native; add HAVE_LOWDOWN_3 config flag.
Markdown / Lowdown
src/libcmd/markdown.cc
Use LOWDOWN_NORELLINK when HAVE_LOWDOWN_3 is present; preserve fallbacks for older Lowdown versions.
Logging
src/libutil/logging.cc
Introduce writeFullLogging(fd, s) to centralize full writes with error suppression; JSONLogger now writes trailing newline and uses the new helper.
Remote FS & Terminal
src/libstore/remote-fs-accessor.cc, src/libutil/terminal.cc
Short-circuit maybeLstat for root paths; change windowSize initialization to double-braced direct-list-initialization.
Sentry upload script
maintainers/upload-debug-info-to-sentry.py
Ensure binaries are uploaded when build-id or separate debuginfo lookup fails by adding the binary to upload list and logging the behavior.
Small Version File
.version
Version string incremented from 2.33.32.33.4.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant AwsCreds as AwsCredsLogic
    participant Env as Environment
    participant SSO
    participant Profile
    participant WebID as STS_WebIdentity
    participant ECS
    participant IMDS

    Client->>AwsCreds: request credentials
    AwsCreds->>Env: check Environment
    alt found
        Env-->>AwsCreds: creds
    else
        AwsCreds->>SSO: check SSO
        alt found
            SSO-->>AwsCreds: creds
        else
            AwsCreds->>Profile: check Profile
            alt found
                Profile-->>AwsCreds: creds
            else
                AwsCreds->>WebID: attempt STS WebIdentity (if TLS ok)
                alt found
                    WebID-->>AwsCreds: creds
                else
                    AwsCreds->>ECS: attempt ECS container creds (if TLS ok & env set)
                    alt ECS added
                        ECS-->>AwsCreds: creds
                    else
                        AwsCreds->>IMDS: fallback to IMDS (only if ECS not added)
                        IMDS-->>AwsCreds: creds or error
                    end
                end
            end
        end
    end
    AwsCreds-->>Client: credentials or error
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested reviewers

  • cole-h

Poem

🐇 I hopped a version, tiny paws and glee,
WebIdentity joined the chain, ECS followed, see?
Tars sorted neatly, logs now write in line,
Lowdown waves hello — markdown looks fine! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 21.43% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Sync with upstream 2.33.4' directly matches the PR's main objective and is confirmed by the version bump in .version file and commit messages.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch sync-2.33.4

Comment @coderabbitai help to get the list of available commands and usage tips.

cole-h
cole-h previously approved these changes Apr 23, 2026
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
doc/manual/rl-next/s3-credential-chain-web-identity.md (1)

17-19: Consider adding TLS-context failure as an additional root cause.

This section currently points to omitted providers only. In code, TLS context creation failure also disables SSO/STS WebIdentity/ECS, which can lead to the same fallback symptom. A short note here would improve troubleshooting.

✍️ Suggested doc tweak
-The typical symptom was a misleading IMDS error
+The typical symptom was a misleading IMDS error
 (`Valid credentials could not be sourced by the IMDS provider`), because
 IMDS is the last provider tried after the correct one was skipped.
+This can also happen if TLS context initialization fails, which makes
+SSO, STS WebIdentity, and ECS providers unavailable.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@doc/manual/rl-next/s3-credential-chain-web-identity.md` around lines 17 - 19,
Add a short note to the s3-credential-chain-web-identity.md section explaining
that TLS context creation failure can disable SSO/STS WebIdentity/ECS credential
providers and cause the same fallback symptom ("Valid credentials could not be
sourced by the IMDS provider"); update the paragraph that discusses omitted
providers to mention "TLS context creation failure" (and optionally "TLS/SSL
handshake or certificate load errors") as an additional root cause and include
guidance to check TLS initialization logs alongside SSO/STS/WebIdentity/ECS
checks.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@doc/manual/rl-next/s3-credential-chain-web-identity.md`:
- Around line 17-19: Add a short note to the s3-credential-chain-web-identity.md
section explaining that TLS context creation failure can disable SSO/STS
WebIdentity/ECS credential providers and cause the same fallback symptom ("Valid
credentials could not be sourced by the IMDS provider"); update the paragraph
that discusses omitted providers to mention "TLS context creation failure" (and
optionally "TLS/SSL handshake or certificate load errors") as an additional root
cause and include guidance to check TLS initialization logs alongside
SSO/STS/WebIdentity/ECS checks.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 7114e7a4-b79f-468a-8ec2-ab4ed30110ed

📥 Commits

Reviewing files that changed from the base of the PR and between 7ab838d and 6a4351a.

📒 Files selected for processing (9)
  • .version
  • doc/manual/rl-next/s3-credential-chain-web-identity.md
  • packaging/binary-tarball.nix
  • src/libcmd/markdown.cc
  • src/libcmd/meson.build
  • src/libstore/aws-creds.cc
  • src/libstore/remote-fs-accessor.cc
  • src/libutil/logging.cc
  • src/libutil/terminal.cc

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 23, 2026

@github-actions github-actions Bot temporarily deployed to pull request April 23, 2026 15:29 Inactive
@edolstra edolstra enabled auto-merge April 23, 2026 15:33
@github-actions github-actions Bot temporarily deployed to pull request April 23, 2026 16:04 Inactive
@edolstra edolstra added this pull request to the merge queue Apr 23, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Apr 23, 2026
@cole-h cole-h added this pull request to the merge queue Apr 23, 2026
Merged via the queue into main with commit 664c11d Apr 23, 2026
31 checks passed
@cole-h cole-h deleted the sync-2.33.4 branch April 23, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants