From 75cc51ed763d219b66eab340fc4a6fe53beb1de5 Mon Sep 17 00:00:00 2001 From: Alexander Tarasov Date: Fri, 27 Mar 2026 10:33:35 +0100 Subject: [PATCH 1/5] rfc(decision): Distroless base images --- README.md | 1 + text/0157-distroless-base-images.md | 35 +++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) create mode 100644 text/0157-distroless-base-images.md diff --git a/README.md b/README.md index de015231..9d4f4b1b 100644 --- a/README.md +++ b/README.md @@ -71,3 +71,4 @@ This repository contains RFCs and DACIs. Lost? - [0146-automatic-in-app-rules](text/0146-automatic-in-app-rules.md): For Java projects, when a code mapping is automatically created, also create an in-app stack trace rules to categorize the frames as in-app - [0148-logs-for-crashes](text/0148-logs-for-crashes.md): Logs for Crashes - [0149-merge-bundler-plugins](text/0149-merge-bundler-plugins.md): Merge sentry-javascript-bundler-plugins into sentry-javascript +- [0157-distroless-base-images](text/0157-distroless-base-images.md): Distroless base images diff --git a/text/0157-distroless-base-images.md b/text/0157-distroless-base-images.md new file mode 100644 index 00000000..a7b8b822 --- /dev/null +++ b/text/0157-distroless-base-images.md @@ -0,0 +1,35 @@ +- Start Date: 2026-03-27 +- RFC Type: decision +- RFC PR: https://github.com/getsentry/rfcs/pull/157 +- RFC Status: draft + +# Summary + +One paragraph explanation of the feature or document purpose. + +# Motivation + +Why are we doing this? What use cases does it support? What is the expected outcome? + +# Background + +The reason this decision or document is required. This section might not always exist. + +# Supporting Data + +[Metrics to help support your decision (if applicable).] + +# Options Considered + +If an RFC does not know yet what the options are, it can propose multiple options. The +preferred model is to propose one option and to provide alternatives. + +# Drawbacks + +Why should we not do this? What are the drawbacks of this RFC or a particular option if +multiple options are presented. + +# Unresolved questions + +- What parts of the design do you expect to resolve through this RFC? +- What issues are out of scope for this RFC but are known? From ee4f3487f52c9e32eaa4b1b2a1c86f77d4a87508 Mon Sep 17 00:00:00 2001 From: Alexander Tarasov Date: Fri, 27 Mar 2026 13:19:49 +0100 Subject: [PATCH 2/5] actual RFC content --- text/0157-distroless-base-images.md | 147 ++++++++++++++++++++++++++-- 1 file changed, 137 insertions(+), 10 deletions(-) diff --git a/text/0157-distroless-base-images.md b/text/0157-distroless-base-images.md index a7b8b822..66390fd8 100644 --- a/text/0157-distroless-base-images.md +++ b/text/0157-distroless-base-images.md @@ -5,31 +5,158 @@ # Summary -One paragraph explanation of the feature or document purpose. +This RFC proposes adopting distroless (minimal) base Docker images across Sentry's containerized applications, replacing the standard Debian-based images currently in use. A distroless image contains only the application and its runtime dependencies — no shell, no package manager, no standard Linux utilities. This significantly reduces image size and the number of OS-level packages that can be exploited or need to be tracked for vulnerabilities. + +We had an internal TSC talk about this, see [Notion](https://www.notion.so/sentry/3208b10e4b5d8086a18ff52013025b3a). # Motivation -Why are we doing this? What use cases does it support? What is the expected outcome? +Standard base images (e.g. `python:3.x-slim`, `node:24-slim`) ship with hundreds of OS packages that the application itself never uses. These packages: + +- Expand the attack surface — each package is a potential vulnerability vector. +- Add ongoing maintenance burden — every new CVE in an unused package still requires triage and remediation. +- Increase image size — more packages mean more storage, more bandwidth to pull images, and slower deployments. + +Switching to distroless addresses all three problems at once. The security benefit is structural: there are simply fewer things that can go wrong, regardless of whether any given CVE is actually exploitable in our environment. # Background -The reason this decision or document is required. This section might not always exist. +A **distroless** image is a Docker image stripped down to the bare minimum needed to run a specific language runtime. The concept was introduced by Google and has since been adopted broadly. Key properties: + +- No shell (`/bin/sh`, `bash`, etc.) +- No package manager (`apt`, `pip` system-wide, etc.) +- No standard Linux utilities (`curl`, `wget`, `ls`, etc.) +- Only the application binary/runtime and its direct OS-level dependencies + +This makes the image unsuitable as a general-purpose Linux environment, which is exactly the point: an attacker who finds a way in cannot easily pivot or execute arbitrary commands. + +Note that Alpine Linux, while smaller than Debian, is not distroless — it still ships a shell, a package manager (`apk`), and general-purpose utilities. It reduces image size but does not meaningfully reduce the attack surface in the same way. + +There are several providers of distroless-style images. The main options evaluated are covered in [Options Considered](#options-considered). # Supporting Data -[Metrics to help support your decision (if applicable).] +## Chartcuterie (Node.js) + +| Base image | Uncompressed size | OS packages | OS vulns (C/H/M/L) | +|---|---|---|---| +| `node:24.14.0-slim` | 1.47 GB | 288 | 1455 (3/148/659/644) | +| `dhi.io/node:24-debian13` | 287 MB | 12 | 10 (0/1/2/7) | + +**5× smaller image, 99% fewer OS vulnerabilities.** + +See: https://github.com/getsentry/chartcuterie/pull/216 + +## Snuba (Python) + +| Base image | Uncompressed size | OS vulns | +|---|---|---| +| Debian slim | 2.96 GB | 949 | +| Google Distroless | 517 MB | 114 | +| Docker Hardened Images | 528 MB | 20 | + +See: https://github.com/getsentry/snuba/pull/7753, https://github.com/getsentry/snuba/pull/7821 + +## Completed migrations + +Several services have already been successfully migrated: + +**Rust:** +- Conduit: https://github.com/getsentry/conduit/pull/12, https://github.com/getsentry/conduit/pull/17 +- Objectstore: https://github.com/getsentry/objectstore/pull/65 +- Relay: https://github.com/getsentry/relay/pull/4940 +- Symbolicator: https://github.com/getsentry/symbolicator/pull/1791 +- Tempest: https://github.com/getsentry/tempest/pull/166 + +**Python:** +- Reload: https://github.com/getsentry/reload/pull/340 +- other smaller internal services + +**Node.js:** +- Chartcuterie: https://github.com/getsentry/chartcuterie/pull/216 # Options Considered -If an RFC does not know yet what the options are, it can propose multiple options. The -preferred model is to propose one option and to provide alternatives. +## Option 1: Docker Hardened Images (dhi.io) — recommended + +Docker Hardened Images (DHI) are minimal, hardened base images published by Docker Inc. + +**Pros:** +- Apache 2.0 license — no restrictions for self-hosted use +- Full tag support (pinnable versions, not just `:latest`) +- Fast updates: e.g. `dhi.io/python:3.13-debian13` tracks `3.13.x` with regular patch releases +- Available for Python, Node.js, and other runtimes we use + +**Cons:** +- Requires Docker login to pull from `dhi.io` directly (mitigated by our mirroring approach) +- Some build systems (e.g. CloudBuild) have issues with hard links in the images — affected workloads should be migrated to GitHub Actions + +## Option 2: Google Distroless + +Google's original distroless images at `gcr.io/distroless/`. + +**Pros:** +- Apache 2.0 license +- Well-established, widely used + +**Cons:** +- Slower upstream updates: e.g. `distroless/python3-debian13` lagged behind the latest Python patch release +- Limited tag granularity for some runtimes + +This is still a valid option for Rust (cc-debian12), and was used in the first wave of migrations (Relay, Objectstore, Conduit, Symbolicator, Tempest). + +## Option 3: Chainguard Images + +**Pros:** +- Very minimal and hardened + +**Cons:** +- Free tier only supports `:latest` tag — pinning specific versions requires a paid subscription +- Licensing restrictions affect self-hosted deployments + +Not suitable given our self-hosted distribution requirements. + +## Option 4: Wiz OS + +**Pros:** +- Hardened base images + +**Cons:** +- Vendor lock-in +- Licensing issues for self-hosted + +Not suitable for the same reasons as Chainguard. + +## Option 5: Do nothing + +Keep using standard slim images, continue triaging CVEs in packages we do not use. + +This is not a good use of engineering time and leaves unnecessary attack surface in place. # Drawbacks -Why should we not do this? What are the drawbacks of this RFC or a particular option if -multiple options are presented. +## Debugging is harder + +Distroless containers have no shell. You cannot `exec` into a running container and run arbitrary commands. Debugging requires: + +- Attaching an ephemeral debug container with a shell to the running pod (e.g. [`sentry-kube debug`](https://github.com/getsentry/sentry-infra-tools/blob/main/sentry_kube/cli/debug.py)) +- Using application-level tooling (e.g. interactive shells provided by the framework) rather than OS-level tools, e.g. `getsentry shell` +- Investing in proper observability (logs, metrics, tracing) instead of ad-hoc inspection + +## Runtime dependencies can be surprising + +Applications sometimes rely on OS-level packages in ways that are not obvious from the code — fonts, shared libraries, locale data, CA certificates, etc. These need to be identified before or during migration. + +**Example:** A migration surfaced a missing fonts dependency that caused a runtime failure. The fix was straightforward once identified, but it required a production incident to discover it. Smoke tests on the built image, or comprehensive self-hosted test runs, would catch these earlier. + +## Build system compatibility + +Some CI/CD build systems have compatibility issues with certain distroless image formats (e.g. hard links). Services affected should migrate to GitHub Actions before or alongside the distroless migration. # Unresolved questions -- What parts of the design do you expect to resolve through this RFC? -- What issues are out of scope for this RFC but are known? +- **Long-term commitment to DHI:** Despite Docker Inc having a history of unexpected licensing and policy changes (Hub rate limiting, Desktop licensing, etc.), DHI was recently made public under Apache 2.0, and a rollback of that decision seems unlikely. If needed, Google Distroless is a practical drop-in fallback — it lags a few patch versions behind but is otherwise compatible. Other solutions may also emerge over time. +- **Smoke tests / image validation in CI:** Should we require a standard CI step that runs basic smoke tests against a newly built distroless image before publishing? This would catch missing runtime dependencies (like the fonts incident) before they reach production. +- **Snuba and getsentry:** These are the largest remaining Python services. The Snuba PoC (https://github.com/getsentry/snuba/pull/7753, https://github.com/getsentry/snuba/pull/7821) showed it is feasible. What is the sequencing and who owns driving this to completion? +- **Services with non-trivial runtime deps:** Some services (e.g. uptime-checker with OpenSSL for certificate validation, or services using external libraries) may need extra work. Are there any blockers that make distroless infeasible for them? +- **Standardizing the dev variant:** The `-dev` variant of DHI images (which includes a shell and debugging tools) is useful for development builds and troubleshooting. Should we define a standard pattern for multi-stage Dockerfiles that use `-dev` at build time and the minimal image at runtime? From 8ef58074d50e1ab7df193cd31feaae3857840825 Mon Sep 17 00:00:00 2001 From: Alexander Tarasov Date: Fri, 27 Mar 2026 13:25:30 +0100 Subject: [PATCH 3/5] upd --- text/0157-distroless-base-images.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/text/0157-distroless-base-images.md b/text/0157-distroless-base-images.md index 66390fd8..95b66e9d 100644 --- a/text/0157-distroless-base-images.md +++ b/text/0157-distroless-base-images.md @@ -157,6 +157,7 @@ Some CI/CD build systems have compatibility issues with certain distroless image - **Long-term commitment to DHI:** Despite Docker Inc having a history of unexpected licensing and policy changes (Hub rate limiting, Desktop licensing, etc.), DHI was recently made public under Apache 2.0, and a rollback of that decision seems unlikely. If needed, Google Distroless is a practical drop-in fallback — it lags a few patch versions behind but is otherwise compatible. Other solutions may also emerge over time. - **Smoke tests / image validation in CI:** Should we require a standard CI step that runs basic smoke tests against a newly built distroless image before publishing? This would catch missing runtime dependencies (like the fonts incident) before they reach production. -- **Snuba and getsentry:** These are the largest remaining Python services. The Snuba PoC (https://github.com/getsentry/snuba/pull/7753, https://github.com/getsentry/snuba/pull/7821) showed it is feasible. What is the sequencing and who owns driving this to completion? +- **Snuba and getsentry:** These are the largest remaining Python services. The Snuba PoC (https://github.com/getsentry/snuba/pull/7753, https://github.com/getsentry/snuba/pull/7821, https://github.com/getsentry/snuba/pull/7829, https://github.com/getsentry/ops/pull/19824) showed it is feasible. What is the sequencing and who owns driving this to completion? +- **Local development compatibility:** Are there any blockers that might disrupt local development workflows when switching to distroless? So far this appears to be a non-issue — for example, Snuba distroless containers work fine in `sentry devservices` (https://github.com/getsentry/snuba/pull/7829). - **Services with non-trivial runtime deps:** Some services (e.g. uptime-checker with OpenSSL for certificate validation, or services using external libraries) may need extra work. Are there any blockers that make distroless infeasible for them? - **Standardizing the dev variant:** The `-dev` variant of DHI images (which includes a shell and debugging tools) is useful for development builds and troubleshooting. Should we define a standard pattern for multi-stage Dockerfiles that use `-dev` at build time and the minimal image at runtime? From 16a332802d99c7e54e1d0d0125e73c6f44bba004 Mon Sep 17 00:00:00 2001 From: Alexander Tarasov Date: Fri, 27 Mar 2026 13:58:09 +0100 Subject: [PATCH 4/5] upd more concerns --- text/0157-distroless-base-images.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/text/0157-distroless-base-images.md b/text/0157-distroless-base-images.md index 95b66e9d..1838647b 100644 --- a/text/0157-distroless-base-images.md +++ b/text/0157-distroless-base-images.md @@ -5,7 +5,7 @@ # Summary -This RFC proposes adopting distroless (minimal) base Docker images across Sentry's containerized applications, replacing the standard Debian-based images currently in use. A distroless image contains only the application and its runtime dependencies — no shell, no package manager, no standard Linux utilities. This significantly reduces image size and the number of OS-level packages that can be exploited or need to be tracked for vulnerabilities. +This RFC proposes adopting distroless (minimal) base Docker images across Sentry's containerized applications, replacing the standard Debian-based images currently in use. A distroless image contains only the application and its runtime dependencies — no shell, no package manager, no standard Linux utilities. This significantly reduces image size and the number of OS-level packages that can be exploited or need to be tracked for vulnerabilities. The goals of this initiative are: (1) migrate the most important services to distroless, and (2) establish distroless as the default baseline for all new containerized services going forward. We had an internal TSC talk about this, see [Notion](https://www.notion.so/sentry/3208b10e4b5d8086a18ff52013025b3a). @@ -88,7 +88,7 @@ Docker Hardened Images (DHI) are minimal, hardened base images published by Dock - Available for Python, Node.js, and other runtimes we use **Cons:** -- Requires Docker login to pull from `dhi.io` directly (mitigated by our mirroring approach) +- Requires Docker login to pull from `dhi.io` directly (mitigated by our [mirroring](https://github.com/getsentry/dhi) approach) - Some build systems (e.g. CloudBuild) have issues with hard links in the images — affected workloads should be migrated to GitHub Actions ## Option 2: Google Distroless @@ -160,4 +160,6 @@ Some CI/CD build systems have compatibility issues with certain distroless image - **Snuba and getsentry:** These are the largest remaining Python services. The Snuba PoC (https://github.com/getsentry/snuba/pull/7753, https://github.com/getsentry/snuba/pull/7821, https://github.com/getsentry/snuba/pull/7829, https://github.com/getsentry/ops/pull/19824) showed it is feasible. What is the sequencing and who owns driving this to completion? - **Local development compatibility:** Are there any blockers that might disrupt local development workflows when switching to distroless? So far this appears to be a non-issue — for example, Snuba distroless containers work fine in `sentry devservices` (https://github.com/getsentry/snuba/pull/7829). - **Services with non-trivial runtime deps:** Some services (e.g. uptime-checker with OpenSSL for certificate validation, or services using external libraries) may need extra work. Are there any blockers that make distroless infeasible for them? +- **Public mirrors for anonymous access:** Pulling directly from `dhi.io` requires a Docker login, which complicates CI pipelines and local image builds for contributors. Should we commit to maintaining public mirrors at `ghcr.io/getsentry/dhi` to allow unauthenticated pulls? See https://github.com/getsentry/dhi. +- **Image freshness:** Migrating to distroless is not a one-time fix — base images still need to be updated as new runtime patch versions are released. Corresponding product teams are responsible for bumping base image versions in line with their application's requirements and compatibility constraints. - **Standardizing the dev variant:** The `-dev` variant of DHI images (which includes a shell and debugging tools) is useful for development builds and troubleshooting. Should we define a standard pattern for multi-stage Dockerfiles that use `-dev` at build time and the minimal image at runtime? From 0383a31f0ff4cab887560a94883c5501112c3ca1 Mon Sep 17 00:00:00 2001 From: Alexander Tarasov Date: Fri, 27 Mar 2026 16:05:06 +0100 Subject: [PATCH 5/5] some questions are pretty much resolved --- text/0157-distroless-base-images.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/text/0157-distroless-base-images.md b/text/0157-distroless-base-images.md index 1838647b..30d1efc8 100644 --- a/text/0157-distroless-base-images.md +++ b/text/0157-distroless-base-images.md @@ -153,13 +153,16 @@ Applications sometimes rely on OS-level packages in ways that are not obvious fr Some CI/CD build systems have compatibility issues with certain distroless image formats (e.g. hard links). Services affected should migrate to GitHub Actions before or alongside the distroless migration. +# Resolved questions + +- **Long-term commitment to DHI:** Despite Docker Inc having a history of unexpected licensing and policy changes (Hub rate limiting, Desktop licensing, etc.), DHI was recently made public under Apache 2.0, and a rollback of that decision seems unlikely. If needed, Google Distroless is a practical drop-in fallback — it lags a few patch versions behind but is otherwise compatible. Other solutions may also emerge over time. We can go with DHI images as a default. +- **Image freshness:** Migrating to distroless is not a one-time fix — base images still need to be updated as new runtime patch versions are released. Corresponding product teams are responsible for bumping base image versions in line with their application's requirements and compatibility constraints. +- **Smoke tests / image validation in CI:** Ideally, an extra CI step should run basic smoke tests against a newly built distroless image before publishing. This would catch missing runtime dependencies (like the fonts incident) before they reach production. +- **Standardizing the dev variant:** The `-dev` variant of DHI images (which includes a shell and debugging tools) is useful for development builds and troubleshooting. For multi-stage Dockerfiles, we should use `-dev` at build time and the minimal image at runtime. + # Unresolved questions -- **Long-term commitment to DHI:** Despite Docker Inc having a history of unexpected licensing and policy changes (Hub rate limiting, Desktop licensing, etc.), DHI was recently made public under Apache 2.0, and a rollback of that decision seems unlikely. If needed, Google Distroless is a practical drop-in fallback — it lags a few patch versions behind but is otherwise compatible. Other solutions may also emerge over time. -- **Smoke tests / image validation in CI:** Should we require a standard CI step that runs basic smoke tests against a newly built distroless image before publishing? This would catch missing runtime dependencies (like the fonts incident) before they reach production. - **Snuba and getsentry:** These are the largest remaining Python services. The Snuba PoC (https://github.com/getsentry/snuba/pull/7753, https://github.com/getsentry/snuba/pull/7821, https://github.com/getsentry/snuba/pull/7829, https://github.com/getsentry/ops/pull/19824) showed it is feasible. What is the sequencing and who owns driving this to completion? - **Local development compatibility:** Are there any blockers that might disrupt local development workflows when switching to distroless? So far this appears to be a non-issue — for example, Snuba distroless containers work fine in `sentry devservices` (https://github.com/getsentry/snuba/pull/7829). - **Services with non-trivial runtime deps:** Some services (e.g. uptime-checker with OpenSSL for certificate validation, or services using external libraries) may need extra work. Are there any blockers that make distroless infeasible for them? -- **Public mirrors for anonymous access:** Pulling directly from `dhi.io` requires a Docker login, which complicates CI pipelines and local image builds for contributors. Should we commit to maintaining public mirrors at `ghcr.io/getsentry/dhi` to allow unauthenticated pulls? See https://github.com/getsentry/dhi. -- **Image freshness:** Migrating to distroless is not a one-time fix — base images still need to be updated as new runtime patch versions are released. Corresponding product teams are responsible for bumping base image versions in line with their application's requirements and compatibility constraints. -- **Standardizing the dev variant:** The `-dev` variant of DHI images (which includes a shell and debugging tools) is useful for development builds and troubleshooting. Should we define a standard pattern for multi-stage Dockerfiles that use `-dev` at build time and the minimal image at runtime? +- **Public mirrors for anonymous access:** Pulling directly from `dhi.io` requires a Docker login, which complicates CI pipelines and local image builds for contributors. Should we commit to maintaining public mirrors at `ghcr.io/getsentry/dhi` to allow unauthenticated pulls? See current PoC: https://github.com/getsentry/dhi.