feat!: Container infrastructure refactor (ADR-T-009)#857
Open
peer-cat wants to merge 15 commits intotorrust:developfrom
Open
feat!: Container infrastructure refactor (ADR-T-009)#857peer-cat wants to merge 15 commits intotorrust:developfrom
peer-cat wants to merge 15 commits intotorrust:developfrom
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #857 +/- ##
===========================================
+ Coverage 67.81% 68.69% +0.88%
===========================================
Files 153 161 +8
Lines 12222 12418 +196
Branches 12222 12418 +196
===========================================
+ Hits 8288 8531 +243
+ Misses 3682 3635 -47
Partials 252 252 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
peer-cat
force-pushed
the
feat/rework_container_system
branch
from
37d86a9 to
8b1717a
Compare
There was a problem hiding this comment.
Pull request overview
Implements ADR-T-009’s container-infrastructure refactor by extracting config parsing and container helper tooling into dedicated workspace crates, reshaping Compose into a production baseline + dev override, and updating tests/docs to reflect schema-mandatory credentials and the new entry-script/probe workflow. Also includes an auth correctness tweak in the permission extractor.
Changes:
- Extracted configuration parsing into
torrust-index-config(+ addedconfig-probe,health-check,auth-keypair,cli-common, and entry-script test harness crates). - Reshaped container runtime/Compose workflow (baseline
compose.yaml,compose.override.yaml,make up-dev/make up-prod, new CI lint guards). - Updated application/tests/docs for mandatory
tracker.token+database.connect_url, TLS spelling fix (tsl→tls), and helper-binary JSON stdout contracts.
Reviewed changes
Copilot reviewed 91 out of 101 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/environments/isolated.rs | Switch test settings fixture to placeholder settings |
| tests/e2e/web/api/v1/contexts/settings/contract.rs | Normalise env-specific fields for settings comparison |
| tests/e2e/config.rs | Make bootstrap config test hermetic under figment::Jail |
| src/web/api/server/v1/extractors/require_permission.rs | Auth/permission extractor error handling tweak |
| src/web/api/server/v1/contexts/settings/mod.rs | Fix docs typo tsl → tls |
| src/web/api/server/mod.rs | Rename TLS type + re-export DynError from config crate |
| src/web/api/mod.rs | Thread TLS option through API start |
| src/tests/web/require_permission.rs | Use Configuration::for_tests() |
| src/tests/services/settings.rs | Use Configuration::for_tests() |
| src/tests/jwt.rs | Use Configuration::for_tests() |
| src/tests/config/mod.rs | Stabilise config tests vs inherited env overrides |
| src/tests/bootstrap/config.rs | Inject mandatory secrets via env overrides in tests |
| src/services/authorization.rs | Re-export permission value types from config crate |
| src/lib.rs | Update container/config docs for mandatory secrets + new guides |
| src/jwt.rs | Update helper-binary naming/contract docs |
| src/config/mod.rs | Re-export config parsing surface + add Configuration::for_tests() |
| src/bootstrap/config.rs | Re-export default config path from config crate |
| src/bin/health_check.rs | Remove legacy health-check binary (moved to package) |
| src/bin/generate_auth_keypair.rs | Remove legacy keypair generator (moved to package) |
| src/app.rs | Rename net tsl → tls usage |
| share/default/config/tracker.public.e2e.container.sqlite3.toml | Remove shipped token secret |
| share/default/config/tracker.private.e2e.container.sqlite3.toml | Remove shipped token secret |
| share/default/config/index.public.e2e.container.toml | Remove creds/paths; keep runtime-driven fields |
| share/default/config/index.public.e2e.container.sqlite3.toml | Remove redundant sample file |
| share/default/config/index.private.e2e.container.sqlite3.toml | Remove creds/paths from sample |
| share/default/config/index.development.sqlite3.toml | Switch to [net.tls] + remove placeholder secrets |
| share/default/config/index.container.toml | Remove creds/paths from sample |
| share/default/config/index.container.sqlite3.toml | Remove redundant sample file |
| share/container/entry_script_lib_sh | Add POSIX shell helper library for entry script |
| packages/index-health-check/tests/health_check.rs | Health-check integration tests |
| packages/index-health-check/src/tests/mod.rs | Health-check unit tests |
| packages/index-health-check/src/lib.rs | Minimal TCP-based health check implementation |
| packages/index-health-check/src/bin/torrust-index-health-check.rs | Health-check JSON/TTY contract binary |
| packages/index-health-check/Cargo.toml | New crate manifest |
| packages/index-entry-script/tests/validate_auth_keys.rs | Host-side tests for shell validate_auth_keys |
| packages/index-entry-script/tests/seed_sqlite.rs | Host-side tests for shell seed_sqlite |
| packages/index-entry-script/src/lib.rs | Test harness to execute sourced shell helpers |
| packages/index-entry-script/Cargo.toml | New crate manifest |
| packages/index-config/tests/shipped_samples.rs | Ensure shipped samples are TOML-valid but secret-free |
| packages/index-config/tests/round_trip.rs | Round-trip tests via TOML/JSON public API |
| packages/index-config/tests/permission_overrides.rs | Permission override parsing tests |
| packages/index-config/src/validator.rs | New validator trait + error type |
| packages/index-config/src/v2/website.rs | Website schema module extraction |
| packages/index-config/src/v2/tracker_statistics_importer.rs | Importer schema module extraction |
| packages/index-config/src/v2/tracker.rs | Make tracker token schema-mandatory + add ApiToken::is_empty() |
| packages/index-config/src/v2/registration.rs | Registration/email schema module extraction |
| packages/index-config/src/v2/permissions.rs | Wire permission override types within config crate |
| packages/index-config/src/v2/net.rs | Rename network TLS config (tsl → tls) |
| packages/index-config/src/v2/mod.rs | Make tracker/database sections schema-mandatory |
| packages/index-config/src/v2/mail.rs | Mail/SMTP schema module extraction |
| packages/index-config/src/v2/logging.rs | Logging schema module extraction |
| packages/index-config/src/v2/image_cache.rs | Image-cache schema module extraction |
| packages/index-config/src/v2/database.rs | Make database connect_url schema-mandatory |
| packages/index-config/src/v2/auth.rs | Auth/JWT schema module extraction + key resolution helpers |
| packages/index-config/src/v2/api.rs | API schema module extraction |
| packages/index-config/src/tests/redaction.rs | Secret redaction tests |
| packages/index-config/src/tests/quirks.rs | Edge-case config tests |
| packages/index-config/src/tests/permissions.rs | Permission types round-trip tests |
| packages/index-config/src/tests/mod.rs | Config crate test suite organisation |
| packages/index-config/src/tests/metadata.rs | Metadata/version tests |
| packages/index-config/src/tests/loader.rs | Loader error/mandatory-field tests |
| packages/index-config/src/test_helpers.rs | Shared placeholder TOML/settings fixtures |
| packages/index-config/src/permissions.rs | Permission value types moved into config crate |
| packages/index-config/src/lib.rs | Config loader + shared constants/types |
| packages/index-config/Cargo.toml | New crate manifest |
| packages/index-config-probe/tests/binary.rs | Probe contract tests (no subprocess) |
| packages/index-config-probe/src/tests/mod.rs | Probe unit tests |
| packages/index-config-probe/src/lib.rs | Probe library (DB/auth subset) |
| packages/index-config-probe/src/bin/torrust-index-config-probe.rs | Probe binary with exit-code contract |
| packages/index-config-probe/Cargo.toml | New crate manifest |
| packages/index-cli-common/src/lib.rs | Shared CLI JSON/TTY/tracing helpers |
| packages/index-cli-common/Cargo.toml | New crate manifest |
| packages/index-auth-keypair/tests/keypair_generation.rs | Keypair integration tests |
| packages/index-auth-keypair/src/tests/mod.rs | Keypair unit tests |
| packages/index-auth-keypair/src/lib.rs | Keypair generation library |
| packages/index-auth-keypair/src/bin/torrust-index-auth-keypair.rs | Keypair binary with JSON stdout |
| packages/index-auth-keypair/Cargo.toml | New crate manifest |
| contrib/dev-tools/su-exec/AUDIT.md | Vendored su-exec provenance + SHA-anchored audit log |
| contrib/dev-tools/container/run.sh | Remove stale container run script |
| contrib/dev-tools/container/e2e/sqlite/run-e2e-tests.sh | Update e2e runner env overrides for mandatory secrets |
| contrib/dev-tools/container/e2e/sqlite/mode/public/e2e-env-up.sh | Update e2e env-up config + inject connect_url override |
| contrib/dev-tools/container/e2e/sqlite/mode/public/e2e-env-down.sh | Update e2e env-down config path |
| contrib/dev-tools/container/e2e/sqlite/mode/private/e2e-env-up.sh | Inject connect_url override for private-mode e2e |
| contrib/dev-tools/container/e2e/mysql/run-e2e-tests.sh | Update mysql e2e runner config + mandatory overrides |
| contrib/dev-tools/container/e2e/mysql/e2e-env-up.sh | Update mysql env-up config + inject connect_url override |
| contrib/dev-tools/container/e2e/mysql/e2e-env-down.sh | Update mysql env-down config path |
| contrib/dev-tools/container/build.sh | Remove stale container build script |
| compose.yaml | Production-shaped baseline compose (with env/port hardening) |
| compose.override.yaml | Dev sandbox override (mailcatcher/tty/default creds) |
| README.md | Update container run docs + compose workflow |
| Makefile | Add up-dev / up-prod entrypoints + env validation |
| Cargo.toml | Expand workspace members + add config crate dependency |
| Cargo.lock | Lockfile updates for new workspace crates |
| CHANGELOG.md | Document ADR-T-009 additions/breaking changes |
| AGENTS.md | Add contributor guidance updates |
| .github/workflows/container.yaml | Add container-infra lint guards + job dependency |
| .containerignore | Exclude docs/adr from container build context |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Implements ADR-T-009’s container-infrastructure refactor by extracting configuration + helper tooling into dedicated workspace crates, reshaping the container/compose story for production vs dev, and updating tests/docs to match the new “mandatory credentials at schema level” configuration contract (plus a small auth correctness fix around “user not found” vs DB errors).
Changes:
- Extracts config parsing into
torrust-index-config(and adds helper crates like config-probe, health-check, auth-keypair, entry-script test harness). - Updates configuration schema/fixtures so
tracker.tokenanddatabase.connect_urlare mandatory (no shipped defaults), and adjusts tests/docs/compose accordingly. - Introduces production-shaped
compose.yaml+ dev-onlycompose.override.yaml, plus CI lints and a Makefile wrapper for validated prod bring-up.
Reviewed changes
Copilot reviewed 91 out of 101 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/environments/isolated.rs | Switches test fixture from Settings::default() to shared placeholder settings fixture. |
| tests/e2e/web/api/v1/contexts/settings/contract.rs | Normalizes environment-dependent settings fields before asserting equality. |
| tests/e2e/config.rs | Makes bootstrap config test hermetic under figment::Jail and injects mandatory overrides. |
| src/web/api/server/v1/extractors/require_permission.rs | Improves error-category preservation/logging when loading user for permission checks fails. |
| src/web/api/server/v1/contexts/settings/mod.rs | Fixes docs typo (tsl → tls) in settings JSON example. |
| src/web/api/server/mod.rs | Renames TSL→TLS, re-exports DynError from extracted config crate, updates comments. |
| src/web/api/mod.rs | Propagates TSL→TLS rename through API start wiring. |
| src/tests/web/require_permission.rs | Uses Configuration::for_tests() in tests after removal of default settings. |
| src/tests/services/settings.rs | Uses Configuration::for_tests() for settings service tests. |
| src/tests/jwt.rs | Uses Configuration::for_tests() for JWT tests. |
| src/tests/config/mod.rs | Adds env-scrubbing helper + updates tests to use placeholder-derived defaults under figment::Jail. |
| src/tests/bootstrap/config.rs | Injects mandatory tracker/database env overrides for bootstrap config test. |
| src/services/authorization.rs | Moves permission value types to config crate and re-exports them here for compatibility. |
| src/lib.rs | Updates public docs for new mandatory config fields + compose split guidance. |
| src/jwt.rs | Updates docs to reference new torrust-index-auth-keypair helper binary and JSON output. |
| src/config/mod.rs | Re-exports config crate parsing surface and adds Configuration runtime wrapper + for_tests(). |
| src/bootstrap/config.rs | Re-exports default config TOML path from config crate for single source of truth. |
| src/bin/health_check.rs | Removes old health check binary (replaced by workspace crate). |
| src/bin/generate_auth_keypair.rs | Removes old keypair generator binary (replaced by workspace crate). |
| src/app.rs | Wires net.tls (renamed from net.tsl) through application startup. |
| share/default/config/tracker.public.e2e.container.sqlite3.toml | Removes shipped tracker token from e2e tracker sample (now operator-supplied). |
| share/default/config/tracker.private.e2e.container.sqlite3.toml | Removes shipped tracker token from e2e tracker sample (now operator-supplied). |
| share/default/config/index.public.e2e.container.toml | Removes credentials/auth-path defaults from shipped sample; keeps structural settings. |
| share/default/config/index.public.e2e.container.sqlite3.toml | Removes old sqlite3 container sample file. |
| share/default/config/index.private.e2e.container.sqlite3.toml | Removes credentials/auth-path defaults from shipped sample. |
| share/default/config/index.development.sqlite3.toml | Updates sample to [net.tls] and removes credential defaults. |
| share/default/config/index.container.toml | Removes credentials/auth-path defaults from shipped container sample. |
| share/default/config/index.container.sqlite3.toml | Removes old sqlite3 container sample file. |
| share/container/entry_script_lib_sh | Adds POSIX-sh helper library for entry script invariants + sqlite seeding. |
| packages/index-health-check/tests/health_check.rs | Adds integration tests for health-check behavior and JSON output fields. |
| packages/index-health-check/src/tests/mod.rs | Adds unit tests covering HTTP parsing, timeouts, IPv4/IPv6 behavior. |
| packages/index-health-check/src/lib.rs | Introduces minimal stdlib-only health check with Happy Eyeballs connect. |
| packages/index-health-check/src/bin/torrust-index-health-check.rs | Adds helper binary wiring (TTY refusal, JSON tracing, JSON stdout). |
| packages/index-health-check/Cargo.toml | Defines new health-check crate and dependencies. |
| packages/index-entry-script/tests/validate_auth_keys.rs | Tests shell helper validate_auth_keys contract host-side via sh. |
| packages/index-entry-script/tests/seed_sqlite.rs | Tests shell helper seed_sqlite contract host-side via sh. |
| packages/index-entry-script/src/lib.rs | Provides harness to embed and source shell lib for host-side tests. |
| packages/index-entry-script/Cargo.toml | Defines entry-script test harness crate. |
| packages/index-config/tests/shipped_samples.rs | Ensures shipped index.*.toml samples are valid TOML but omit credentials and fail schema until overridden. |
| packages/index-config/tests/round_trip.rs | Adds round-trip tests through TOML/JSON APIs using public surface. |
| packages/index-config/tests/permission_overrides.rs | Validates [[permissions.overrides]] parsing, ordering, and error cases. |
| packages/index-config/src/validator.rs | Adds validator trait + config validation error type (with message). |
| packages/index-config/src/v2/website.rs | Adds website configuration types and defaults in extracted config crate. |
| packages/index-config/src/v2/tracker_statistics_importer.rs | Adds tracker statistics importer settings with defaults. |
| packages/index-config/src/v2/tracker.rs | Makes token mandatory, removes defaults, and adds ApiToken::is_empty() helper. |
| packages/index-config/src/v2/registration.rs | Adds registration/email config types with defaults. |
| packages/index-config/src/v2/permissions.rs | Adjusts permission override import to config crate location. |
| packages/index-config/src/v2/net.rs | Renames tsl → tls and uses extracted Tls type. |
| packages/index-config/src/v2/mod.rs | Makes tracker/database sections mandatory (no schema-level default); removes Settings::default(). |
| packages/index-config/src/v2/mail.rs | Adds mail/SMTP config types and defaults. |
| packages/index-config/src/v2/logging.rs | Adds logging config + Threshold enum. |
| packages/index-config/src/v2/image_cache.rs | Adds image cache config with defaults. |
| packages/index-config/src/v2/database.rs | Makes connect_url mandatory and removes defaults. |
| packages/index-config/src/v2/auth.rs | Adds auth config with PEM/path resolution helpers and defaults. |
| packages/index-config/src/v2/api.rs | Adds API paging config with defaults. |
| packages/index-config/src/tests/redaction.rs | Adds tests for secret redaction behavior in settings. |
| packages/index-config/src/tests/quirks.rs | Adds edge-case tests (TLS empty strings, IPv6, unicode token, validator behavior). |
| packages/index-config/src/tests/permissions.rs | Adds tests for permission value types round-tripping and invariants. |
| packages/index-config/src/tests/mod.rs | Adds test suite structure and shared test helper exports. |
| packages/index-config/src/tests/metadata.rs | Adds tests for metadata/version/app/purpose serialization and defaults. |
| packages/index-config/src/tests/loader.rs | Adds tests covering loader success + failure modes and error mapping. |
| packages/index-config/src/test_helpers.rs | Adds canonical placeholder TOML + placeholder settings fixture for cross-crate tests. |
| packages/index-config/src/permissions.rs | Moves permission value types into config crate for schema reuse + re-export. |
| packages/index-config/src/lib.rs | Introduces extracted config crate API (types, loader, constants, TLS type, Info/Error). |
| packages/index-config/Cargo.toml | Defines new config crate and dependency set. |
| packages/index-config-probe/tests/binary.rs | Adds contract tests for probe exit codes and JSON emission without spawning the binary. |
| packages/index-config-probe/src/tests/mod.rs | Adds unit tests for probe behavior (sqlite path extraction, auth key probing, scheme gates). |
| packages/index-config-probe/src/lib.rs | Adds config-probe library emitting container-relevant JSON from loaded settings. |
| packages/index-config-probe/src/bin/torrust-index-config-probe.rs | Adds config-probe helper binary with exit-code contract and panic hook mapping. |
| packages/index-config-probe/Cargo.toml | Defines new config-probe crate and dependency set. |
| packages/index-cli-common/src/lib.rs | Adds shared helper-binary scaffolding (TTY refusal, JSON tracing, JSON emit, --debug). |
| packages/index-cli-common/Cargo.toml | Defines CLI-common crate. |
| packages/index-auth-keypair/tests/keypair_generation.rs | Adds integration tests for JSON output + PEM validity and uniqueness. |
| packages/index-auth-keypair/src/tests/mod.rs | Adds unit tests for keypair generation outputs. |
| packages/index-auth-keypair/src/lib.rs | Adds RSA-2048 keypair generator library returning JSON-serializable output. |
| packages/index-auth-keypair/src/bin/torrust-index-auth-keypair.rs | Adds helper binary for keypair generation using shared CLI scaffolding. |
| packages/index-auth-keypair/Cargo.toml | Defines auth-keypair crate and dependencies. |
| contrib/dev-tools/su-exec/AUDIT.md | Adds provenance + append-only SHA-256 audit log for vendored su-exec.c. |
| contrib/dev-tools/container/run.sh | Removes stale container run helper script. |
| contrib/dev-tools/container/e2e/sqlite/run-e2e-tests.sh | Updates e2e runner to supply mandatory overrides and updated TOML paths. |
| contrib/dev-tools/container/e2e/sqlite/mode/public/e2e-env-up.sh | Updates compose env-up script for new TOML + mandatory DB connect URL override. |
| contrib/dev-tools/container/e2e/sqlite/mode/public/e2e-env-down.sh | Updates compose env-down script for renamed TOML. |
| contrib/dev-tools/container/e2e/sqlite/mode/private/e2e-env-up.sh | Adds mandatory DB connect URL override for private mode. |
| contrib/dev-tools/container/e2e/mysql/run-e2e-tests.sh | Updates mysql e2e runner to supply mandatory overrides and updated TOML path. |
| contrib/dev-tools/container/e2e/mysql/e2e-env-up.sh | Updates compose env-up script for new TOML + mandatory DB connect URL override. |
| contrib/dev-tools/container/e2e/mysql/e2e-env-down.sh | Updates compose env-down script for renamed TOML. |
| contrib/dev-tools/container/build.sh | Removes stale container build helper script. |
| compose.yaml | Refactors compose baseline for production posture (no dev sidecars/default creds, localhost-bound ports). |
| compose.override.yaml | Adds dev-only overrides (mailcatcher, tty, permissive credential defaults). |
| README.md | Updates container run instructions and documents compose split workflow. |
| Makefile | Adds up-dev / up-prod targets with env validation for production bring-up. |
| Cargo.toml | Expands workspace members and adds dependency on extracted config crate. |
| Cargo.lock | Updates lockfile for new workspace crates and dependency graph. |
| CHANGELOG.md | Documents ADR-T-009 changes, breaking config requirements, and extracted helper crates. |
| AGENTS.md | Adds commit-message guidance, POSIX-path guidance, and updates atomic-write instructions. |
| .github/workflows/container.yaml | Adds container-infra lint job (compose baseline audit, su-exec hash guard, env-var docs guard). |
| .containerignore | Excludes adr/ and docs/ from container build context. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
peer-cat
force-pushed
the
feat/rework_container_system
branch
from
239faf6 to
51e8269
Compare
peer-cat
force-pushed
the
feat/rework_container_system
branch
from
4f6c532 to
c466920
Compare
Adopt ADR-T-009 to fix correctness, security, and maintainability
issues across the Containerfile, compose.yaml, entry script, and
container docs.
Security & correctness (Phase 1):
- Entry script `set -x` is now gated behind `DEBUG=1` so credentials
in env-var expansions no longer leak into container logs.
- Fix MySQL compose healthcheck: it referenced a non-existent
`/run/secrets/db-password` and silently fell back to no password;
use `$$MYSQL_ROOT_PASSWORD` instead.
- Bind dev-only ports to `127.0.0.1` (MySQL 3306, tracker
6969/7070/1212, mailcatcher 1025/1080) so they no longer listen on
every interface.
- Expose `IMPORTER_API_PORT` (3002) in the Containerfile and map it
on `127.0.0.1` in compose; the release HEALTHCHECK already probes
it.
- Annotate compose credentials as DEV-ONLY with a TODO to migrate to
Docker secrets.
- Fix entry-script `USER_ID` guard: `[ -z "$USER_ID" ] && [ ... -lt
1000 ]` short-circuited to false when unset; corrected to `||`.
- Remove stale `contrib/dev-tools/container/build.sh` and `run.sh`
(wrong build-arg, wrong mount paths, missing `Dockerfile`).
- Fix `USER_UID` → `USER_ID` typo in `docs/containers.md`.
Maintainability & operations (Phase 2):
- Pin `cargo-binstall` bootstrap to tag `v1.18.1` (was `main`) in
both the chef and tester stages.
- Pin MySQL image to `8.0.45` and switch the deprecated
`--default-authentication-plugin` flag to `--authentication-policy`.
- Add `restart: unless-stopped` to the index and tracker compose
services.
- Document the debug image's intentional HEALTHCHECK omission, the
busybox applet subset (`sh`, `cat`, `ls`, `env`), and the
`DEBUG=1` entry-script tracing knob.
- Upgrade base images from Debian bookworm to trixie (`rust:trixie`,
`gcc:trixie`, `cc-debian13`).
- Drop redundant `--tests --benches --examples` from `cargo chef
cook` / `cargo nextest archive` — already covered by
`--all-targets`.
- Strip leftover `TORRUST_TRACKER_USER_UID` exports from the E2E
scripts; the tracker uses `USER_ID`.
- Rename `.dockerignore` → `.containerignore` for Podman
compatibility.
- Tidy: fix Containerfile typo ("wat") and trailing whitespace on
the release HEALTHCHECK line.
Adjacent fixes:
- `Info::parse_args`: treat an empty `TORRUST_INDEX_CONFIG_TOML` env
var the same as unset, so an empty value no longer overrides the
config file path.
- `RequirePermission`: log non-`UserNotFound` DB errors during the
permission lookup so operators can distinguish a 404 from a
database outage masquerading as one.
- Doc-comment fix: `TorrentRequest` → `AddTorrentRequest` in
`add_torrent_handler`.
AGENTS.md:
- Add "Commit Messages" guideline (review recent style first).
- Add "POSIX Paths" guideline on treating paths as opaque bytes.
Supersede the tactical "Container Infrastructure Hardening" ADR (Phases 1 & 2 already landed in a2ea512) with a structural refactor ADR and a companion implementation plan. The new ADR records nine decisions (D1–D9) derived from nine principles (P1–P9), each traceable to one or more diagnostic items (R1–R10) catalogued in the appendix: D1 Split compose into baseline + override D2 Strip credentials from defaults; mandatory connect_url D3 Single source of truth for auth-key paths D4 Two parallel runtime bases; root-only utilities D5 Helper binaries as separate workspace crates D6 Drop build-time ARG for runtime concerns D7 Refuse-if-root entry-script guard D8 Vendored su-exec gains an internal audit record D9 Build hygiene and test-stage coupling The implementation plan details nine phases with dependency ordering, Containerfile snippets, shell fragments, and executable acceptance scripts for every criterion. AGENTS.md gains a note that the helper crates (index-health-check, index-auth-keypair, index-config, index-config-probe) share the T- prefix.
Drop build-time ARG API_PORT / ARG IMPORTER_API_PORT from the Containerfile and keep only the runtime ENV defaults (3001, 3002), so listener and HEALTHCHECK still resolve and --env overrides keep working — though image metadata (docker inspect / docker port) now unconditionally reflects the defaults. Exclude /adr/ and /docs/ from the build context via .containerignore (and fix its missing trailing newline), document the EXPOSE / --env caveat in docs/containers.md, and mark D6 and the build-context part of D9 as landed in the ADR and implementation plan.
Land Phase 2 of ADR-T-009 by hoisting the two container helper
binaries out of the application crate and into their own workspace
crates, behind a shared scaffolding library. The application's
HTTP/TLS dep closure is no longer dragged into the helpers.
New workspace crates:
- `packages/index-cli-common/` — the universal P9 scaffolding
(`refuse_if_stdout_is_tty`, `init_json_tracing`, `emit`,
`BaseArgs`) shared by every Torrust Index helper binary.
- `packages/index-health-check/` — hosts `torrust-index-health-check`
(renamed from `health_check`), rewritten on top of
`std::net::TcpStream` with Happy Eyeballs IPv6/IPv4 fallback.
No `reqwest`, `tokio`, `hyper`, or TLS in the dep closure.
- `packages/index-auth-keypair/` — hosts `torrust-index-auth-keypair`
(renamed from `torrust-generate-auth-keypair`). Output format
changed from raw concatenated PEM blocks to a single JSON
object `{"private_key_pem": "...", "public_key_pem": "..."}`,
so consumers no longer need to scan for PEM markers.
Each helper crate splits domain logic into `src/lib.rs` with a thin
`src/bin/<binary>.rs` shim, so integration tests in `tests/` can
exercise the public surface directly. The old `src/bin/health_check.rs`
and `src/bin/generate_auth_keypair.rs` files are removed, along with
the `[[bin]]` entry for `torrust-generate-auth-keypair` on the root
crate.
Container changes:
- Add a pristine `jq_donor` stage to the `Containerfile` and copy
`/usr/bin/jq` into the runtime image with `--chmod=0500 --chown=0:0`.
When Phase 4 splits the runtime base, this `COPY` line must be
re-added to both `runtime_release` and `runtime_debug`.
- Release `HEALTHCHECK` now invokes `torrust-index-health-check`;
drop the trailing `|| exit 1` since the binary already exits
non-zero on failure.
- Entry script invokes `torrust-index-auth-keypair` and consumes
its JSON output via `jq -r .private_key_pem` / `jq -r .public_key_pem`
instead of `sed` PEM-block extraction.
- Helper TTY-refusal exit code unified on 2 (was 1 for the keypair
helper) via the shared `refuse_if_stdout_is_tty`.
Docs:
- ADR-T-007 Phase 6 updated for the renamed binary, new JSON output,
`jq` consumer recipe, and the new crate location.
- ADR-T-009 Phase 2 marked landed, with notes on the
`src/lib.rs` + `src/bin/` layout refinement and the Phase 4
follow-up for re-adding the `jq` `COPY` line.
- D5 status updated; `CHANGELOG.md` and `src/jwt.rs` module doc
updated for the renames and JSON output.
`cargo tree -e normal -p torrust-index-health-check` and
`-p torrust-index-auth-keypair` confirm neither helper links
`reqwest`, `tokio`, `hyper`, `rustls`, `native-tls`, or `openssl`.
The new packages ship roughly twenty crate-level and integration
tests, all passing.
Land Phase 3 of ADR-T-009 by hoisting the parsing surface of `src/config/` into a new leaf workspace crate so the configuration schema no longer drags `tokio`, `reqwest`, `sqlx`, `hyper`, `rustls`, `native-tls`, or `openssl` into anything that only needs to read a TOML file. This is the foundation Phase 6 (`index-config-probe`) will sit on top of. New workspace crate: - `packages/index-config/` — owns the v2 schema modules (`api`, `auth`, `database`, `image_cache`, `logging`, `mail`, `net`, `registration`, `tracker`, `tracker_statistics_importer`, `website`, `permissions`), the `validator`, `load_settings`, `Info`, `Error`, the `CONFIG_OVERRIDE_*` / `ENV_VAR_CONFIG_TOML*` constants, `Tls`, and a `pub type DynError` alias that breaks the inward dependency on the web layer. The runtime `Configuration` wrapper holding `tokio::sync::RwLock<Settings>` and its `async` accessors stay in the root crate, beneath a thin `pub use torrust_index_config::*;` re-export shim in `src/config/mod.rs`. Every existing `use crate::config::*;` call site continues to compile unchanged. Permission value types: - `Role`, `Action`, `Effect`, `PermissionOverride`, and `RoleParseError` move into `torrust_index_config::permissions` and are re-exported from `crate::services::authorization` for backwards compatibility. The `Permissions` *trait* and the `PermissionMatrix` runtime policy stay in the root crate — only the value types the schema needs to deserialise `[permissions.overrides]` move down. Dep closure: - The new crate's non-stdlib deps are `serde`, `serde_json`, `serde_with`, `figment`, `toml`, `url`, `camino`, `derive_more`, `thiserror`, `tracing`, and `lettre` (the latter with `default-features = false` and only the `builder` and `serde` features, because the schema parses `smtp.from` / `smtp.reply_to` directly into `lettre::message::Mailbox`). `serde_json` is used by `Settings::to_json` and the crate's own tests. `cargo tree -p torrust-index-config -e normal` confirms `tokio`, `reqwest`, `sqlx`, `hyper`, `rustls`, `native-tls`, and `openssl` are all absent. The root `Cargo.toml` drops its now-unused direct deps on `camino` and `serde_with`. Breaking change — `Tsl` → `Tls`: - The original spelling was a typo. Corrected as a clean break alongside the crate extraction: type, field, serde wire key (`[net.tsl]` → `[net.tls]` in operator TOMLs, `"tsl"` → `"tls"` in the settings JSON API response), local variables, the shipped TOML defaults in `share/default/config/index.development.sqlite3.toml`, and the JSON example in `src/web/api/server/v1/contexts/settings/mod.rs` were all updated together. `grep -rE 'Tsl|\.tsl' src/ share/ packages/` returns zero hits. No compatibility alias is provided. Tests: - Crate-level tests live in `packages/index-config/src/tests/` (`loader`, `metadata`, `permissions`, `quirks`, `redaction`). - Public-API integration tests live in `packages/index-config/tests/` (`permission_overrides`, `round_trip`, `shipped_samples`). - All pass alongside the existing workspace suite. Docs: - ADR-T-009 implementation plan gains a "Phase Status" table, marks Phase 3 as landed, and amends the §3 dep list with `serde_json` and `lettre` plus their rationale. - `CHANGELOG.md` records the new crate, the schema/permission moves, and the `Tsl` → `Tls` breaking rename. - AGENTS.md "Replacing a File" guideline corrected to use `mv tmp file` atomic rename instead of `rm` + recreate.
Restructure the Containerfile around two parallel runtime bases layered onto a shared base-agnostic asset bundle: - runtime_release on lean gcr.io/distroless/cc-debian13, shipping a single root-only /bin/busybox (0700 root:root) plus a curated set of applet symlinks (sh, adduser, addgroup, install, mkdir, dirname, chown, chmod, tr, mktemp, cat, printf, rm, echo, grep). The unprivileged torrust user gets EACCES on the busybox binary after privilege drop. - runtime_debug on the :debug variant, keeping the full /busybox/ tree on PATH so operators retain the complete applet set as the application user. Both bases pull from a shared runtime_assets stage and a preflight_gate that aggregates donor-validation stages (busybox_preflight, adduser_preflight) so a future base reshuffle fails at build time rather than first boot. Other Phase 4 changes: - Tighten helper binaries (torrust-index-health-check, torrust-index-auth-keypair) to 0500 root:root in both images; the application binary keeps 0755. HEALTHCHECK runs as root, so the tightened mode is sufficient. - Bring the debug target in line with release: same HEALTHCHECK block, same ENTRYPOINT, default CMD set to torrust-index so the debug image is a drop-in replacement. Operators reach a shell with `docker run … sh`. - Pin PATH in both runtime bases so the entry script's bare-name lookups resolve deterministically. - Switch the entry script to the busybox adduser short-option form so the same invocation works on both bases. - D7: replace the `USER_ID >= 1000` guard with "non-negative integer, not 0". The previous rule rejected legitimate configurations (rootless Podman with subuid remapping, low-UID CI runners, BSD-derived hosts) without stating its intent; the property the entry script actually enforces is "do not run as root". BREAKING for anyone relying on the old guard. - Document the Podman `--format docker` requirement for HEALTHCHECK (OCI manifests have no field for the directive; without the flag Podman silently drops it). Refs: ADR-T-009 §D4, §D7.
…se 5) Land Phase 5 of ADR-T-009 by removing schema-level defaults for the two operator-mandatory secrets and stripping the corresponding literals out of every shipped sample. Zero-config startup is now intentionally rejected: the operator must supply `tracker.token` and `database.connect_url` (via env-var override or a side-loaded TOML), or the loader fails with a precise serde `missing field` error. Schema changes (`packages/index-config/`): - Drop `impl Default for Settings`, `impl Default for Tracker`, `impl Default for Database`, and the matching `#[serde(default = "...")]` attributes on `Settings::tracker`, `Settings::database`, `Tracker::token`, and `Database::connect_url`. The `[tracker]` and `[database]` sections themselves are now mandatory in the parsed TOML. - Remove the trailing `figment.join(Serialized::defaults( Settings::default()))` step from `load_settings`. Optional sub-sections still default through their per-field `#[serde(default = "...")]` attributes; mandatory fields no longer have anywhere to silently come from. - Drop `tracker.token` from the `check_mandatory_options` array. Its absence now surfaces through serde rather than the bespoke pre-flight check, giving a single consistent error shape for every missing mandatory field. Shipped TOMLs (`share/default/config/`): - Remove `connect_url`, `token`, `[mail.smtp]`, and the auth-key paths from every `index.*.container.*.toml`, the dev sample, and both `tracker.*.e2e.container.sqlite3.toml` files. Empty `[tracker]`, `[auth]`, and `[database]` headers remain so the intent is visible. Operator-facing wiring: - `compose.yaml` gains a default `TORRUST_INDEX_CONFIG_OVERRIDE_DATABASE__CONNECT_URL` matching the SQLite path the entry script materialises, alongside the existing `TRACKER__TOKEN` default. - The mysql and sqlite e2e runner scripts export `TRACKER__TOKEN` and `DATABASE__CONNECT_URL` overrides for the host-side `cargo test` process, mirroring what the container receives. - `docs/containers.md` documents the new bare-`docker run` failure mode, marks both env vars as **Required**, and updates every copy-pasteable invocation. Test-fixture consolidation: Removing the ambient `Settings::default()` deletes the single fixture roughly forty tests across both crates relied on. The implementation introduces a single source of truth: - New `#[doc(hidden)] pub mod test_helpers` in `torrust-index-config` exposing `PLACEHOLDER_TOML` (the canonical "minimal but legal" TOML) and `placeholder_settings()` (loads it via `load_settings`, panicking on failure). `pub` so integration test binaries in either crate can reach it; `#[doc(hidden)]` so it stays out of the public API surface. - `packages/index-config/src/tests/mod.rs` re-exports `PLACEHOLDER_TOML` under its historical `MINIMUM_VALID_TOML` alias for in-tree tests. - `packages/index-config/tests/round_trip.rs` and `tests/permission_overrides.rs` import the helper directly. - The root crate's `Configuration::for_tests` (new, `#[cfg(test)]`, `pub(crate)`) replaces the deleted `impl Default for Configuration` and loads from the same constant. - `tests/environments/isolated.rs::ephemeral` calls `placeholder_settings()` to seed its baseline. `tests/e2e/config.rs` intentionally does *not* use the helper: it exercises the real shipped sample plus env-var overrides (mirroring the production `initialize_configuration` flow) and would lose its purpose if it swapped in the placeholder. Hermeticity guard: `Configuration::for_tests` and `load_settings` always merge any `TORRUST_INDEX_CONFIG_OVERRIDE_*` (and `TORRUST_INDEX_CONFIG_TOML[_PATH]`) that happen to be set in the surrounding shell — and the e2e runner scripts now export them. `src/tests/config/mod.rs` gains a `clear_inherited_config_env()` helper that strips those names inside a `figment::Jail` closure so default-configuration assertions stay deterministic when the suite is re-run after an e2e session. Loader tests (`packages/index-config/src/tests/loader.rs`): - Rewrite `missing_tracker_token_is_rejected` to expect a serde `ConfigError` mentioning `token` instead of the old `MissingMandatoryOption` variant. - Add `missing_database_connect_url_is_rejected` and `missing_database_section_is_rejected` covering the two new failure paths. - Update the minimum-valid-TOML fixture and every other test in the module to include a `[database]` section with a `connect_url`. Shipped-sample tests (`packages/index-config/tests/shipped_samples.rs`): The "every shipped TOML loads" smoke test inverts: shipped samples are now expected to be structurally valid but operator-incomplete. - `every_shipped_index_toml_is_valid_toml` parses each sample as raw TOML. - `every_shipped_index_toml_omits_credentials` asserts no shipped sample contains `connect_url`, `token =`, `[mail.smtp]`, `private_key_path`, or `public_key_path` (ADR-T-009 §D2). - `every_shipped_index_toml_demands_runtime_secrets` asserts the schema rejects each sample with a missing-field error mentioning `token` or `connect_url`. - `development_sqlite3_uses_info_threshold` switches to raw TOML parsing since the sample no longer schema-loads on its own. Docs: - ADR-T-009 marks Phase 5 as landed (2026-04-24) and the §D2 design decision as resolved by Phase 5. - The implementation plan corrects the pre-Phase-3 paths (`src/config/v2/...` → `packages/index-config/src/v2/...`) and adds §5.3 documenting the test-fixture consolidation rationale. Refs: ADR-T-009 §D2.
…009 Phase 6)
Land Phase 6 of ADR-T-009 by introducing the third and final helper
binary, `torrust-index-config-probe`. The probe loads the application's
`Settings` through `torrust-index-config` (no bespoke parsing) and
emits the container-relevant subset as a single JSON object on
stdout, giving the entry script (Phase 7) a stable, typed interface
to dispatch on instead of re-parsing TOML in shell.
New crate (`packages/index-config-probe/`):
- Binary `torrust-index-config-probe` emits one line of JSON with
`schema`, `database.driver` (`sqlite` / `mysql`, modelled
internally as a `Driver` enum), `database.path` (sqlite path or
`null`), and `auth.{private,public}_key` records carrying
`pem_set`, `path_set`, `source` ∈ {`pem`,`path`,`none`}, and the
resolved `path`. Empty-string-equals-absent collapsing happens
here, at the container boundary, so a bare `${VAR}` in compose
that substitutes to `""` is treated as unset.
- Refuses to run on a TTY (P8), never echoes PEM material on
stdout, and exits with documented codes: 0 success, 1
panic/I-O (panics mapped from Rust's default 101 via
`set_hook` in `main`), 2 TTY/clap-parse, 3 loader failure
(underlying error forwarded verbatim to stderr via `tracing`),
4 empty `tracker.token`, 5 unsupported database scheme.
- The sqlite path-extraction logic handles all three URL shapes
(`sqlite::memory:`, `sqlite://data.db?mode=rwc`,
`sqlite:///var/...`) and percent-decodes the hierarchical
branch via `decode_utf8_lossy`. The trade-off is documented
inline: non-UTF-8 bytes become `U+FFFD`, which is acceptable
for v1 of the schema.
- Direct deps are limited to the parsing-surface re-exports from
`torrust-index-config` plus `clap`, `serde_json`, `tracing`,
`url`, and `percent-encoding` — no HTTP, no async runtime.
- Tests: in-tree unit tests pin the wire format of both enums
(`AuthKeySource`, `Driver`) with lowercase-serialisation
assertions, plus the auth-key matrix and the URL-spelling
table; `tests/binary.rs` exercises the compiled binary's exit
codes and JSON shape end-to-end.
Loader changes (`packages/index-config/`):
- New `Info::from_env(default)` constructor: the JSON-safe sibling
of `Info::new` that reads `TORRUST_INDEX_CONFIG_TOML[_PATH]`
exactly the same way but skips the diagnostic `println!`s, so
helper binaries with a JSON-only stdout contract (P9) can share
the application loader without corrupting their output stream.
- `Info::new` now routes its "loading extra configuration from …"
diagnostics through `tracing` (stderr) instead of `println!`
(stdout), and is reimplemented on top of `Info::from_env`. The
application's existing `bootstrap::config` call site picks the
stderr-routed messages up transparently.
- New `pub const DEFAULT_CONFIG_TOML_PATH`, re-exported from the
application as `crate::bootstrap::config::DEFAULT_PATH_CONFIG`,
so application, helpers, and integration tests share one source
of truth for the default config-TOML location.
- New `ApiToken::is_empty()` accessor so the probe can reject
`tracker.token = ""` at the container boundary without reaching
into the type's byte representation. Defence in depth against
`#[derive(Deserialize)]` bypassing `ApiToken::new`'s `assert!`.
Docs:
- ADR-T-009 marks the helper-binary status block as landed
(Phase 2 + Phase 6 together shipped all three helpers) and
records that the script-side mutual-exclusion / override-export
logic from §D3 still belongs to Phase 7.
- The implementation plan marks Phase 6 as Done, expands the field-
semantics block to cover the empty-string collapsing, narrows the
driver dispatch to `sqlite` / `mysql` (mirroring the application's
own `databases::database::get_driver`), and documents the
default-config-path single source of truth.
- README gains the ADR-T-009 entry alongside the other ADRs.
- CHANGELOG records the new crate, the two new `Info` constructors,
the new const, the new `ApiToken::is_empty()`, and the
`println!`→`tracing` switch on `Info::new`.
Refs: ADR-T-009 §D3, P9.
Land Phase 7 of ADR-T-009 by rewriting the container entry
script around the Phase 6 config probe and a sourced shell
library, and by adding a host-side Rust test crate that
exercises the pure helpers without spinning up a container.
The script is now the single source of truth for the
container's default auth-key paths, and dispatches database
seeding from the probe's typed `database.driver` field rather
than re-parsing TOML in shell.
Entry script (`share/container/entry_script_sh`):
- Switch to `set -eu`. Unchecked failures abort startup
immediately; references to unset variables are errors.
Existing `${var:-}` patterns audited for `nounset`-safety.
- Source pure helpers from a new sibling library at
`/usr/local/lib/torrust/entry_script_lib_sh` instead of
defining them inline. The library has no top-level side
effects, so sourcing is safe both inside the container
and inside the host-side Rust tests.
- Make `adduser` idempotent: container restarts re-execute
the entry script against the same writable layer, so the
`torrust` user already exists on subsequent boots. Skip
the call rather than letting `adduser` fail under `set -e`.
- Demote `TORRUST_INDEX_DATABASE_DRIVER` to a first-boot
TOML selector. It chooses which default `index.toml` is
installed at `/etc/torrust/index/index.toml` and no longer
drives runtime database decisions; those come from the
probe.
- Invoke `/usr/bin/torrust-index-config-probe` after the
default TOML is in place but *before* the script exports
any `TORRUST_INDEX_CONFIG_OVERRIDE_*` of its own, so the
probe sees only operator-supplied values. Gate on
`.schema == "1"` so a probe/script version skew fails
loudly instead of silently misinterpreting unknown fields.
- Enforce the §7.1 auth-key invariants via the new
`validate_auth_keys` helper: PEM and PATH mutually
exclusive within each key, both keys configured or
neither (no half-pair), and a single delivery mechanism
across the pair (no mixed PEM/PATH).
- Implement the §D3 three-way auth-key dispatch. For
`source=pem` the application loads from the env var
directly. For `source=path` the script honours the
resolved path. For `source=none` the script applies the
container defaults (`/etc/torrust/index/auth/private.pem`
and `.../public.pem`) **and exports the corresponding
`TORRUST_INDEX_CONFIG_OVERRIDE_AUTH__*_PATH` override**
so the application sees the same paths the script
materialises. The defaults live only in the script; no
constant is duplicated between two files.
- Apply a volumes-only directory guard to auth-key parents
(cases 2 and 3): auto-create directories only under
`/etc/torrust/index/`, `/var/lib/torrust/index/`, or
`/var/log/torrust/index/`; anything else is the
operator's responsibility and aborts with a precise
error.
- Drive database seeding from the probe's `database.driver`
field. `sqlite` invokes `seed_sqlite` against the
resolved `database.path`; `mysql` is a no-op (the
application connects directly). The historical
`inst "$default_database" "$install_database"` call,
which seeded the DB unconditionally at a hard-coded
path, is gone — the seed now lands at whatever path the
operator's `connect_url` actually names.
Sourced shell library (`share/container/entry_script_lib_sh`,
new file, mode 0444 root:root):
- Hosts `inst`, `key_configured`, `validate_auth_keys`,
and `seed_sqlite`. Strict POSIX `sh`; no bashisms.
Functions emit human-readable diagnostics on stderr and
use `exit 1` for unrecoverable input violations, so
host-side tests must invoke them in a subshell to
observe the exit status.
- `seed_sqlite` covers all five §7.2 outcomes: empty path
(probe-bug error), `:memory:` (info, no-op), relative
path (warn, no-op), absolute-non-empty (untouched), and
absolute-missing (volumes-only `mkdir` + `inst()`).
- `key_configured` uses an explicit allow-list against
`pem|path` so any future probe value defaults to "not
configured" rather than being silently treated as
configured.
New workspace crate (`packages/index-entry-script/`):
- Test-only `[lib]` with no runtime code of its own.
`dev-dependencies` are `tempfile` only. Exists purely
as a home for `tests/` that drive the shell library via
`sh` subprocess and assert exit codes / stderr contents,
so the helpers run inside `cargo test --workspace` while
the production code remains POSIX `sh`.
- The library contents are embedded via `include_str!` and
written to a per-process tempfile on first use. This
sidesteps the cargo-nextest archive →
`--target-dir-remap` workflow used by the container
`test` stage, where a path computed from
`CARGO_MANIFEST_DIR` would no longer exist at run time.
- `tests/validate_auth_keys.rs` exercises every branch of
the §7.1 invariants. `tests/seed_sqlite.rs` exercises
every §7.2 outcome that does not require chowning into
a container-managed volume; the missing-under-volume
seeded path and the §7.1 case-3 export end-to-end belong
in the container e2e suite (Phase 8 / 9).
Containerfile:
- Copy the new shell library into both runtime bases
(`runtime_assets` feeding the release base, and
`runtime_debug` directly) at
`/usr/local/lib/torrust/entry_script_lib_sh` with
`0444 root:root` — world-readable, root-owned, not
executable on its own (it is sourced, not exec'd).
- Stage the `jq` runtime properly. Debian trixie's `jq`
is dynamically linked against `libjq.so.1` and
`libonig.so.5`; distroless cc-debian13 ships neither.
The `jq_donor` stage now re-stages the binary plus both
shared libraries under deterministic `/jq/` paths, and
both runtime bases COPY all three artefacts into the
multiarch dirs registered in `/etc/ld.so.conf.d/`.
Without this `jq` exec'd from the entry script failed
with a missing-shared-library error at first boot.
- Promote `torrust-index-config-probe` to the same
Phase-2 helper posture as the other root-only helpers:
copied into both `test` and `test_debug` extraction
stages and chowned/chmodded `0500 root:root`. It is
invoked only from the entry script's pre-su-exec phase,
so root-only is sufficient.
E2E test (`tests/e2e/web/api/v1/contexts/settings/contract.rs`):
The "GET /settings" admin contract test compared the live
container's effective configuration against the host-side
loader's view of the same TOML. Phase 7 makes this comparison
legitimately diverge in two normalisable ways: the entry
script may default `auth.{private,public}_key_path` for
`source=none` (the host loader does not), and the container's
absolute `database.connect_url` references the bind-mount
target of the host runner's path. Normalise both fields on
both sides before asserting equality; everything else still
must match exactly.
Docs:
- ADR-T-009 §D3 status moves from "Partially landed" to
"Landed", crediting Phase 6 for the JSON probe and
Phase 7 for the script-side mutual-exclusion checks,
three-way dispatch, and `..._AUTH__*_PATH` export.
- The §3 helper-crate table gains a `torrust-index-entry-script`
row (`dev-dependencies` only, test harness for the
sourced shell library).
- The implementation plan marks Phase 7 as Done, fixes the
default-template path in the §7.2 snippet
(`sqlite3.db` → `index.sqlite3.db`), and adds §7.5
documenting the sourced-library extraction and the
test-coverage split between host-side and container e2e.
- `docs/containers.md` documents the new `set -eu` posture,
the seven-step boot sequence, the volumes-only guard for
auth-key parents, and the conditions under which the
SQLite database file is auto-populated. The bare
`docker/podman run` examples now include the two
mandatory overrides from Phase 5 so the documented
recipes succeed against the new schema.
Refs: ADR-T-009 §D3, §D4, §7.
Land Phase 8 of ADR-T-009 by reshaping `compose.yaml`
into a production-shaped baseline, factoring the dev
sandbox extras into a sibling `compose.override.yaml`,
and wrapping both invocation paths in a top-level
`Makefile`. Bring-up against `podman-compose 1.5.0` /
`podman 5.8.2` surfaced five latent bugs in earlier
phases that only manifest when the full stack actually
starts; per the §8.6 addendum all five are fixed in the
same change so the documented `make up-dev` flow
reaches a healthy `200 OK` on `:3001`.
Compose split (`compose.yaml`, `compose.override.yaml`):
- `compose.yaml` is now the deployment template: no
`mailcatcher`, no `tty`, external ports bound to
`127.0.0.1` (except the index API on `:3001`), and
every credential referenced as bare `${VAR}` with no
default and no `:?required` assertion. Validation is
deferred to `make up-prod` and the in-container
config probe (defence in depth, ADR-T-009 §8.1).
- Operator selectors that have a sensible
cross-environment default (`TORRUST_INDEX_DATABASE`,
`TORRUST_INDEX_DATABASE_DRIVER`,
`MYSQL_DATABASE`, …) keep their `${VAR:-default}`
form so the documented `docker compose up` flow keeps
working out of the box. The `e2e_testing_*` defaults
baked into the previous `compose.yaml` are gone —
that suffix is e2e-runner state, not a production
default.
- `depends_on` switches from short-form to long-form
with explicit `condition:` clauses. Short-form would
cause `compose.override.yaml` to silently *replace*
the base's tracker/mysql edges when re-attaching
`mailcatcher`; long-form merges additively. The
index now waits on `mysql: service_healthy` rather
than `service_started`, picking up the existing
mysql healthcheck.
- All three Docker Hub images
(`mysql:8.0.45`, `dockage/mailcatcher:0.8.2`,
`torrust/tracker:develop`) are fully qualified as
`docker.io/...`. Short-name resolution requires a
TTY prompt that podman cannot honour
non-interactively, and operators should not be
relying on per-host `unqualified-search-registries`
config regardless.
- `compose.override.yaml` (new) re-attaches
`mailcatcher` via long-form `depends_on`, restores
`tty: true` on `index` / `tracker`, and supplies
permissive `${VAR:-default}` defaults for every
credential the baseline leaves blank. Compose v2
auto-loads it for `docker compose up`; `make up-prod`
excludes it via explicit `--file compose.yaml`.
Top-level `Makefile` (new):
- `make up-dev` runs plain `docker compose up`, picking
up the override.
- `make up-prod` runs `_validate-prod-env` first, then
`docker compose --file $(COMPOSE_FILE) up -d --wait`
with the override excluded. The validator uses POSIX
`sh -uc` with `: "${VAR:?message}"` per required
variable, so the file is dependency-free and
shellcheck-clean. Required vars: `USER_ID`,
`TORRUST_INDEX_CONFIG_OVERRIDE_TRACKER__TOKEN`,
`TORRUST_INDEX_CONFIG_OVERRIDE_DATABASE__CONNECT_URL`,
`TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN`,
plus `MYSQL_ROOT_PASSWORD` when the baseline still
carries the local `mysql:` service (defence in depth
on top of the config probe and the MySQL entrypoint's
own empty-password rejection).
- `COMPOSE_FILE` is overridable so acceptance tests and
operators with non-default layouts can redirect the
recipe without filesystem manipulation.
Compose's own `COMPOSE_FILE` env var is ignored once
`--file` is set on the command line, so the
make-variable is the only reliable knob.
§8.6 inline fixes (issues surfaced during end-to-end
bring-up; each would block any operator following the
documented workflow):
1. **Test binaries baking compile-time paths**
(Phases 6, 7).
`packages/index-config-probe/tests/binary.rs` and
`packages/index-entry-script/tests/seed_sqlite.rs`
resolved their fixtures via `env!("CARGO_BIN_EXE_…")`
and `CARGO_MANIFEST_DIR`, both of which bake the
build-time path into the test executable and break
under cargo-nextest's archive →
`--target-dir-remap` flow used in the container
`test` stage. The probe test now drives the contract
through the library API; the entry-script crate
embeds `entry_script_lib_sh` via `include_str!` and
materialises it to a per-process `tempfile` at run
time. `tempfile` is consequently promoted from
`dev-dependencies` to `dependencies` so the helpers
(`run_sh`, `run_sh_with_args`) that the integration
tests call remain compilable from outside the
`cfg(test)` boundary; the diagnostic message on the
UTF-8 conversion is updated to mention the tempfile
provenance.
2. **Missing `addgroup` in the curated busybox applets**
(Phase 4 / 7). Distroless `cc-debian13` ships
`/etc/passwd` and `/etc/group`, but busybox
`adduser -D -u UID NAME` writes only `/etc/passwd`;
`getgrnam("torrust")` then fails and breaks the
entry script's `install -g torrust …` step with
`unknown group torrust`. The implementation-plan
applet loop in §4 gains `addgroup`, and the entry
script now does
`addgroup -g "$USER_ID" torrust` followed by
`adduser -D -s /bin/sh -u "$USER_ID" -G torrust
torrust`. Both steps are guarded by
`grep '^torrust:' /etc/{group,passwd}` so a
container restart stays a no-op under `set -e`.
3. **`jq` shipped without its shared libs** (Phase 2).
The `jq_donor` stage installed jq via `apt-get`
while the runtime stages copied only `/usr/bin/jq`,
so jq aborted with `libjq.so.1: cannot open shared
object file` inside the lean `cc-debian13` runtime
base. The donor stage now (a) copies
`libjq.so.1` and `libonig.so.5` alongside the binary
and (b) asserts an `ldd` allow-list
(`libc.so.6 libjq.so.1 libm.so.6 libonig.so.5
ld-linux-x86-64.so.2 linux-vdso.so.1`) so a future
donor-base upgrade that drags in a new transitive
dep fails the build instead of producing a silently
broken image. The check normalises both bare
sonames and absolute paths from `ldd`'s column 1 so
`/lib64/ld-linux-…` and `ld-linux-…` are treated
identically.
4. **Empty-string env vars treated as configuration
TOML** (Phases 3, 8).
The compose baseline's
`TORRUST_INDEX_CONFIG_TOML=${TORRUST_INDEX_CONFIG_TOML}`
propagated an empty string into the container
whenever the host var was unset, and
`Info::from_env` then handed that zero-byte TOML to
the loader, which rejected it as
`Missing mandatory option: logging.threshold`.
`compose.yaml` now uses Compose's bare-name
pass-through form (`- TORRUST_INDEX_CONFIG_TOML`,
no `=`) for both `TORRUST_INDEX_CONFIG_TOML` and
`TORRUST_TRACKER_CONFIG_TOML`, so an unset host var
is omitted entirely rather than forwarded as empty.
(The companion loader-side fix in `Info::from_env`
was landed earlier in this branch.)
5. **`USER_ID` missing from `_validate-prod-env`**
(Phase 8). `compose.yaml` references `${USER_ID}`
with no default — it is an operator-supplied
numeric selector, not a credential, but neither does
it have a safe cross-environment default. Without
wrapper validation, an unset `USER_ID` propagated to
the entry script's `addgroup -g ""` / `adduser -u
""` calls and failed deep inside container start.
The `_validate-prod-env` recipe now requires
`USER_ID` alongside the credentials, with a
contextual error message
("required (numeric host UID owning ./storage)").
The entry-script-side failure remains the
authoritative gate; this is defence-in-depth in the
spirit of §8.1.
E2E runner scripts
(`contrib/dev-tools/container/e2e/{mysql,sqlite/mode/{private,public}}/e2e-env-up.sh`):
Each runner gains an explicit
`TORRUST_INDEX_CONFIG_OVERRIDE_DATABASE__CONNECT_URL`
matching its driver and seeded path — the sqlite
variants point at
`/var/lib/torrust/index/database/e2e_testing_sqlite3.db?mode=rwc`,
the mysql variant at
`mysql://root:root_secret_password@mysql:3306/torrust_index_e2e_testing`.
This unblocks the same Phase 5 schema requirement at
the e2e harness boundary, mirroring the application
override the operator-facing path now demands.
Docs:
- ADR-T-009 §D1 status flips to "Landed 2026-04-24
via Phase 8" with a backlink to the implementation
plan.
- The implementation plan marks Phase 8 as Done, adds
the missing `USER_ID` validator entry to the
Makefile snippet (with the rationale comment), and
documents the §8.6 addendum in full so the
earlier-phase touch-ups are traceable from a single
place. Phase 9 gains §9.1.3, which calls out the
pending CI lint that re-runs the
`mailcatcher|MAILER|SMTP|smtp_` audit against
`compose.yaml` only (matching the override is a
false positive) so the audit trail is enforced
rather than reviewer-discretionary.
- `docs/containers.md` gains a "Compose Split"
section pointing at both files, documents
`make up-dev` / `make up-prod` and the validator's
required-env list, and explains the defence-in-depth
relationship with the in-container config probe. The
Phase 7 "compose split will provide …" forward
reference is replaced with a real link to
`compose.override.yaml`.
Refs: ADR-T-009 §D1, §8.
…-009 Phase 9)
Land Phase 9 of ADR-T-009 by wiring the documentation,
audit-trail, and CI guards that turn the preceding eight
phases into a defended baseline. With this commit the ADR
flips from "Proposed" to "Implemented (Phases 1–9 complete)"
and the implementation-plan tracker marks every phase as
Done. There are no behavioural changes to the running
application; everything here is enforcement scaffolding,
operator-facing prose, and the matching test/sample-file
fallout.
CI guards (`.github/workflows/container.yaml`):
- New `lints` job (Container infra) gating the existing
`test` matrix via `needs: lints`. Three independent
audits run before any image is built:
- `compose-baseline-no-mailcatcher` strips `# …` comments
from `compose.yaml` (preserving line numbers via `awk`
so error annotations point at the real source line)
and greps the result for
`mailcatcher|MAILER|SMTP|smtp_`. The override file is
deliberately excluded — it is *expected* to mention
`mailcatcher` — and the comment-stripping prevents the
explanatory header in `compose.yaml` from tripping the
audit. Re-introducing the dev mail sidecar into the
production-shaped baseline now fails the build with a
`::error file=compose.yaml::…` annotation referencing
ADR-T-009 §8.1 / §9.1.3.
- `su-exec-audit` parses the most recent `SHA-256:` line
from the `## Audit Log` section of
`contrib/dev-tools/su-exec/AUDIT.md` and compares it
to `sha256sum contrib/dev-tools/su-exec/su-exec.c`.
Any drift (file changed without a matching audit
entry, or audit entry missing) fails the build with a
pointer to ADR-T-009 §9.2 and the exact append-required
SHA. Reviewer-discretionary drift is no longer
possible.
- `entry-env-docs` parses the
`# ENTRY_ENV_VARS:` … `# END_ENTRY_ENV_VARS` manifest
block in `share/container/entry_script_sh` and asserts
that every variable listed appears somewhere in
`docs/containers.md`. Also asserts that
`compose.override.yaml` is mentioned. Adding an env
var to the entry script without documenting it now
fails the build (ADR-T-009 §9 / Acceptance Criterion
torrust#7).
Vendoring audit (`contrib/dev-tools/su-exec/AUDIT.md`, new):
- Records provenance (upstream `ncopa/su-exec`, MIT,
vendored 2023-10-14 in repo commit `1f5351db`),
initial SHA-256, and the choice rationale against
`gosu`/`setpriv`/`su`/`runuser` (binary size on
distroless, dependency footprint, and PID-1/signal
preservation respectively).
- Documents the two re-audit triggers — file-change
(automated, the CI guard above) and CVE (manual) —
and explicitly rejects a calendar trigger per
ADR-T-009 §D8: static frozen C with a finite review
surface does not decay with time.
- Opens the append-only `## Audit Log` with the
2026-04-21 initial-audit entry: full read-through
against upstream, byte-identical, no shell/network/
signal/env surprises, no findings. The entry ends
with the structured `SHA-256:` line the CI guard
parses.
Entry-script manifest
(`share/container/entry_script_sh`):
- New canonical `# ENTRY_ENV_VARS:` … `# END_ENTRY_ENV_VARS`
block at the top of the script enumerates every env
var the script consults — including the dynamically
constructed `AUTH__{PRIVATE,PUBLIC}_KEY_{PEM,PATH}`
names that a naive grep of `${...}` expansions would
miss. This is the single source of truth the CI lint
reads; the docs/env-table is the consumer.
Shipped-config consolidation
(`share/default/config/`):
- Drop the `.{mysql,sqlite3}.toml` suffix split for the
container-shipped defaults now that Phase 5 made
`database.connect_url` mandatory (the file no longer
encodes a driver choice). Two pairs collapse into
single files:
- `index.container.{mysql,sqlite3}.toml` →
`index.container.toml` (the `mysql` variant was
already the canonical content; the `sqlite3` variant
is removed).
- `index.public.e2e.container.{mysql,sqlite3}.toml` →
`index.public.e2e.container.toml` (kept the more
explicit `listed = false` / `private = false`
settings from the sqlite variant for clarity).
- `index.private.e2e.container.sqlite3.toml` and
`index.development.sqlite3.toml` lose their now-empty
`[auth]` stanza (auth keys are never seeded from the
shipped TOML — they come from the entry script's
defaults or operator overrides). The development
template additionally moves `[net.tls]` above
`[tracker]` to keep all top-level sections grouped.
- `packages/index-config/tests/shipped_samples.rs`
follows the rename in lockstep so the samples remain
exercised by the loader's parse + validate pipeline.
- All four e2e runner scripts under
`contrib/dev-tools/container/e2e/{mysql,sqlite/mode/public}/`
point at the unified file names.
Operator docs:
- `docs/containers.md` gains:
- a "Test-Stage Gate" section documenting the
Containerfile's `test` / `test_debug` build-time
test stages, the deliberate absence of a
`--skip-tests` escape hatch, and the supported
`#[ignore]`-then-fix escalation path. Backlinked
to ADR-T-009 §D9.
- a `TZ` row in the env-var table covering the
Containerfile's pass-through default of `Etc/UTC`.
- a pointer at the entry-script manifest block,
explaining the CI contract that keeps the table
and the manifest in sync.
- prose updates reflecting the single
`index.container.toml` default (no more
`{sqlite3,mysql}` split).
- `README.md`:
- Container quickstart blocks now show the two
mandatory overrides (`TRACKER__TOKEN`,
`DATABASE__CONNECT_URL`) so a copy-paste
`docker run` / `podman run` succeeds against the
Phase 5 schema. A leading note explains *why* a
bare invocation now fails, with a link to
ADR-T-009 §D2.
- New "Compose (development sandbox)" subsection
introducing `make up-dev` / `make up-prod` and
pointing at the Compose Split docs section.
- The ADR-T-009 entry in the "Architecture Decision
Records" list is rewritten to match the
actually-landed scope (runtime base split, three
helper crates, mandatory schema fields, Compose
split, vendored-`su-exec` audit) rather than the
earlier "hardening" framing.
- `CHANGELOG.md`:
- Expands the ADR-T-009 entry under `Added` to
summarise Phases 1–9 in operator-visible terms
(runtime image split with the
`0700`/`0500 root:root` posture on the lean
toolset, three extracted helper crates including
their renames from the old `health_check` /
`torrust-generate-auth-keypair` names, the
Compose split with the two `make` wrappers, and
the new vendoring audit + CI guard).
- Three new **BREAKING** entries under `Changed`
spelling out the operator-facing migrations:
mandatory `database.connect_url` /
`tracker.token` (with the precise override env-var
names), the PEM/PATH mutual-exclusion contract on
auth keys (D3), and the narrowed scope of
`TORRUST_INDEX_DATABASE_DRIVER` as a first-boot
TOML selector only (no longer a runtime database
dispatcher).
ADR & implementation plan:
- `adr/009-container-infrastructure-refactor.md`
status flips from "Proposed" to "Implemented
(Phases 1–9 complete)".
- `adr/009-implementation-plan.md` marks Phase 9 as
Done, and corrects the §9 changelog-prose template:
the `[net.tsl]` → `[net.tls]` wire-key rename was
already logged at Phase 3 (config-crate extraction)
and must not be re-stated under the Phase 9 summary
bullet. The plan now calls this out inline so a
future re-reader does not duplicate the entry.
Repo housekeeping:
- `AGENTS.md` reorders the helper-crate enumeration
(`index-cli-common` to the end of the list) so the
three Phase-2/6 helpers cluster together. No
behavioural meaning — the share-the-`T-`-prefix rule
is unchanged.
Refs: ADR-T-009 §D8, §D9, §9.
Now that ADR-T-009 Phases 1–9 have all landed, fold the implementation plan into the ADR proper and retire the standalone plan document. The ADR now carries a single decision-log section (§D1–§D8) that supersedes the dual "ADR §N / plan §M" numbering the working docs grew during the phased rollout. - Delete adr/009-implementation-plan.md (3.2k lines); the surviving decisions live in the consolidated ADR. - Rework adr/009-container-infrastructure-refactor.md: drop the "Implementation plan" pointer, mark status simply "Implemented", and re-anchor every internal reference onto the §D-series IDs. - Re-point all stale §N / §9.x / §7.3 cross-references in the CI workflow, Containerfile, entry script, su-exec AUDIT, the containers guide, and the config-probe rustdoc onto their §D equivalents (or onto Acceptance Criterion #N where the plan numbering had no ADR analogue). - Rewrite the CHANGELOG [Unreleased] block as a release-shaped narrative (Highlights / Breaking changes / Added / Changed / Removed / Fixed / Security) rather than a chronological dump of phase bullets. - Update src/lib.rs rustdoc and README.md to reflect the §D2 decision: tracker.token and database.connect_url are now mandatory schema fields with no shipped defaults, so the docker quick-start now shows the required env-var overrides and the Configuration section names both fields as mandatory. Also correct TORRUST_INDEX_CONFIG_PATH → TORRUST_INDEX_CONFIG_TOML_PATH and link the new container guide / ADR alongside the legacy docker docs pointer. - Mention the standalone torrust-index-config crate in the Configuration section so readers know where the parsing surface lives. No code or runtime behaviour changes; this is purely the post-merge documentation tidy-up for ADR-T-009.
Bring the SQLite end-to-end runner up against
`podman 5.8.2` / `podman-compose 1.5.0` without regressing
the Docker-Compose-v2 path. The five issues fixed here all
surface only when the harness is driven by something other
than the upstream Docker daemon, which is why they slipped
past Phase 8's bring-up against the operator workflow.
Containerfile:
- Copy `entry_script_lib_sh` into the release runtime base
alongside `entry.sh`. The file is sourced unconditionally
at start-up; without it the container exits immediately
with `can't open /usr/local/lib/torrust/entry_script_lib_sh`.
The debug runtime base already pulled it directly from the
build context — the release base routed everything via the
`runtime_assets` collector and this one path was missed.
E2E `e2e-env-up.sh` (public + private SQLite modes):
- Export `BUILDAH_FORMAT=docker` so the Containerfile's
`HEALTHCHECK` directive survives the build under podman.
The default OCI format silently drops it, leaving
`torrust-index-1` permanently un-healthy and tripping
the wait-for-healthy gate downstream.
- Pre-create the `./storage/{index,tracker}/{lib/database,log,etc}`
bind-mount sources. Docker auto-creates missing host paths;
podman does not, and combined with `:Z` SELinux relabel the
failure mode is an opaque `getxattr ... no such file or
directory` rather than the friendlier "bind source path does
not exist".
- Replace `docker compose up --pull always` with a standalone
`docker compose pull || true` followed by a plain `up`.
`podman-compose` rejects the Compose-v2 `--pull=always`
syntax (it accepts a boolean `--pull` paired with
`--pull-always`); `compose pull` works under both. The
`|| true` keeps the script resilient to transient registry
hiccups and to images that only exist locally
(e.g. `localhost/torrust_index`).
E2E `run-e2e-tests.sh`:
- Guard against being run outside the repo root: require a
git checkout, refuse if `pwd` is not the toplevel, and
identify the repo by `Cargo.toml` package name rather than
remote URL so forks and renamed clones still work. The
script does destructive things (`rm -rf ./storage`,
overwrites `.env`) that are only safe inside the project
root.
- Prepend `$HOME/.cargo/bin` to `PATH` so rustup-managed
toolchains are visible regardless of the invoking shell's
setup. `~/.cargo/env` is only sourced in interactive login
shells, which CI runners and `bash -c` invocations are not.
- Wipe `./storage` before each run, routing through
`podman unshare rm -rf` when the active engine is podman.
Files written from inside a rootless container are owned by
a sub-UID from `/etc/subuid`, which a host-side `rm -rf`
cannot remove. The engine is detected via
`docker --version | grep -qi podman` so the `podman-docker`
shim is handled correctly.
- Skip `cargo install imdl` when `imdl` is already on PATH,
and pass `--locked` when it is not — the install dominated
the script's wall-clock time on every re-run.
- Probe whether containers come up named with `-` (Compose v2)
or `_` (podman-compose v1.x) and compose the
`wait_for_container_to_be_healthy.sh` arguments from the
detected separator. The probe runs once per phase and
defaults to `-` when nothing matches.
`wait_for_container_to_be_healthy.sh`:
- Treat a `null` `.State.Health` (image declares no
`HEALTHCHECK`) the same as `{}` and fall back to the plain
`.State.Status == "running"` check. Without this the loop
spins until the retry budget runs out for any image without
a healthcheck — common for upstream images such as
`docker.io/torrust/tracker`, and for any locally-built image
whose layers ended up in OCI format.
No changes to the application or to the production compose
flow; this is purely the e2e-harness portability fix that
unblocks the SQLite suite under podman.
…-T-009)
Add the missing tests that were flagged when sweeping the
helper packages for `pub` / `pub(crate)` items with no direct
exercise. Pure additions — no library code is touched beyond
wiring a `mod tests;` into `index-cli-common`'s `lib.rs`.
`index-cli-common`:
- New `src/tests/mod.rs` covering the `emit` happy path,
writer-error and serialisation-error branches via a
`FailingWriter` and an `AlwaysFails` `Serialize` impl, plus
the `BaseArgs` clap-flatten contract (default + `--debug`).
`refuse_if_stdout_is_tty` and `init_json_tracing` are
deliberately left to per-binary integration tests because
they mutate global process state.
- New `tests/public_api.rs` pinning the externally-visible
surface (`emit`, `BaseArgs`) the way every helper binary
consumes it, including clap's `exit_code() == 2` mapping
for unknown flags that ADR-T-009 §6.1 relies on.
`index-config`:
- New `src/tests/info.rs` for `Info::new` / `Info::from_env`
/ `Tls::default`. Tests serialise on a file-local mutex
around `TORRUST_INDEX_CONFIG_TOML*` and pin the
empty-env-var-collapses-to-unset behaviour that keeps a
bare `${VAR}` in compose from clobbering the on-disk
config.
- New `src/tests/tracker.rs` covering
`Tracker::override_tracker_api_token` (replace +
idempotent), the `ApiToken` accessors (`Display`,
`as_bytes`, `is_empty` via the deserialiser-bypass path
that's the only way to materialise an empty token), and
the schema-level `Tracker` defaults.
`index-config-probe`:
- Extend `src/tests/mod.rs` with `Display` and
`std::error::Error` checks for `ProbeError`. The probe
binary logs `error = %e`, so `Display` is the
user-visible diagnostic and worth pinning.
`index-entry-script`:
- New `tests/run_sh.rs` giving the no-args `run_sh` sibling
its own coverage: trivial-snippet success, library
pre-sourcing (via `validate_auth_keys`), `set -u` aborting
on unbound variables, and `lib_path()` stability across
calls. The other integration files use `run_sh_with_args`
exclusively, so without this file `run_sh` carried zero
coverage despite being part of the public surface.
`index-health-check`:
- Extend `src/tests/mod.rs` with three error-path tests:
`HealthCheckError`'s `Display` / `Debug`, the
unsupported-scheme rejection (the probe is HTTP-only by
design), and the resolver-error branch via an
RFC-6761 `.invalid` host.
A few dozen new test cases in total, lifting these helper
crates to the same level of public-API coverage that the
ADR-T-009 acceptance criteria already required of the
runtime path.
peer-cat
force-pushed
the
feat/rework_container_system
branch
from
c1a5f2a to
0036fc0
Compare
There was a problem hiding this comment.
Pull request overview
Implements ADR-T-009 end-to-end by refactoring the container/deployment surface: configuration becomes schema-enforced (no shipped secrets), helper binaries are extracted into standalone workspace crates, and Compose/CI are restructured to create an auditable, production-shaped baseline.
Changes:
- Extracts configuration schema/loader and several container helper binaries into new workspace crates, updating the main crate to consume/re-export them.
- Hardens container + operator workflows: mandatory
tracker.token/database.connect_url, new entry-script shell library + test harness, audited vendoredsu-exec, and new container CI lints/guards. - Splits Compose into
compose.yaml(prod-shaped) andcompose.override.yaml(dev sandbox) and addsmake up-dev/make up-prodwrappers.
Reviewed changes
Copilot reviewed 100 out of 111 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/environments/isolated.rs | Switch test settings fixture to shared placeholder settings. |
| tests/e2e/web/api/v1/contexts/settings/contract.rs | Normalize env-specific fields before asserting settings contract. |
| tests/e2e/config.rs | Make e2e config test hermetic via figment::Jail and env overrides for mandatory fields. |
| src/web/api/server/v1/extractors/require_permission.rs | Preserve DB error category; avoid masking outages as 404. |
| src/web/api/server/v1/contexts/torrent/handlers.rs | Fix doc comment to reference AddTorrentRequest. |
| src/web/api/server/v1/contexts/settings/mod.rs | Update example settings JSON key from tsl to tls. |
| src/web/api/server/mod.rs | Rename Tsl→Tls, re-export DynError, and update TLS wiring/comments. |
| src/web/api/mod.rs | Propagate Tls rename through API start interface. |
| src/tests/web/require_permission.rs | Use Configuration::for_tests() instead of default config. |
| src/tests/services/settings.rs | Use Configuration::for_tests() for settings service tests. |
| src/tests/jwt.rs | Use Configuration::for_tests() for JWT tests. |
| src/tests/config/mod.rs | Add env-scrubbing helper and shift tests to for_tests() / placeholder-based baselines. |
| src/tests/bootstrap/config.rs | Inject mandatory config overrides into bootstrap tests. |
| src/services/authorization.rs | Move permission value types to config crate; re-export from service layer. |
| src/lib.rs | Update crate docs for new container requirements, Compose split, and config env var name. |
| src/jwt.rs | Update docs for renamed/extracted auth-keypair helper and JSON output contract. |
| src/bootstrap/config.rs | Re-export default config TOML path from config crate. |
| src/bin/health_check.rs | Remove old reqwest/tokio-based health check binary (moved to new crate). |
| src/bin/generate_auth_keypair.rs | Remove old auth keypair generator binary (moved to new crate). |
| src/app.rs | Rename settings field net.tsl→net.tls. |
| share/default/config/tracker.public.e2e.container.sqlite3.toml | Remove shipped tracker token from e2e tracker sample. |
| share/default/config/tracker.private.e2e.container.sqlite3.toml | Remove shipped tracker token from e2e tracker sample. |
| share/default/config/index.public.e2e.container.toml | Strip credentials and dev-only sections from shipped container e2e config. |
| share/default/config/index.public.e2e.container.sqlite3.toml | Remove obsolete sqlite3-specific e2e container sample. |
| share/default/config/index.private.e2e.container.sqlite3.toml | Strip shipped token/connect_url and other dev-only sections. |
| share/default/config/index.development.sqlite3.toml | Remove shipped secrets; rename [net.tsl] to [net.tls]. |
| share/default/config/index.container.toml | Strip shipped credentials and dev-only mail/auth path defaults. |
| share/default/config/index.container.sqlite3.toml | Remove obsolete sqlite3 container sample with shipped secrets. |
| share/container/entry_script_lib_sh | Add pure POSIX sh helper library for entry script invariants and sqlite seeding. |
| packages/index-health-check/tests/health_check.rs | Add integration tests for std::net-based health check logic. |
| packages/index-health-check/src/lib.rs | New std-only health check library with Happy Eyeballs connect logic. |
| packages/index-health-check/src/bin/torrust-index-health-check.rs | New helper binary emitting JSON and using shared CLI scaffolding. |
| packages/index-health-check/Cargo.toml | New crate manifest for health-check helper. |
| packages/index-entry-script/tests/validate_auth_keys.rs | Host-side integration tests for shell auth-key validation helper. |
| packages/index-entry-script/tests/seed_sqlite.rs | Host-side integration tests for shell sqlite seeding helper. |
| packages/index-entry-script/tests/run_sh.rs | Smoke tests for the shell harness helpers. |
| packages/index-entry-script/src/lib.rs | New harness for sourcing/execing embedded shell library via sh. |
| packages/index-entry-script/Cargo.toml | New crate manifest for entry-script test harness. |
| packages/index-config/tests/shipped_samples.rs | Validate shipped index.*.toml samples are TOML-valid and credential-free. |
| packages/index-config/tests/round_trip.rs | Round-trip tests for placeholder settings through TOML/JSON APIs. |
| packages/index-config/tests/permission_overrides.rs | Integration tests for permissions override parsing and error cases. |
| packages/index-config/src/validator.rs | Add schema validator trait and validation error type. |
| packages/index-config/src/v2/website.rs | New v2 website/legal content config types with defaults. |
| packages/index-config/src/v2/tracker_statistics_importer.rs | New tracker statistics importer config type with defaults. |
| packages/index-config/src/v2/tracker.rs | Make tracker token mandatory; add ApiToken::is_empty for container boundary checks. |
| packages/index-config/src/v2/registration.rs | New registration/email verification config types. |
| packages/index-config/src/v2/permissions.rs | Wire permissions overrides to config-crate value types. |
| packages/index-config/src/v2/net.rs | Rename network TLS field and type import. |
| packages/index-config/src/v2/mod.rs | Make [tracker]/[database] sections mandatory; remove Settings::default. |
| packages/index-config/src/v2/mail.rs | New mail/SMTP config types with defaults. |
| packages/index-config/src/v2/logging.rs | New logging threshold enum and conversion to LevelFilter. |
| packages/index-config/src/v2/image_cache.rs | New image cache quota/timeout config with defaults. |
| packages/index-config/src/v2/database.rs | Make database.connect_url mandatory; remove default connect URL. |
| packages/index-config/src/v2/auth.rs | New auth/JWT key config resolution helpers and defaults. |
| packages/index-config/src/v2/api.rs | New API paging config with defaults. |
| packages/index-config/src/tests/tracker.rs | Unit tests for tracker token override and ApiToken accessors. |
| packages/index-config/src/tests/redaction.rs | Unit tests for secret redaction behavior. |
| packages/index-config/src/tests/quirks.rs | Unit tests for edge cases (TLS empty strings, IPv6, unicode tokens, etc.). |
| packages/index-config/src/tests/permissions.rs | Unit tests for permission value types and round-trips. |
| packages/index-config/src/tests/mod.rs | Test module index + shared helpers for hermetic config tests. |
| packages/index-config/src/tests/metadata.rs | Unit tests for metadata/version serialization and display. |
| packages/index-config/src/tests/loader.rs | Unit tests for load_settings error cases and mandatory-field behavior. |
| packages/index-config/src/tests/info.rs | Env-var resolution tests for Info constructors with serialization guard. |
| packages/index-config/src/test_helpers.rs | Shared placeholder TOML/settings fixture (replaces Settings::default() usage). |
| packages/index-config/src/permissions.rs | Move permission value types into config crate. |
| packages/index-config/Cargo.toml | New config crate manifest and dependencies. |
| packages/index-config-probe/tests/binary.rs | Contract tests for config-probe exit codes and JSON shape without spawning binary. |
| packages/index-config-probe/src/tests/mod.rs | Unit tests for database/auth probing logic and error messages. |
| packages/index-config-probe/src/lib.rs | New library for emitting typed, JSON-serializable probe output. |
| packages/index-config-probe/src/bin/torrust-index-config-probe.rs | New helper binary mapping loader/probe outcomes to documented exit codes. |
| packages/index-config-probe/Cargo.toml | New config-probe crate manifest and deps. |
| packages/index-cli-common/tests/public_api.rs | Integration tests for public CLI scaffolding API (emit, BaseArgs). |
| packages/index-cli-common/src/tests/mod.rs | Unit tests for JSON emit behavior and clap parsing of BaseArgs. |
| packages/index-cli-common/src/lib.rs | New shared helper crate for JSON emit, tracing init, and TTY refusal. |
| packages/index-cli-common/Cargo.toml | New cli-common crate manifest. |
| packages/index-auth-keypair/tests/keypair_generation.rs | Integration tests for keypair generation output correctness and uniqueness. |
| packages/index-auth-keypair/src/tests/mod.rs | Unit tests for JSON and PEM parsing of generated keys. |
| packages/index-auth-keypair/src/lib.rs | New RSA keypair generator library producing JSON payload. |
| packages/index-auth-keypair/src/bin/torrust-index-auth-keypair.rs | New helper binary emitting JSON and using shared CLI scaffolding. |
| packages/index-auth-keypair/Cargo.toml | New auth-keypair crate manifest. |
| contrib/dev-tools/su-exec/AUDIT.md | Add provenance/audit log for vendored su-exec.c and CI-verifiable SHA. |
| contrib/dev-tools/container/run.sh | Remove legacy ad-hoc container run script. |
| contrib/dev-tools/container/functions/wait_for_container_to_be_healthy.sh | Add fallback for images lacking HEALTHCHECK; avoid infinite wait. |
| contrib/dev-tools/container/e2e/sqlite/run-e2e-tests.sh | Harden e2e runner (repo-root guard, rootless podman storage cleanup, tool detection, env overrides). |
| contrib/dev-tools/container/e2e/sqlite/mode/public/e2e-env-up.sh | Podman portability fixes and inject mandatory config overrides for container run. |
| contrib/dev-tools/container/e2e/sqlite/mode/public/e2e-env-down.sh | Update env var to reflect renamed e2e TOML sample. |
| contrib/dev-tools/container/e2e/sqlite/mode/private/e2e-env-up.sh | Podman portability fixes and inject mandatory config overrides for container run. |
| contrib/dev-tools/container/e2e/mysql/run-e2e-tests.sh | Update e2e runner to inject mandatory config overrides for host-side tests. |
| contrib/dev-tools/container/e2e/mysql/e2e-env-up.sh | Update mysql e2e env to new TOML sample and inject mandatory connect_url override. |
| contrib/dev-tools/container/e2e/mysql/e2e-env-down.sh | Update env var to reflect renamed e2e TOML sample. |
| contrib/dev-tools/container/build.sh | Remove legacy ad-hoc container build script. |
| compose.yaml | Split into production-shaped baseline: no dev sidecars, no default creds, loopback-bound ports, long-form depends_on. |
| compose.override.yaml | Add dev-only overrides: mailcatcher, tty, and permissive credential defaults. |
| adr/007-jwt-system-refactor.md | Update ADR to reflect renamed/extracted auth-keypair helper and JSON output. |
| README.md | Update container quickstart and document new Compose split + mandatory overrides. |
| Makefile | Add up-dev / up-prod targets and production env validation. |
| Cargo.toml | Add new workspace members and depend on extracted torrust-index-config crate. |
| Cargo.lock | Lockfile updates for new workspace crates and dependency changes. |
| AGENTS.md | Document commit-message guidance, POSIX path handling, helper-crate ADR prefix convention, and atomic file replacement. |
| .github/workflows/container.yaml | Add container-infra lints: compose baseline audit, su-exec SHA audit, entry env var docs check. |
| .containerignore | Exclude docs/adr/etc from container build context. |
Comments suppressed due to low confidence (1)
packages/index-config/src/v2/net.rs:12
- Spelling/grammar issues in the module docs (e.g. "The the" and "por"). Consider correcting to "The base URL" and "port" to keep the config crate docs clean.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Three independent follow-ups noticed while sweeping the helper packages. None of them change observable behaviour on the happy path; they each close a small gap that was easy to miss. `index-config-probe`: - Initialise JSON tracing *before* `refuse_if_stdout_is_tty`. The TTY refusal emits a structured log line on the way out, so it needs the subscriber installed first; otherwise the diagnostic is silently dropped and the operator only sees the non-zero exit code with no explanation. `index-entry-script`: - Run the `run_sh_with_args` snippet under `set -eu`, matching the discipline `run_sh` already enforces. Without it the args variant would happily mask unbound variables and unchecked command failures, which defeats the point of having the entry script's own `set -eu` contract. `index-health-check`: - In the `serve_once` test helper, `.expect()` the request read instead of binding the `Result` to `_n` and dropping it. The test only needs to unblock the client so it sees the canned response, but a real IO error here was previously swallowed — now it surfaces loudly with a message pointing at the helper.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lands ADR-T-009 end to end — Phases 1–9 — turning the container surface from "a single multi-stage
Dockerfileplus a free-form entry script" into a structurally split, audit-tracked, CI-guarded baseline. Fourteen commits on top of the develop HEAD, every commit references back to a numbered decision (D1–D9) in the consolidated ADR.Why
The pre-refactor container surface had grown nine independent pain points (R1–R10 in the ADR appendix), among them:
API_PORT,IMPORTER_API_PORT) as build-timeARGs.reqwest,tokio,hyper,rustls, andnative-tlsinto anything that linked them.mailcatcher,tty: true) into a single file, and bound dev ports on every host interface.set -x'd (leaking creds), had a buggyUSER_IDshort-circuit, and re-parsed TOML in shell to dispatch on the database driver.su-exechad no audit record, no provenance trail, and no drift detection.This PR lands a structural fix for each.
Highlights
runtime_releaseongcr.io/distroless/cc-debian13with a single root-only/bin/busybox(0700 root:root) and a curated applet set, plus aruntime_debugon:debugfor operator escape-hatches. Both share aruntime_assetscollector and apreflight_gatethat aggregates donor-validation stages so future base reshuffles fail at build time. (D4)packages/index-cli-common— shared P9 scaffolding (refuse_if_stdout_is_tty,init_json_tracing,emit,BaseArgs).packages/index-health-check—torrust-index-health-check(renamed fromhealth_check), rewritten onstd::net::TcpStreamwith Happy Eyeballs.packages/index-auth-keypair—torrust-index-auth-keypair(renamed fromtorrust-generate-auth-keypair), now emits{"private_key_pem": "...", "public_key_pem": "..."}instead of concatenated PEM blocks. (D5)packages/index-confignow owns the v2 schema, validator, loader, and permission value types.cargo tree -p torrust-index-config -e normalconfirmstokio,reqwest,sqlx,hyper,rustls,native-tls, andopensslare all absent. The root crate keeps a thinpub use torrust_index_config::*;shim so every existing call site compiles unchanged.packages/index-config-probe—torrust-index-config-probe— emits a single line of JSON withdatabase.driver,database.path, andauth.{private,public}_key.{pem_set,path_set,source,path}, giving the entry script a stable typed interface to dispatch on instead of re-parsing TOML in shell. (D3, P9)set -eu, sources pure helpers from entry_script_lib_sh, enforces the auth-key invariants (PEM/PATH mutual exclusion within a key, both-or-neither across the pair, single mechanism across the pair), and dispatches database seeding from the probe's typeddatabase.driverfield. A new test-only cratepackages/index-entry-scriptdrives the shell library fromcargo testviashsubprocess. (D3)USER_ID >= 1000rule rejected legitimate rootless-Podman / low-UID-CI / BSD configurations without stating its intent. The property the script actually enforces — do not run as root — is now the literal check: "non-negative integer, not 0". (D7, breaking for anyone relying on the old rule.)mailcatcher, notty, ports bound to127.0.0.1except:3001, every credential as a bare${VAR}with no default); compose.override.yaml re-attaches the dev sidecars; a top-level Makefile wraps both flows asmake up-dev/make up-prod. The prod recipe runs a_validate-prod-envPOSIX-shvalidator first as defence in depth. (D1)ncopa/su-exec, MIT, vendored 2023-10-14), the choice rationale againstgosu/setpriv/su/runuser, the file-change and CVE re-audit triggers (no calendar trigger, per D8), and an append-only## Audit Log. (D8)testmatrix:compose-baseline-no-mailcatcher— re-introducing the dev mail sidecar into the production-shaped baseline fails the build.su-exec-audit— drift between the file's SHA-256 and the latest## Audit Logentry fails the build.entry-env-docs— every var listed in the entry script's# ENTRY_ENV_VARS:manifest must appear in containers.md. (D9)ARG API_PORT/ARG IMPORTER_API_PORTare gone (runtimeENVdefaults remain). adr and docs are excluded from the build context. Helper binaries are tightened to0500 root:root; the application binary keeps0755. (D6, D9)Breaking changes
[net.tsl](typo)[net.tls]— wire key, type, and JSON API response all renamed in lockstep. No alias.tracker.tokenTORRUST_INDEX_CONFIG_OVERRIDE_TRACKER__TOKEN.database.connect_urlTORRUST_INDEX_CONFIG_OVERRIDE_DATABASE__CONNECT_URL.TORRUST_INDEX_DATABASE_DRIVERindex.tomlis installed at/etc/torrust/index/index.toml. Runtime decisions come fromconnect_url.USER_IDguard>= 1000health_check,torrust-generate-auth-keypairtorrust-index-health-check,torrust-index-auth-keypairtorrust-index-auth-keypairoutputjq -r .private_key_pem1for the keypair helper,2for healthcheck2via the sharedrefuse_if_stdout_is_tty.Operator migration
The
# Containerquickstart blocks in README.md and containers.md now show the two mandatory overrides. The minimum invocation is:For Podman:
--format dockeris required so theHEALTHCHECKdirective survives the build (OCI manifests have no field for it). The e2e harness handles this automatically viaBUILDAH_FORMAT=docker.Phase map
01e7c7097e58e98308247457b6f0d0d8e7eindex-configextracted;Tsl→Tls.3818c20runtime_release/runtime_debug; D7USER_IDrule.8b4bfebtracker.token/database.connect_urlmandatory.1f473dfindex-config-probe,Info::from_env.a2a8527set -eu, sourced library,index-entry-script.d8b6ad257b76e0AUDIT.md, env-var manifest.b79e32d009-implementation-plan.mdretired; refs re-anchored on §D-series.362996cpodman 5.8.2/podman-compose 1.5.0.0036fc0pub/pub(crate)items in the new crates.Tests
A few hundred new tests across the four new crates plus the entry-script harness; the workspace suite (
cargo test --workspace --all-targets --all-features) passes in bothdevandreleaseprofiles, with and without default features. Doc-tests andcargo docboth build.References
[Unreleased].