From de7e5b60bfd27d8fe37c99d55779964de672fe4d Mon Sep 17 00:00:00 2001 From: Xuehao Xu Date: Wed, 15 Jul 2026 10:01:27 +0800 Subject: [PATCH 1/5] docs: propose declarative sandbox features --- docs/community/oseps.md | 1 + oseps/0015-declarative-sandbox-features.md | 952 +++++++++++++++++++++ oseps/README.md | 3 + 3 files changed, 956 insertions(+) create mode 100644 oseps/0015-declarative-sandbox-features.md diff --git a/docs/community/oseps.md b/docs/community/oseps.md index 1460a7d40..9125b85cc 100644 --- a/docs/community/oseps.md +++ b/docs/community/oseps.md @@ -27,6 +27,7 @@ See the [OSEP contributing guide](https://github.com/opensandbox-group/OpenSandb | [OSEP-0012](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0012-credential-vault.md) | Credential Vault and Credential Proxy | implemented | 2026-06-23 | | [OSEP-0013](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0013-isolated-execution-api.md) | Isolated Execution API | implementing | 2026-06-23 | | [OSEP-0014](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0014-multi-tenancy.md) | Multi-Tenancy Support for Kubernetes Runtime | draft | 2026-04-29 | +| [OSEP-0015](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0015-declarative-sandbox-features.md) | Declarative Sandbox Features | draft | 2026-07-14 | ## Status Definitions diff --git a/oseps/0015-declarative-sandbox-features.md b/oseps/0015-declarative-sandbox-features.md new file mode 100644 index 000000000..576cba296 --- /dev/null +++ b/oseps/0015-declarative-sandbox-features.md @@ -0,0 +1,952 @@ +--- +title: Declarative Sandbox Features +authors: + - "@GodBlf" +creation-date: 2026-07-14 +last-updated: 2026-07-14 +status: draft +--- + +# OSEP-0015: Declarative Sandbox Features + + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Requirements](#requirements) +- [Proposal](#proposal) + - [Public API](#public-api) + - [Administrator-Controlled Feature Registry](#administrator-controlled-feature-registry) + - [Feature OCI Artifact Contract](#feature-oci-artifact-contract) + - [Docker Provisioning](#docker-provisioning) + - [Kubernetes Provisioning](#kubernetes-provisioning) + - [Compatibility Matrix](#compatibility-matrix) + - [Security Boundary](#security-boundary) + - [Notes/Constraints/Caveats](#notesconstraintscaveats) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Name, Version, and Platform Resolution](#name-version-and-platform-resolution) + - [Activation and Environment Conflict Rules](#activation-and-environment-conflict-rules) + - [Persistence and Read-After-Create Consistency](#persistence-and-read-after-create-consistency) + - [Failure Semantics](#failure-semantics) + - [Implementation Phases](#implementation-phases) +- [Test Plan](#test-plan) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed](#infrastructure-needed) +- [Upgrade & Migration Strategy](#upgrade--migration-strategy) + + +## Summary + +Add a first-class, declarative `features` field to sandbox creation so callers +can request administrator-approved, versioned tools such as Git or uv without +building a custom sandbox image or running a package manager during creation. +The server resolves each request through an administrator-controlled registry +to a digest-pinned, platform-specific OCI feature image, provisions its +self-contained payload before execd and the user entrypoint start, and returns +the immutable resolution as `resolvedFeatures`. + +This proposal addresses the additive half of +[issue #1289](https://github.com/opensandbox-group/OpenSandbox/issues/1289). +It deliberately does not define `deny_tools` or a command blacklist. Hiding or +rejecting command names is not an enforceable security boundary; a separate +security/execution-profile OSEP is required for tool restriction. + +## Motivation + +Today a user who needs Git, uv, Node.js, Playwright, or another tool in a +sandbox must choose one of three imperative paths: + +1. Build and distribute a custom sandbox base image. +2. Change `entrypoint` or environment variables to bootstrap tools; +3. Call execd after creation and run a base-image package manager or download + script. + +These paths make otherwise identical sandbox requests depend on mutable image +tags, Linux distribution package managers, internet availability, and repeated +downloads. They are also difficult for operators to audit because the server +does not resolve or record the installed toolchain. + +The lifecycle API already states that `extensions` is for provider-specific, +experimental, or temporary behavior and that standard parameters should become +core API fields. Declarative features are a cross-provider public contract and +therefore must not be encoded in `extensions`. + +### Goals + +- Add an additive, structured `features` field to `CreateSandboxRequest`. +- Let SDKs accept convenient `"name"` and `"name@version"` inputs while keeping + the online JSON representation structured and unambiguous. +- Resolve every requested feature to an exact version, OCI manifest digest, + operating system, and architecture before a successful create returns. +- Restrict feature supply to an administrator-managed allowlist of + digest-pinned OCI images. +- Install self-contained feature payloads before execd and the user entrypoint + start, without depending on software in the sandbox base image. +- Preserve caller order for installation and activation. +- Reuse the existing Docker pre-start execd injection pattern and the existing + Kubernetes execd init-container plus shared `emptyDir` pattern. +- Make create and get responses reproducible by persisting the resolved feature + set in provider metadata. +- Fail clearly for unknown, duplicate, incompatible, conflicting, or unsupported + requests. No requested feature may be silently ignored. +- Keep requests without `features` behaviorally compatible with current + sandbox creation. + +### Non-Goals + +- Client-supplied OCI image or artifact references. +- An online marketplace, public feature catalog service, or independent feature + registry API. +- Running `apt`, `apk`, `yum`, or another base-image package manager while a + sandbox is created. +- Downloading a feature's transitive dependencies from the internet during + sandbox creation. +- Transitive feature dependency declaration or dependency solving. +- Adding, removing, or upgrading a feature after sandbox creation. +- Pool-backed creation through `extensions.poolRef`. +- Snapshot restore or including feature state in a snapshot contract. +- Windows feature images or activation. +- Tool denial, command-name filtering, `PATH` hiding, or a `deny_tools` API. +- Treating a feature as a sandbox isolation or privilege boundary. + +## Requirements + +- `features` MUST be a top-level core field, not an `extensions` convention. +- A wire-level feature request MUST contain a canonical lowercase `name` and MAY + contain an exact `version`. +- Omitting `version` MUST select the administrator registry's configured default + for that feature. It MUST NOT mean `latest` or a version range. +- A successful create response for a non-empty feature request, and subsequent + reads of that Sandbox resource, MUST include the ordered `resolvedFeatures` + with name, exact resolved version, platform-specific OCI digest, OS, and + architecture. +- Callers MUST NOT submit an OCI repository, tag, digest, URL, installer script, + or registry credentials in the public feature request. +- Every registry entry MUST be digest-pinned and MUST declare at least one + supported platform. +- Registry configuration MUST be validated when the server starts. A configured + but invalid registry MUST prevent startup. +- A feature payload MUST be self-contained and relocatable under + `/opt/opensandbox/features//`. +- Feature provisioning MUST NOT require package managers, interpreters, shells, + or libraries from the sandbox base image. +- Feature provisioning MUST NOT download feature dependencies from the internet. + Pulling the administrator-approved, digest-pinned OCI image is the only + required supply-chain fetch. +- Feature activation MUST complete before execd and the user entrypoint start. +- Features MUST install and activate in request order. +- Unknown names, unknown versions, duplicate names, incompatible platforms, and + activation environment conflicts MUST fail creation explicitly. +- A provisioning failure MUST clean up a partially created Docker container or + Kubernetes workload using the provider's existing rollback behavior. +- Version 1 MUST support Linux image-based standard creation through Docker and + all current Kubernetes workload providers. +- Version 1 MUST reject pool mode, snapshot restore, and Windows when `features` + is non-empty. + +## Proposal + +The lifecycle server gains a feature resolver between request validation and +provider provisioning. The resolver accepts only names and exact optional +versions, looks them up in a registry loaded at server startup, computes the +platform candidates, and produces an immutable ordered resolution plan. + +```text +CreateSandboxRequest.features + | + v + validate names/order/duplicates + | + v + administrator feature registry + (name, version, platform -> digest) + | + v + ordered resolution/provisioning plan + / \ + v v + Docker pre-start copy Kubernetes init containers + \ / + v v + activation before execd and entrypoint + | + v + persisted resolvedFeatures -> create/get/diagnostics +``` + +### Public API + +The eventual lifecycle OpenAPI change is additive. `CreateSandboxRequest` gains +the following field: + +```yaml +features: + type: array + maxItems: 32 + items: + $ref: '#/components/schemas/FeatureRequest' + description: >- + Ordered administrator-approved features to provision before sandbox start. +``` + +The request schema is intentionally small: + +```yaml +FeatureRequest: + type: object + additionalProperties: false + required: [name] + properties: + name: + type: string + description: Canonical lowercase feature name. + version: + type: string + description: Optional exact version. Omission selects the registry default. +``` + +For example: + +```yaml +image: + uri: python:3.12 +entrypoint: ["python", "/app/main.py"] +resourceLimits: + cpu: "500m" + memory: "512Mi" +features: + - name: git + version: "2.47.1" + - name: uv +``` + +The online JSON is always an array of objects. String shorthand is not accepted +by the server. Sandbox SDKs MAY expose `"git"` and `"git@2.47.1"` conveniences, +but adapters MUST convert those values to `FeatureRequest` objects before JSON +serialization. A literal `@` in a feature name is not supported, so SDK parsing +uses the final `@` as the version separator and rejects an empty component. + +The CLI implementation in a later phase provides a repeatable option: + +```text +osb sandbox create --feature git --feature git-lfs@3.7.0 ... +``` + +`CreateSandboxResponse` and `Sandbox` gain `resolvedFeatures`: + +```yaml +ResolvedFeature: + type: object + additionalProperties: false + required: [name, version, digest, platform] + properties: + name: + type: string + version: + type: string + digest: + type: string + description: Platform-specific OCI manifest digest, including sha256:. + platform: + $ref: '#/components/schemas/PlatformSpec' +``` + +```json +{ + "resolvedFeatures": [ + { + "name": "git", + "version": "2.47.1", + "digest": "sha256:0123456789abcdef...", + "platform": {"os": "linux", "arch": "amd64"} + }, + { + "name": "uv", + "version": "0.8.3", + "digest": "sha256:fedcba9876543210...", + "platform": {"os": "linux", "arch": "amd64"} + } + ] +} +``` + +The response array preserves request order. It is required on a successful +create with non-empty `features`. It may be omitted for requests without +features and for resources created by older servers. A pending Kubernetes +resource may omit it until node platform resolution completes, but the server +MUST persist it before returning a successful create response. + +Feature names match `^[a-z0-9](?:[a-z0-9._-]*[a-z0-9])?$` and are at most 63 +characters. The server rejects non-canonical case rather than silently folding +it. Versions are exact, case-sensitive registry keys, at most 128 characters, +and match `^[0-9A-Za-z](?:[0-9A-Za-z._+-]*[0-9A-Za-z])?$`. Ranges, wildcards, +`latest`, path separators, and blank versions are invalid. A request may contain +at most 32 features. Repeating a normalized feature name is invalid even when +the two entries request different versions. + +### Administrator-Controlled Feature Registry + +An operator enables features by configuring a server-side registry file path. +The first implementation uses a static TOML file loaded once at process startup; +changing it requires a server restart. This is configuration, not a network +service and not an end-user API. + +An illustrative registry is: + +```toml +schema_version = 1 + +[[features]] +name = "git" +default_version = "2.47.1" + +[[features.versions]] +version = "2.47.1" +image = "registry.example.com/opensandbox/features/git@sha256:" + +[[features.versions.platforms]] +os = "linux" +arch = "amd64" +digest = "sha256:" + +[[features.versions.platforms]] +os = "linux" +arch = "arm64" +digest = "sha256:" + +[[features]] +name = "uv" +default_version = "0.8.3" + +[[features.versions]] +version = "0.8.3" +image = "registry.example.com/opensandbox/features/uv@sha256:" + +[[features.versions.platforms]] +os = "linux" +arch = "amd64" +digest = "sha256:" +``` + +`image` is always a complete OCI reference with an `@sha256:` digest. It may +identify a multi-platform OCI index or a single-platform manifest. Every +declared platform also pins the child manifest digest returned in +`resolvedFeatures`. For a single-platform image, the image and child digest may +be the same. + +Server startup validation MUST reject: + +- an unknown registry schema version; +- an invalid or non-canonical feature name; +- duplicate feature names; +- a feature with no versions; +- an invalid or duplicate version within one feature; +- a missing default version or a default that does not name exactly one entry; +- an image reference that uses a tag, omits a digest, or uses a digest algorithm + not supported by the implementation; +- an empty platform list; +- duplicate or unnormalized OS/architecture pairs; +- a platform declaration without a pinned child digest; +- a non-Linux platform in the version 1 registry; +- an OCI index whose descriptors do not contain the declared child digest for + the declared platform; and +- artifact metadata whose feature name, version, or platform disagrees with the + registry entry. + +OCI registry credentials, mirrors, trust roots, and pull policy are operator +configuration. They never appear in `FeatureRequest`. A configured registry +that cannot be validated is a startup error so the server does not accept +requests against a partially trusted catalog. If no feature registry path is +configured, ordinary requests continue to work and non-empty `features` +requests fail with a clear `FEATURE_REGISTRY_NOT_CONFIGURED` error. + +The registry is the allowlist. Digest pinning makes resolution immutable and +cacheable, but does not by itself prove publisher identity or code quality. +Operators remain responsible for building, scanning, reviewing, and promoting +feature artifacts. Signature enforcement, SBOM policy, and transparency logs +may be added later without allowing caller-supplied references. + +### Feature OCI Artifact Contract + +A feature is distributed as a normal OCI image so existing Docker and +Kubernetes image distribution, authentication, mirrors, and node layer caches +can be reused. The image has the following required layout: + +```text +/opt/opensandbox-feature/ + metadata.json + activation.json + payload/ + install +``` + +- `metadata.json` declares artifact schema version, feature name, version, OS, + and architecture. These values must match the registry and selected manifest. +- `payload/` is the complete relocatable filesystem tree. It is copied to + `/opt/opensandbox/features//` without running a sandbox base + image package manager. +- `activation.json` is declarative. It describes scalar environment assignments + and ordered path-list prepend/append operations using `${FEATURE_ROOT}` as the + only substitution. It cannot contain shell commands, dependency downloads, + transitive feature requirements, or arbitrary host paths. +- `install` is a self-contained executable used by Kubernetes feature init + containers. It must not rely on a shell, libc, package manager, interpreter, + or binary from the sandbox image. It validates the mounted destination and + copies the payload atomically. + +An example activation document is: + +```json +{ + "schemaVersion": 1, + "set": [ + { + "name": "GIT_EXEC_PATH", + "value": "${FEATURE_ROOT}/libexec/git-core" + } + ], + "prependPath": [ + { + "name": "PATH", + "values": ["${FEATURE_ROOT}/bin"] + } + ] +} +``` + +Installers and server-side extractors MUST reject archive path traversal, +absolute payload paths, writes outside the assigned feature root, and special +device nodes. Payload modes and symlinks are preserved only when their resolved +targets remain inside the feature root. Installation uses a temporary sibling +directory followed by an atomic rename so a failed feature never appears +complete. + +Version 1 has no feature-to-feature dependency metadata and no install hook +beyond deterministic payload placement. If feature B requires feature A, the +administrator must publish B with that dependency in B's self-contained +payload, or callers must request both and the artifacts must be independently +valid. The resolver never inserts, reorders, or upgrades features implicitly. + +### Docker Provisioning + +The Docker provider already creates a temporary container from the configured +execd image, extracts execd/bootstrap artifacts, caches them by platform, creates +the sandbox container without starting it, copies the artifacts through the +Docker archive API, and starts the container only after injection succeeds. +Feature provisioning extends that sequence: + +1. Resolve the effective Linux OS/architecture from the explicit request + platform or the Docker daemon and selected sandbox image. +2. Resolve every feature to the registry's platform-specific manifest digest. +3. Pull the digest-pinned feature image using operator credentials. +4. Extract and validate its metadata, payload, and activation document using a + temporary container, following the existing execd extraction pattern. +5. Cache the validated payload by `(manifest digest, OS, architecture)`. Cache + population is concurrency-safe and uses a temporary entry followed by an + atomic ready marker. +6. Create the stopped sandbox container with a compact ordered + `resolvedFeatures` Docker label. +7. Copy each payload into its final feature root in request order, compile the + activation plan, and inject execd/bootstrap artifacts. +8. Start the sandbox container. The bootstrap applies the compiled activation + environment before it launches execd and the user entrypoint. + +The cache key never uses a mutable tag, requested name, or default-version +alias. A new digest is a new cache entry. Cached content is validated before it +is marked ready, and failed population removes the temporary entry. + +Any pull, validation, copy, activation compilation, or start failure follows +the current `_create_and_start_container` rollback contract: remove the partial +sandbox container, clean provider-created sidecars/volumes as applicable, and +return an explicit create error. A feature failure must not leave a stopped +container that looks like a sandbox. + +### Kubernetes Provisioning + +Both existing Kubernetes workload providers already use an `execd-installer` +init container to copy execd and `bootstrap.sh` into a shared `emptyDir`, which +the main sandbox container mounts at `/opt/opensandbox`. Feature provisioning +extends the generated Pod spec without changing the public workload CRDs: + +1. Keep the existing `execd-installer` init container. +2. Add one feature init container per requested feature, in request order. Each + uses the registry's digest-pinned OCI image and invokes its self-contained + installer. +3. Mount the shared runtime-assets `emptyDir` into each feature init container + and place payloads under `features//`. The main container sees + them at `/opt/opensandbox/features//`. +4. Add a final activation-compiler init container from the pinned execd image. + It validates all activation documents against the request environment and + writes one ordered, shell-escaped activation file into the shared volume. +5. Start the main container only after every init container succeeds. Its + bootstrap sources the compiled activation file before execd and the user + entrypoint. + +Kubernetes init containers are sequential, so request order is preserved. The +Pod retains `automountServiceAccountToken: false`; feature init containers do +not receive API credentials and do not need network access after kubelet pulls +their OCI image. Implementations SHOULD apply the existing restricted +init-container security defaults and grant no additional Linux capabilities. + +When `platform` is explicit, all selected features must contain that platform. +When it is omitted, the resolver computes the intersection of platforms across +the selected feature versions and constrains Pod scheduling to that +intersection. The init-container `image` uses the pinned OCI index digest, so +kubelet selects the matching child manifest after node assignment. The server +then reads the assigned node OS/architecture, verifies the expected child +digest, and patches the ordered resolved metadata onto the workload before a +successful create returns. If the intersection is empty or the assigned +platform cannot be verified, creation fails. + +This approach relies on the node's normal OCI layer cache; version 1 does not +introduce a cluster-wide shared payload cache, daemonset, or persistent volume. +Every sandbox receives a fresh `emptyDir`, while common feature layers are +downloaded once per node according to normal kubelet/runtime cache policy. + +The provider must validate the final merged Pod template. Operator templates +may add unrelated init containers, but they may not remove or reorder generated +feature init containers, replace their digest-pinned images, or shadow the +feature volume mount. Init failure, scheduling failure, metadata-patch failure, +or readiness timeout uses the existing workload deletion and PVC cleanup path. + +### Compatibility Matrix + +| Creation mode | Docker | Kubernetes | Version 1 behavior | +|---|---:|---:|---| +| Linux image-based standard creation | Yes | Yes | Supported | +| `extensions.poolRef` / pool mode | No | No | Reject non-empty `features` | +| Snapshot restore | No | No | Reject non-empty `features` | +| Windows | No | No | Reject non-empty `features` | +| Add/remove/upgrade after create | No | No | No API; immutable for sandbox lifetime | +| Caller-supplied OCI artifact | No | No | Rejected by schema and resolver | + +"Kubernetes" includes the current BatchSandbox template-mode provider and the +current kubernetes-sigs agent-sandbox provider. It does not include the +BatchSandbox pool path. Unsupported combinations fail before provider side +effects whenever possible. The server never drops `features` and proceeds with +an unmodified sandbox. + +Requests that omit `features` retain current behavior on Linux, Windows, image, +snapshot, and pool flows. An empty array is equivalent to omission. + +### Security Boundary + +Feature OCI artifacts are trusted supply code. Their executables eventually run +inside the sandbox, their activation can affect the environment inherited by +execd and the user process, and their Kubernetes installer runs before the main +container. Version 1 controls this risk with an administrator allowlist, +digest-pinned indexes and manifests, platform/metadata verification, constrained +installation roots, and explicit operator registry credentials. + +This mechanism does not make a sandbox more isolated and must not be presented +as a security profile. A malicious or compromised allowlisted feature can still +affect all processes in its sandbox. Operators should apply their normal image +build provenance, vulnerability scanning, SBOM, review, and promotion controls +before adding a digest to the registry. + +This OSEP intentionally does not add `deny_tools`. An execd command-string or +tool-name blacklist and removing directories from `PATH` are usability controls, +not security controls. They are bypassable by: + +- invoking the same binary through an absolute path; +- renaming or copying a binary before invoking it; +- invoking behavior through an interpreter, dynamic linker, library API, or a + caller-supplied static binary; and +- bypassing execd entirely by starting a process through the user entrypoint, a + child process, another in-container service, or another runtime execution + path. + +Removing a known binary has the same problem when the filesystem is writable or +the caller can upload code. A command policy enforced only at one execd endpoint +cannot constrain all process creation in the container. + +Tool restriction belongs in a later security/execution-profile OSEP based on +mechanisms the runtime can actually enforce, such as `networkPolicy` and egress +policy, container capabilities, seccomp, AppArmor, a read-only root filesystem, +and the isolated execution model in +[OSEP-0013](0013-isolated-execution-api.md). Phase 4 below is reserved for that +separate proposal and implementation. + +### Notes/Constraints/Caveats + +- Defaults are operator policy. Omitting `version` is reproducible only after + the server records the exact resolved version and digest. Repeating the same + request after an administrator changes a default may produce a newer feature; + pin a version when cross-time reproducibility is required. +- Feature order is observable through path precedence. Reordering a request may + change which executable is selected when two features provide the same binary + name, even when their scalar environment assignments do not conflict. +- Feature payloads consume sandbox writable storage and increase cold-start + latency. Version 1 does not add separate feature quotas. +- The feature root is immutable by lifecycle API contract, but it is not + necessarily read-only inside every sandbox. Runtime hardening is separate. +- Registry reload, garbage collection, signature policy, and catalog discovery + are deliberately deferred. Static startup loading keeps the first trust and + consistency model reviewable. +- Diagnostics SHOULD report the persisted resolution and the feature/install + stage that failed, but diagnostics must never expose registry credentials. + +### Risks and Mitigations + +| Risk | Mitigation | +|---|---| +| Compromised feature artifact affects execd or the user process | Administrator allowlist, index and child digest pinning, metadata verification, and trusted artifact promotion | +| Mutable defaults reduce repeatability | Return and persist exact version, digest, OS, and architecture; allow callers to pin exact versions | +| Multi-platform index differs from declared catalog | Validate OCI descriptors and selected child digest against the registry | +| Archive traversal or symlink escape writes outside feature root | Strict path validation, reject special devices, install into a temporary root, then atomically rename | +| Concurrent Docker pulls corrupt cache | Digest/platform cache key, per-key locking, validation, and atomic ready marker | +| Feature environments conflict | Declarative activation format and deterministic pre-start conflict validation | +| Kubernetes template overrides generated security/provisioning fields | Validate the final merged Pod spec before API submission | +| Feature init fails after workload creation | Surface init status, delete the workload, and use existing managed-PVC cleanup safeguards | +| Large feature sets increase startup latency and metadata size | Maximum 32 requests, compact persisted JSON, layer caching, and documented operational metrics | +| Users mistake tool denial for a security boundary | Exclude `deny_tools`, document bypasses, and require a separate enforceable-policy OSEP | + +## Design Details + +### Name, Version, and Platform Resolution + +Resolution is deterministic and ordered: + +1. Treat omitted or empty `features` as no feature request. +2. Validate the list length and each structured request. +3. Reject a repeated canonical name, retaining the original request indexes in + the error message. +4. Look up each name in the startup-validated registry. +5. Use the requested exact version, or the feature's single configured default. +6. Compute the effective platform or candidate-platform intersection. +7. Select and verify the platform-specific manifest digest for every feature. +8. Produce the immutable ordered resolution plan used by provisioning and + provider metadata. + +There is no semantic-version comparison, range selection, dependency solver, +or implicit fallback to another version. An unknown explicit version fails even +if the feature has a default. An explicit platform fails if any feature lacks +that exact normalized OS/architecture. In version 1, every effective platform +must have `os=linux`. + +For Docker, effective platform resolution occurs before sandbox container +creation and uses existing platform normalization and image inspection. For +Kubernetes without an explicit platform, the registry intersection is applied +as scheduling constraints and exact selection completes after node assignment. +The create operation remains pending until this selection, init completion, +workload readiness, and resolved-metadata persistence all succeed. + +### Activation and Environment Conflict Rules + +Provisioning copies activation documents without executing them. A trusted +activation compiler validates and merges them in request order into a generated +file under `/opt/opensandbox/features/.resolved/`. Bootstrap sources that file +before the existing execd launch and before dispatching the requested +entrypoint. + +The following rules make activation deterministic: + +- `${FEATURE_ROOT}` expands only to the selected feature's final absolute root. + No other shell or environment interpolation is performed. +- Variable names must match `^[A-Za-z_][A-Za-z0-9_]*$`. +- `set` assigns a scalar. Two features assigning the same scalar variable is a + conflict, even when values happen to match. +- A feature scalar assignment that collides with a caller-provided `env` key is + a conflict. There is no hidden precedence rule. +- OpenSandbox-reserved bootstrap variables, including `EXECD`, `EXECD_ENVS`, + and feature-plan paths, cannot be assigned by a feature. +- `prependPath` and `appendPath` are the only composable operations. Entries are + applied in request order, empty elements are rejected, and duplicate expanded + path elements are rejected as an activation conflict. +- A variable cannot be used by both scalar `set` and a path-list operation. +- Activation cannot unset a variable, execute a command, read a secret, or + reference a path outside its own feature root in version 1. + +Docker performs this compilation before container start. Kubernetes performs it +in the final activation-compiler init container. A compiler failure is a create +failure, not a warning. The same conformance fixtures must be used by both +providers so conflict and ordering behavior cannot drift. + +### Persistence and Read-After-Create Consistency + +The resolution is stored as compact ordered JSON in provider-owned metadata: + +- Docker uses the immutable `opensandbox.io/resolved-features` container label. +- Kubernetes uses the + `sandbox.opensandbox.io/resolved-features` workload annotation. + +The persisted object contains only the public resolved fields: name, version, +platform child digest, OS, and architecture. It does not contain registry +credentials, mutable tags, local cache paths, or activation environment values. + +Docker resolves before container creation, so the label is present from the +first observable container state. Kubernetes may initially create a workload +with an ordered unresolved plan so scheduling can select a node; after node +assignment it patches the exact resolved annotation. The server does not return +create success until that patch is readable. If persistence fails, creation is +rolled back. + +Create response, get response, list items, and diagnostics read the same +provider metadata rather than re-resolving against the current registry. This +ensures an administrator default change does not rewrite the reported state of +an existing sandbox. Malformed persisted metadata is a diagnostic/provider +error and must not be silently replaced with current registry data. + +### Failure Semantics + +Feature errors use the lifecycle API's existing structured error envelope. The +implementation introduces stable machine-readable reasons such as: + +| Condition | HTTP class | Reason | +|---|---:|---| +| Registry not configured | 400 | `FEATURE_REGISTRY_NOT_CONFIGURED` | +| Invalid name/version or duplicate request | 400 | `INVALID_FEATURE_REQUEST` | +| Unknown name | 400 | `FEATURE_NOT_FOUND` | +| Unknown exact version | 400 | `FEATURE_VERSION_NOT_FOUND` | +| Unsupported pool/snapshot/Windows combination | 400 | `FEATURES_UNSUPPORTED_FOR_CREATION_MODE` | +| No compatible platform | 400 | `FEATURE_PLATFORM_UNSUPPORTED` | +| Activation collision | 400 | `FEATURE_ACTIVATION_CONFLICT` | +| Approved OCI registry unavailable | 503 | `FEATURE_IMAGE_UNAVAILABLE` | +| Digest/artifact validation or install failure | 500 | `FEATURE_PROVISIONING_FAILED` | + +Validation errors identify the request index and feature name where safe. +Supply-chain errors identify name, version, and expected digest but do not +include credentials. Kubernetes errors also report the failed init-container +name and termination reason. Provider rollback failure is logged and surfaced +through current cleanup diagnostics rather than hiding the original feature +failure. + +Unknown fields in `FeatureRequest`, raw strings in online JSON, caller OCI +references, and non-empty `features` on an unsupported creation mode are hard +errors. There is no best-effort mode. + +### Implementation Phases + +Implementation proceeds only after this OSEP is approved: + +**Phase 1: contract, resolver, and providers** + +- Add the lifecycle OpenAPI request/response schemas and examples. +- Add matching server Pydantic schemas and provider metadata codecs. +- Add feature-registry configuration, startup validation, resolution, and OCI + artifact validation. +- Implement Docker extraction/cache/pre-start injection and cleanup. +- Implement ordered Kubernetes feature init containers, shared `emptyDir`, + activation compilation, scheduling constraints, persistence, and cleanup for + every existing non-pool workload provider. + +**Phase 2: generated clients and Sandbox SDK alignment** + +- Regenerate lifecycle clients from the OpenAPI source of truth. +- Add aligned public feature request/convenience input and resolved response + models to the Python, JavaScript, Kotlin, C#, and Go Sandbox SDKs. +- Add cross-language serialization tests proving string convenience inputs + produce structured wire JSON and resolved metadata is preserved. + +**Phase 3: CLI, documentation, catalog examples, and end-to-end coverage** + +- Add repeatable CLI `--feature NAME[@VERSION]` support and help/output tests. +- Publish user and operator documentation, including registry and artifact + authoring guidance. +- Add an example feature catalog for tests and documentation. This OSEP itself + does not create that catalog or publish real feature images. +- Add Docker and Kubernetes cross-runtime end-to-end tests. + +**Phase 4: separate enforceable security/execution profile proposal** + +- Write and review a separate OSEP for security/execution profiles. +- Base restrictions on enforceable runtime primitives such as egress policy, + capabilities, seccomp, AppArmor, read-only filesystems, and OSEP-0013 isolated + execution, not command strings or `PATH` filtering. +- Implement any approved profile only after its own security review. + +Merging this OSEP does not implement issue #1289. The issue should remain open +until the OSEP is approved and the end-to-end implementation, SDKs, CLI, +documentation, and cross-runtime tests are complete. Phase 4 may be tracked by +a separate issue because this OSEP rejects the proposed blacklist as a security +mechanism. + +## Test Plan + +Phase 1 must include focused tests at each ownership boundary. + +**Contract and server schema tests** + +- OpenAPI validation accepts ordered structured requests with and without exact + versions and rejects raw strings, extra fields, invalid names/versions, more + than 32 entries, and duplicates. +- Existing create payloads without `features` remain valid. +- Create, get, and list schemas serialize the same `resolvedFeatures` shape and + preserve order. +- Pool, snapshot, and Windows combinations return the documented explicit + error rather than dropping the field. + +**Registry and resolver unit tests** + +- Startup rejects every invalid-registry case listed above, including tag-only + references, missing child digests, duplicate keys, invalid defaults, missing + platforms, and OCI descriptor mismatches. +- Exact versions and defaults resolve correctly; missing names/versions never + fall back. +- Platform normalization, explicit selection, Kubernetes intersection, empty + intersection, and Docker effective-platform behavior are covered for amd64 + and arm64. +- Request order is stable through resolution and persisted JSON encoding. + +**Artifact and activation conformance tests** + +- A minimal feature installs into the exact target root in a base image without + `apt`, `apk`, `yum`, shell, Python, or internet access. +- Payload permissions and safe relative symlinks are preserved. +- Absolute paths, `..` traversal, escaping symlinks, device nodes, metadata + mismatch, and writes outside the assigned root fail atomically. +- Scalar conflicts, caller-env conflicts, reserved variables, mixed scalar/path + operations, duplicate path segments, and malformed activation documents fail. +- Ordered path composition is identical in Docker and Kubernetes fixtures. +- Bootstrap observes feature activation before both execd and the user + entrypoint start. + +**Docker provider tests** + +- Images are pulled only by pinned digest and use operator credentials. +- Cache keys include manifest digest and platform; concurrent cache population + produces one validated ready entry; failed population leaves no usable entry. +- Feature payloads are injected after container creation and before start in + request order, alongside existing execd/bootstrap injection. +- Failures at pull, extraction, validation, copy, compile, and start remove the + partial sandbox and preserve existing sidecar/managed-volume cleanup. +- Create followed by get returns byte-equivalent resolved feature metadata from + the Docker label even after registry defaults change. + +**Kubernetes provider tests** + +- Both BatchSandbox template mode and agent-sandbox manifests contain ordered + feature init containers, the shared `emptyDir` mounts, a final activation + compiler, and a main container that cannot start first. +- Generated feature images retain digest references after template merge. +- Platform intersection becomes scheduling constraints; explicit mismatches and + empty intersections fail before workload creation. +- Node-selected OS/architecture maps to the declared child digest and is + persisted before create success. +- Init failure, scheduling failure, annotation patch failure, and readiness + timeout delete the workload and follow existing managed-PVC cleanup rules. +- Create followed by get returns the persisted annotation rather than resolving + the current registry. + +**Phase 2 and Phase 3 tests** + +- Python, JavaScript, Kotlin, C#, and Go SDK tests cover typed requests, + `name@version` convenience parsing where offered, structured JSON output, and + resolved response conversion. +- CLI tests cover repeated `--feature`, parsing errors, exact SDK calls, and + JSON/YAML rendering. +- Docker and Kubernetes end-to-end tests create an offline minimal sandbox with + two ordered features, run each installed binary through execd, inspect + activation, compare create/get metadata, and verify cleanup. +- Multi-architecture end-to-end coverage verifies that the returned child + digest and OS/architecture match the runtime-selected platform. + +## Drawbacks + +- This adds an operator-maintained artifact supply chain and server startup + dependency on validating its registry. +- Installing payloads at create time is slower than baking the same tools into a + sandbox base image, especially on a cold Docker host or Kubernetes node. +- Static registry reload and no dependency solver make version 1 less flexible + than devcontainer feature ecosystems. +- A self-contained payload may duplicate libraries across features and consume + more disk than distribution package management. +- Kubernetes platform resolution without an explicit platform adds scheduling + and metadata-patch complexity. +- Digest pinning and an allowlist reduce supply-chain ambiguity but do not make + feature code safe. + +## Alternatives + +### Let callers submit arbitrary OCI references + +Rejected. It would turn the lifecycle API into an arbitrary pre-start code +execution and registry-credential surface, bypass administrator review, make +egress/trust policy harder to enforce, and make supportability depend on +unbounded artifact formats. The public request therefore contains only name and +exact optional version; administrators own all OCI references and credentials. + +### Run `apt`, `apk`, or `yum` during creation + +Rejected. It depends on the sandbox distribution, root privileges, package +manager presence, mutable repository metadata, DNS and internet availability, +and installation scripts outside the OpenSandbox supply chain. It is also slow +and difficult to reproduce. Version 1 feature payloads are self-contained and +need no base-image package manager or dependency download. + +### Enforce `deny_tools` with execd command strings or tool names + +Rejected as a security mechanism. String matching does not identify the program +that the kernel will execute and controls only the execd paths where the filter +is installed. Absolute paths, renamed/copied binaries, interpreters, dynamic +linkers, uploaded binaries, and non-execd process entrypoints all bypass it. +Removing a path entry is equally bypassable. Enforceable restriction requires a +separate OSEP and runtime controls such as egress policy, capabilities, seccomp, +AppArmor, read-only filesystems, and isolated execution. + +### Continue requiring custom sandbox images + +This remains a valid optimization for stable, frequently used toolchains and +may have better cold-start performance. It is not sufficient as the only API: +it forces every user to operate an image build pipeline and does not give the +lifecycle server a structured, persisted feature resolution. Declarative +features complement rather than replace custom images. + +### Encode features in `extensions` + +Rejected. `extensions` is opaque and intended for provider-specific, +experimental, or temporary behavior. It cannot express a generated public SDK +model or a stable cross-provider response contract, and using it would conflict +with the lifecycle specification's source-of-truth guidance. + +### Use mutable tags instead of digests + +Rejected. Tags cannot support reproducible create/get metadata, safe cache keys, +or an auditable administrator allowlist. Human-readable versions remain in the +registry, but every index and platform manifest is pinned by digest. + +## Infrastructure Needed + +The proposal requires no new online service. Implementations reuse configured +OCI registries, Docker image storage, and Kubernetes node layer caches. Operators +need a static feature registry file and OCI credentials/trust configuration for +the allowlisted repositories. + +Phase 3 needs CI-owned example feature images for amd64 and arm64, published by +digest to a repository accessible from Docker and Kind end-to-end jobs. Those +images are test/documentation fixtures, not a public marketplace. Their build +pipeline should produce provenance and SBOMs using the project's normal release +controls. + +## Upgrade & Migration Strategy + +The API addition is backward-compatible for callers that do not send +`features`. A server may be upgraded with no registry configured; all existing +creation flows remain unchanged, while feature requests receive an explicit +configuration error. + +Rollout order follows the implementation phases: server/OpenAPI/providers first, +then regenerated clients and handwritten SDK alignment, then CLI/docs/examples +and end-to-end coverage. New SDKs send `features` only when a caller requests +them. They must not downgrade the request into `extensions` when connected to an +older server; the older server's unsupported-field response is safer than +silently creating a sandbox without requested tools. + +Registry defaults may change only for future creates. Existing sandboxes retain +their payloads and persisted `resolvedFeatures` for their lifetime. Removing a +registry version does not rewrite or invalidate a running sandbox, though it +prevents new resolution of that version. Operators should retain old digests in +their OCI registry until no supported workflow needs to recreate them. + +Rollback consists of stopping new feature requests or removing the configured +registry path and rolling back the server. Existing sandboxes continue to run +because their payload is already local and activation occurs inside their +lifecycle. There is no data migration and no dynamic uninstall operation. + +OSEP approval authorizes implementation planning; it does not mean the feature +is implemented. GitHub issue #1289 remains open until this proposal is approved +and Phases 1 through 3 are complete end to end. The security-profile work in +Phase 4 proceeds under its own proposal and review. diff --git a/oseps/README.md b/oseps/README.md index 2f464d273..f3589d465 100644 --- a/oseps/README.md +++ b/oseps/README.md @@ -18,3 +18,6 @@ This is the complete list of OpenSandbox Enhancement Proposals: | [OSEP-0010](0010-opentelemetry-instrumentation.md) | OpenTelemetry Metrics and Logs (execd, egress, and ingress) | implemented | 2026-07-03 | | [OSEP-0011](0011-secure-access-endpoint.md) | Secure Access on GetEndpoint and Signed Endpoint | implemented | 2026-04-25 | | [OSEP-0012](0012-credential-vault.md) | Credential Vault and Credential Proxy | implementing | 2026-06-10 | +| [OSEP-0013](0013-isolated-execution-api.md) | Isolated Execution API | implementing | 2026-06-23 | +| [OSEP-0014](0014-multi-tenancy.md) | Multi-Tenancy Support for Kubernetes Runtime | draft | 2026-05-07 | +| [OSEP-0015](0015-declarative-sandbox-features.md) | Declarative Sandbox Features | draft | 2026-07-14 | From 22ce39f70541ff320b9b6c176200c779937770c8 Mon Sep 17 00:00:00 2001 From: Xuehao Xu Date: Thu, 16 Jul 2026 14:37:05 +0800 Subject: [PATCH 2/5] docs: address OSEP-0015 review feedback --- docs/community/oseps.md | 4 +- oseps/0015-declarative-sandbox-features.md | 95 ++++++++++++++++++---- oseps/README.md | 2 +- 3 files changed, 83 insertions(+), 18 deletions(-) diff --git a/docs/community/oseps.md b/docs/community/oseps.md index 9125b85cc..777e30f4a 100644 --- a/docs/community/oseps.md +++ b/docs/community/oseps.md @@ -26,8 +26,8 @@ See the [OSEP contributing guide](https://github.com/opensandbox-group/OpenSandb | [OSEP-0011](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0011-secure-access-endpoint.md) | Secure Access on GetEndpoint and Signed Endpoint | implemented | 2026-04-25 | | [OSEP-0012](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0012-credential-vault.md) | Credential Vault and Credential Proxy | implemented | 2026-06-23 | | [OSEP-0013](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0013-isolated-execution-api.md) | Isolated Execution API | implementing | 2026-06-23 | -| [OSEP-0014](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0014-multi-tenancy.md) | Multi-Tenancy Support for Kubernetes Runtime | draft | 2026-04-29 | -| [OSEP-0015](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0015-declarative-sandbox-features.md) | Declarative Sandbox Features | draft | 2026-07-14 | +| [OSEP-0014](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0014-multi-tenancy.md) | Multi-Tenancy Support for Kubernetes Runtime | draft | 2026-05-07 | +| [OSEP-0015](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0015-declarative-sandbox-features.md) | Declarative Sandbox Features | draft | 2026-07-16 | ## Status Definitions diff --git a/oseps/0015-declarative-sandbox-features.md b/oseps/0015-declarative-sandbox-features.md index 576cba296..3582d7ac7 100644 --- a/oseps/0015-declarative-sandbox-features.md +++ b/oseps/0015-declarative-sandbox-features.md @@ -3,7 +3,7 @@ title: Declarative Sandbox Features authors: - "@GodBlf" creation-date: 2026-07-14 -last-updated: 2026-07-14 +last-updated: 2026-07-16 status: draft --- @@ -17,6 +17,7 @@ status: draft - [Requirements](#requirements) - [Proposal](#proposal) - [Public API](#public-api) + - [Feature Capability Negotiation](#feature-capability-negotiation) - [Administrator-Controlled Feature Registry](#administrator-controlled-feature-registry) - [Feature OCI Artifact Contract](#feature-oci-artifact-contract) - [Docker Provisioning](#docker-provisioning) @@ -125,6 +126,9 @@ therefore must not be encoded in `extensions`. architecture. - Callers MUST NOT submit an OCI repository, tag, digest, URL, installer script, or registry credentials in the public feature request. +- SDKs and the CLI MUST confirm lifecycle feature capability before sending a + non-empty `features` request. An older or incompatible server MUST be detected + before sandbox creation is attempted. - Every registry entry MUST be digest-pinned and MUST declare at least one supported platform. - Registry configuration MUST be validated when the server starts. A configured @@ -287,6 +291,43 @@ and match `^[0-9A-Za-z](?:[0-9A-Za-z._+-]*[0-9A-Za-z])?$`. Ranges, wildcards, at most 32 features. Repeating a normalized feature name is invalid even when the two entries request different versions. +### Feature Capability Negotiation + +Current pre-feature servers ignore unknown `CreateSandboxRequest` fields because +their Pydantic model does not forbid extras. A new SDK therefore cannot safely +assume that an older server will reject `features`: doing so could start the user +entrypoint without the requested tools. + +Phase 1 adds an additive lifecycle `GET /v1/capabilities` endpoint. A server that +implements this OSEP returns at least: + +```json +{ + "declarativeSandboxFeatures": { + "apiVersion": 1 + } +} +``` + +Additional capability fields may be added later. Clients ignore unknown fields, +but MUST understand the advertised declarative-feature API version before using +it. SDKs MUST call this endpoint, or use a result cached for the same resolved +server base URL and authentication context, before the first create with a +non-empty `features` list. + +A `404`, missing `declarativeSandboxFeatures`, unsupported `apiVersion`, invalid +response, or transport failure means feature creation is unavailable. The SDK +MUST raise an unsupported-server/capability error without sending the create +request. It MUST NOT probe by sending a feature request, create and then delete +an unverified sandbox, or downgrade the request into `extensions`. The CLI uses +the same SDK guard. Callers using raw HTTP are responsible for performing the +same handshake. + +Capability support and registry readiness are distinct. A feature-aware server +advertises API version 1 even when no registry file is configured; a subsequent +feature create then returns `FEATURE_REGISTRY_NOT_CONFIGURED`. This distinction +lets clients tell an old server from an operator configuration error. + ### Administrator-Controlled Feature Registry An operator enables features by configuring a server-side registry file path. @@ -394,8 +435,11 @@ can be reused. The image has the following required layout: transitive feature requirements, or arbitrary host paths. - `install` is a self-contained executable used by Kubernetes feature init containers. It must not rely on a shell, libc, package manager, interpreter, - or binary from the sandbox image. It validates the mounted destination and - copies the payload atomically. + or binary from the sandbox image. It validates the mounted destination, copies + the payload atomically, and persists the validated activation document into + the shared activation-input directory for the final compiler. The Docker + server-side extractor performs the same persistence step without executing + `install`. An example activation document is: @@ -475,12 +519,17 @@ extends the generated Pod spec without changing the public workload CRDs: 2. Add one feature init container per requested feature, in request order. Each uses the registry's digest-pinned OCI image and invokes its self-contained installer. -3. Mount the shared runtime-assets `emptyDir` into each feature init container - and place payloads under `features//`. The main container sees - them at `/opt/opensandbox/features//`. +3. Mount the shared runtime-assets `emptyDir` into each feature init container. + Place payloads under `features//` and copy each validated + `activation.json` to + `features/.resolved/activation-inputs/-.json`. The main + container sees the payload at + `/opt/opensandbox/features//`. Both writes must complete before + that feature init container succeeds. 4. Add a final activation-compiler init container from the pinned execd image. - It validates all activation documents against the request environment and - writes one ordered, shell-escaped activation file into the shared volume. + It reads the persisted activation inputs, validates them against the request + environment, verifies that their indexes exactly match the resolution plan, + and writes one ordered, shell-escaped activation file into the shared volume. 5. Start the main container only after every init container succeeds. Its bootstrap sources the compiled activation file before execd and the user entrypoint. @@ -638,9 +687,11 @@ workload readiness, and resolved-metadata persistence all succeed. Provisioning copies activation documents without executing them. A trusted activation compiler validates and merges them in request order into a generated -file under `/opt/opensandbox/features/.resolved/`. Bootstrap sources that file -before the existing execd launch and before dispatching the requested -entrypoint. +file under `/opt/opensandbox/features/.resolved/`. The uncompiled documents live +under `.resolved/activation-inputs/` and use a zero-padded request index in each +filename, so a missing, duplicated, or reordered input is detectable. Bootstrap +sources only the compiled output before the existing execd launch and before +dispatching the requested entrypoint. The following rules make activation deterministic: @@ -725,6 +776,8 @@ Implementation proceeds only after this OSEP is approved: **Phase 1: contract, resolver, and providers** - Add the lifecycle OpenAPI request/response schemas and examples. +- Add the lifecycle capability endpoint and advertise + `declarativeSandboxFeatures.apiVersion = 1`. - Add matching server Pydantic schemas and provider metadata codecs. - Add feature-registry configuration, startup validation, resolution, and OCI artifact validation. @@ -738,6 +791,8 @@ Implementation proceeds only after this OSEP is approved: - Regenerate lifecycle clients from the OpenAPI source of truth. - Add aligned public feature request/convenience input and resolved response models to the Python, JavaScript, Kotlin, C#, and Go Sandbox SDKs. +- Require all SDKs to negotiate lifecycle feature capability before sending a + non-empty feature request, with no create attempt on older servers. - Add cross-language serialization tests proving string convenience inputs produce structured wire JSON and resolved metadata is preserved. @@ -774,6 +829,8 @@ Phase 1 must include focused tests at each ownership boundary. versions and rejects raw strings, extra fields, invalid names/versions, more than 32 entries, and duplicates. - Existing create payloads without `features` remain valid. +- The capability endpoint advertises API version 1 independently of registry + configuration and remains forward-compatible with unknown capability fields. - Create, get, and list schemas serialize the same `resolvedFeatures` shape and preserve order. - Pool, snapshot, and Windows combinations return the documented explicit @@ -803,6 +860,8 @@ Phase 1 must include focused tests at each ownership boundary. - Ordered path composition is identical in Docker and Kubernetes fixtures. - Bootstrap observes feature activation before both execd and the user entrypoint start. +- Missing, duplicated, or reordered persisted activation inputs fail before the + main container starts. **Docker provider tests** @@ -821,6 +880,8 @@ Phase 1 must include focused tests at each ownership boundary. - Both BatchSandbox template mode and agent-sandbox manifests contain ordered feature init containers, the shared `emptyDir` mounts, a final activation compiler, and a main container that cannot start first. +- Every feature init container persists its validated activation document in the + indexed shared input path before the final compiler runs. - Generated feature images retain digest references after template merge. - Platform intersection becomes scheduling constraints; explicit mismatches and empty intersections fail before workload creation. @@ -836,6 +897,8 @@ Phase 1 must include focused tests at each ownership boundary. - Python, JavaScript, Kotlin, C#, and Go SDK tests cover typed requests, `name@version` convenience parsing where offered, structured JSON output, and resolved response conversion. +- Against a pre-feature server that ignores unknown request fields, every SDK + detects the missing capability and proves that no create request was sent. - CLI tests cover repeated `--feature`, parsing errors, exact SDK calls, and JSON/YAML rendering. - Docker and Kubernetes end-to-end tests create an offline minimal sandbox with @@ -930,10 +993,12 @@ configuration error. Rollout order follows the implementation phases: server/OpenAPI/providers first, then regenerated clients and handwritten SDK alignment, then CLI/docs/examples -and end-to-end coverage. New SDKs send `features` only when a caller requests -them. They must not downgrade the request into `extensions` when connected to an -older server; the older server's unsupported-field response is safer than -silently creating a sandbox without requested tools. +and end-to-end coverage. Phase 1 must deploy the capability endpoint before any +feature-aware SDK or CLI release. New SDKs send `features` only when a caller +requests them and only after capability negotiation succeeds. Because current +pre-feature servers silently ignore unknown request fields, a missing or +incompatible capability response is a client-side hard failure and no create +request is sent. Clients must not downgrade the request into `extensions`. Registry defaults may change only for future creates. Existing sandboxes retain their payloads and persisted `resolvedFeatures` for their lifetime. Removing a diff --git a/oseps/README.md b/oseps/README.md index f3589d465..f72cf98d3 100644 --- a/oseps/README.md +++ b/oseps/README.md @@ -20,4 +20,4 @@ This is the complete list of OpenSandbox Enhancement Proposals: | [OSEP-0012](0012-credential-vault.md) | Credential Vault and Credential Proxy | implementing | 2026-06-10 | | [OSEP-0013](0013-isolated-execution-api.md) | Isolated Execution API | implementing | 2026-06-23 | | [OSEP-0014](0014-multi-tenancy.md) | Multi-Tenancy Support for Kubernetes Runtime | draft | 2026-05-07 | -| [OSEP-0015](0015-declarative-sandbox-features.md) | Declarative Sandbox Features | draft | 2026-07-14 | +| [OSEP-0015](0015-declarative-sandbox-features.md) | Declarative Sandbox Features | draft | 2026-07-16 | From 06dbc58e90fb2999e5af5cfa863b3e377d10c8eb Mon Sep 17 00:00:00 2001 From: Xuehao Xu Date: Fri, 17 Jul 2026 14:34:27 +0800 Subject: [PATCH 3/5] docs: harden OSEP feature rollout and staging --- docs/community/oseps.md | 2 +- oseps/0015-declarative-sandbox-features.md | 154 ++++++++++++++------- oseps/README.md | 2 +- 3 files changed, 105 insertions(+), 53 deletions(-) diff --git a/docs/community/oseps.md b/docs/community/oseps.md index 777e30f4a..4f95338f7 100644 --- a/docs/community/oseps.md +++ b/docs/community/oseps.md @@ -27,7 +27,7 @@ See the [OSEP contributing guide](https://github.com/opensandbox-group/OpenSandb | [OSEP-0012](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0012-credential-vault.md) | Credential Vault and Credential Proxy | implemented | 2026-06-23 | | [OSEP-0013](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0013-isolated-execution-api.md) | Isolated Execution API | implementing | 2026-06-23 | | [OSEP-0014](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0014-multi-tenancy.md) | Multi-Tenancy Support for Kubernetes Runtime | draft | 2026-05-07 | -| [OSEP-0015](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0015-declarative-sandbox-features.md) | Declarative Sandbox Features | draft | 2026-07-16 | +| [OSEP-0015](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0015-declarative-sandbox-features.md) | Declarative Sandbox Features | draft | 2026-07-17 | ## Status Definitions diff --git a/oseps/0015-declarative-sandbox-features.md b/oseps/0015-declarative-sandbox-features.md index 3582d7ac7..d899fa5ee 100644 --- a/oseps/0015-declarative-sandbox-features.md +++ b/oseps/0015-declarative-sandbox-features.md @@ -3,7 +3,7 @@ title: Declarative Sandbox Features authors: - "@GodBlf" creation-date: 2026-07-14 -last-updated: 2026-07-16 +last-updated: 2026-07-17 status: draft --- @@ -129,6 +129,9 @@ therefore must not be encoded in `extensions`. - SDKs and the CLI MUST confirm lifecycle feature capability before sending a non-empty `features` request. An older or incompatible server MUST be detected before sandbox creation is attempted. +- The lifecycle feature capability MUST be a fleet-wide readiness assertion for + every backend behind the advertised base URL. It MUST remain disabled during + a mixed-version rollout or rollback. - Every registry entry MUST be digest-pinned and MUST declare at least one supported platform. - Registry configuration MUST be validated when the server starts. A configured @@ -142,6 +145,9 @@ therefore must not be encoded in `extensions`. required supply-chain fetch. - Feature activation MUST complete before execd and the user entrypoint start. - Features MUST install and activate in request order. +- A feature-supplied Kubernetes installer MUST write only to its own staging + volume. It MUST NOT receive a writable mount containing execd, bootstrap, + another feature's staging output, or final activation inputs. - Unknown names, unknown versions, duplicate names, incompatible platforms, and activation environment conflicts MUST fail creation explicitly. - A provisioning failure MUST clean up a partially created Docker container or @@ -311,9 +317,9 @@ implements this OSEP returns at least: Additional capability fields may be added later. Clients ignore unknown fields, but MUST understand the advertised declarative-feature API version before using -it. SDKs MUST call this endpoint, or use a result cached for the same resolved -server base URL and authentication context, before the first create with a -non-empty `features` list. +it. SDKs MUST call this endpoint immediately before every create with a +non-empty `features` list. They MUST NOT reuse a cached positive result for a +later create. A `404`, missing `declarativeSandboxFeatures`, unsupported `apiVersion`, invalid response, or transport failure means feature creation is unavailable. The SDK @@ -323,6 +329,22 @@ an unverified sandbox, or downgrade the request into `extensions`. The CLI uses the same SDK guard. Callers using raw HTTP are responsible for performing the same handshake. +The response is a deployment-level readiness assertion, not evidence about only +the backend that served the GET. An operator MUST NOT advertise +`declarativeSandboxFeatures` at a load-balanced base URL until every backend +that can receive the subsequent create request implements API version 1. During +Phase 1 rollout, feature capability remains disabled while old instances are +drained; it is enabled only after the fleet reaches that compatibility floor. +This fleet gate closes the GET/create race in which the two requests reach +different versions. + +Rollback reverses that sequence: revoke the fleet capability first, wait for +in-flight capability checks and feature creates to drain, and only then admit an +older backend. The server deployment and ingress/gateway configuration MUST make +this gate atomic from the client's base-URL perspective. Per-instance capability +advertising in a mixed fleet is non-conformant, even if the SDK performs a fresh +GET before each create. + Capability support and registry readiness are distinct. A feature-aware server advertises API version 1 even when no registry file is configured; a subsequent feature create then returns `FEATURE_REGISTRY_NOT_CONFIGURED`. This distinction @@ -435,11 +457,11 @@ can be reused. The image has the following required layout: transitive feature requirements, or arbitrary host paths. - `install` is a self-contained executable used by Kubernetes feature init containers. It must not rely on a shell, libc, package manager, interpreter, - or binary from the sandbox image. It validates the mounted destination, copies - the payload atomically, and persists the validated activation document into - the shared activation-input directory for the final compiler. The Docker - server-side extractor performs the same persistence step without executing - `install`. + or binary from the sandbox image. It copies the payload, metadata, and + activation document into the feature's dedicated staging volume. It never + writes the shared runtime-assets volume or a final feature path. The Docker + server-side extractor does not execute `install`; trusted server code performs + validation and final placement directly. An example activation document is: @@ -461,12 +483,13 @@ An example activation document is: } ``` -Installers and server-side extractors MUST reject archive path traversal, -absolute payload paths, writes outside the assigned feature root, and special -device nodes. Payload modes and symlinks are preserved only when their resolved -targets remain inside the feature root. Installation uses a temporary sibling -directory followed by an atomic rename so a failed feature never appears -complete. +The feature-supplied installer is not trusted to validate restrictions on its +own output. A trusted OpenSandbox extractor or merger MUST reject archive path +traversal, absolute payload paths, writes outside the assigned feature root, and +special device nodes before moving staged content into runtime assets. Payload +modes and symlinks are preserved only when their resolved targets remain inside +the feature root. Final installation uses a temporary sibling directory followed +by an atomic rename so a failed feature never appears complete. Version 1 has no feature-to-feature dependency metadata and no install hook beyond deterministic payload placement. If feature B requires feature A, the @@ -515,17 +538,19 @@ init container to copy execd and `bootstrap.sh` into a shared `emptyDir`, which the main sandbox container mounts at `/opt/opensandbox`. Feature provisioning extends the generated Pod spec without changing the public workload CRDs: -1. Keep the existing `execd-installer` init container. -2. Add one feature init container per requested feature, in request order. Each - uses the registry's digest-pinned OCI image and invokes its self-contained - installer. -3. Mount the shared runtime-assets `emptyDir` into each feature init container. - Place payloads under `features//` and copy each validated - `activation.json` to - `features/.resolved/activation-inputs/-.json`. The main - container sees the payload at - `/opt/opensandbox/features//`. Both writes must complete before - that feature init container succeeds. +1. Keep the existing `execd-installer` init container. Only trusted OpenSandbox + init containers and the main container mount the runtime-assets `emptyDir`. +2. Allocate one dedicated staging `emptyDir` per requested feature and add one + feature init container in request order. Each uses the registry's + digest-pinned OCI image, mounts only its own staging volume as writable, and + invokes its self-contained installer. It cannot mount runtime assets or + another feature's staging volume. +3. Add a trusted `feature-merger` init container from the pinned execd image. + It mounts every feature staging volume read-only and runtime assets writable. + In request order it validates metadata, paths, types, modes, symlinks, and the + activation document, then atomically installs payloads under + `features//` and activation inputs under + `features/.resolved/activation-inputs/-.json`. 4. Add a final activation-compiler init container from the pinned execd image. It reads the persisted activation inputs, validates them against the request environment, verifies that their indexes exactly match the resolution plan, @@ -539,6 +564,9 @@ Pod retains `automountServiceAccountToken: false`; feature init containers do not receive API credentials and do not need network access after kubelet pulls their OCI image. Implementations SHOULD apply the existing restricted init-container security defaults and grant no additional Linux capabilities. +The final merged Pod spec MUST prove that feature-supplied containers have no +mount path or volume reference that aliases runtime assets or another staging +volume. When `platform` is explicit, all selected features must contain that platform. When it is omitted, the resolver computes the intersection of platforms across @@ -558,8 +586,10 @@ downloaded once per node according to normal kubelet/runtime cache policy. The provider must validate the final merged Pod template. Operator templates may add unrelated init containers, but they may not remove or reorder generated feature init containers, replace their digest-pinned images, or shadow the -feature volume mount. Init failure, scheduling failure, metadata-patch failure, -or readiness timeout uses the existing workload deletion and PVC cleanup path. +isolated staging or runtime-assets mounts. They also may not replace the trusted +merger/compiler images or make a staging mount writable in those containers. +Init failure, scheduling failure, metadata-patch failure, or readiness timeout +uses the existing workload deletion and PVC cleanup path. ### Compatibility Matrix @@ -583,12 +613,13 @@ snapshot, and pool flows. An empty array is equivalent to omission. ### Security Boundary -Feature OCI artifacts are trusted supply code. Their executables eventually run -inside the sandbox, their activation can affect the environment inherited by -execd and the user process, and their Kubernetes installer runs before the main -container. Version 1 controls this risk with an administrator allowlist, -digest-pinned indexes and manifests, platform/metadata verification, constrained -installation roots, and explicit operator registry credentials. +Feature OCI artifacts are approved supply code, but provisioning does not assume +their installer is correct. Their executables eventually run inside the sandbox +and their activation can affect the environment inherited by execd and the user +process. Version 1 controls this risk with an administrator allowlist, +digest-pinned indexes and manifests, platform/metadata verification, isolated +installer staging, trusted final merging, constrained installation roots, and +explicit operator registry credentials. This mechanism does not make a sandbox more isolated and must not be presented as a security profile. A malicious or compromised allowlisted feature can still @@ -648,6 +679,8 @@ separate proposal and implementation. | Archive traversal or symlink escape writes outside feature root | Strict path validation, reject special devices, install into a temporary root, then atomically rename | | Concurrent Docker pulls corrupt cache | Digest/platform cache key, per-key locking, validation, and atomic ready marker | | Feature environments conflict | Declarative activation format and deterministic pre-start conflict validation | +| Mixed server versions race capability GET against create | Fleet-wide capability gate; disable during rollout/rollback; no positive SDK caching | +| Feature installer overwrites execd, bootstrap, or another feature | Per-feature staging volumes only; trusted merger owns the runtime-assets mount and final paths | | Kubernetes template overrides generated security/provisioning fields | Validate the final merged Pod spec before API submission | | Feature init fails after workload creation | Surface init status, delete the workload, and use existing managed-PVC cleanup safeguards | | Large feature sets increase startup latency and metadata size | Maximum 32 requests, compact persisted JSON, layer caching, and documented operational metrics | @@ -778,13 +811,16 @@ Implementation proceeds only after this OSEP is approved: - Add the lifecycle OpenAPI request/response schemas and examples. - Add the lifecycle capability endpoint and advertise `declarativeSandboxFeatures.apiVersion = 1`. +- Add the fleet-wide feature capability gate and document rollout/drain + integration for the supported deployment paths. - Add matching server Pydantic schemas and provider metadata codecs. - Add feature-registry configuration, startup validation, resolution, and OCI artifact validation. - Implement Docker extraction/cache/pre-start injection and cleanup. -- Implement ordered Kubernetes feature init containers, shared `emptyDir`, - activation compilation, scheduling constraints, persistence, and cleanup for - every existing non-pool workload provider. +- Implement ordered Kubernetes feature init containers with isolated staging + volumes, a trusted merger into runtime assets, activation compilation, + scheduling constraints, persistence, and cleanup for every existing non-pool + workload provider. **Phase 2: generated clients and Sandbox SDK alignment** @@ -831,6 +867,9 @@ Phase 1 must include focused tests at each ownership boundary. - Existing create payloads without `features` remain valid. - The capability endpoint advertises API version 1 independently of registry configuration and remains forward-compatible with unknown capability fields. +- Mixed-version rollout tests prove capability remains absent until every + backend behind the base URL is compatible; rollback tests revoke it and drain + in-flight work before an older backend becomes eligible. - Create, get, and list schemas serialize the same `resolvedFeatures` shape and preserve order. - Pool, snapshot, and Windows combinations return the documented explicit @@ -878,10 +917,14 @@ Phase 1 must include focused tests at each ownership boundary. **Kubernetes provider tests** - Both BatchSandbox template mode and agent-sandbox manifests contain ordered - feature init containers, the shared `emptyDir` mounts, a final activation - compiler, and a main container that cannot start first. -- Every feature init container persists its validated activation document in the - indexed shared input path before the final compiler runs. + feature init containers with per-feature staging volumes, a trusted merger, + a final activation compiler, and a main container that cannot start first. +- Feature-supplied containers mount only their own staging volume; malicious or + buggy writes cannot alter execd, bootstrap, another feature's output, runtime + assets, or final activation inputs. +- The trusted merger mounts staging read-only, rejects escaping or malformed + output, and persists each validated activation document in the indexed shared + input path before the final compiler runs. - Generated feature images retain digest references after template merge. - Platform intersection becomes scheduling constraints; explicit mismatches and empty intersections fail before workload creation. @@ -899,6 +942,9 @@ Phase 1 must include focused tests at each ownership boundary. resolved response conversion. - Against a pre-feature server that ignores unknown request fields, every SDK detects the missing capability and proves that no create request was sent. +- Load-balanced mixed-version tests route capability and create to different + candidate instances and prove the fleet gate prevents feature capability from + being advertised until both are compatible. - CLI tests cover repeated `--feature`, parsing errors, exact SDK calls, and JSON/YAML rendering. - Docker and Kubernetes end-to-end tests create an offline minimal sandbox with @@ -993,12 +1039,15 @@ configuration error. Rollout order follows the implementation phases: server/OpenAPI/providers first, then regenerated clients and handwritten SDK alignment, then CLI/docs/examples -and end-to-end coverage. Phase 1 must deploy the capability endpoint before any -feature-aware SDK or CLI release. New SDKs send `features` only when a caller -requests them and only after capability negotiation succeeds. Because current -pre-feature servers silently ignore unknown request fields, a missing or -incompatible capability response is a client-side hard failure and no create -request is sent. Clients must not downgrade the request into `extensions`. +and end-to-end coverage. Phase 1 must deploy compatible feature handling to every +backend behind a base URL while the fleet capability remains disabled. After old +instances are drained, the operator enables the deployment-level capability; +only then may a feature-aware SDK or CLI be released against that endpoint. New +SDKs perform a fresh capability check for every non-empty feature create and do +not cache a positive result. Because current pre-feature servers silently ignore +unknown request fields, a missing or incompatible capability response is a +client-side hard failure and no create request is sent. Clients must not +downgrade the request into `extensions`. Registry defaults may change only for future creates. Existing sandboxes retain their payloads and persisted `resolvedFeatures` for their lifetime. Removing a @@ -1006,10 +1055,13 @@ registry version does not rewrite or invalidate a running sandbox, though it prevents new resolution of that version. Operators should retain old digests in their OCI registry until no supported workflow needs to recreate them. -Rollback consists of stopping new feature requests or removing the configured -registry path and rolling back the server. Existing sandboxes continue to run -because their payload is already local and activation occurs inside their -lifecycle. There is no data migration and no dynamic uninstall operation. +Rollback first revokes feature capability for the whole base URL, then waits for +in-flight capability checks and feature creates to drain before any older server +is admitted. Removing the registry path alone is not a compatibility gate; a +feature-aware fleet without a registry remains capable but returns +`FEATURE_REGISTRY_NOT_CONFIGURED`. Existing sandboxes continue to run because +their payload is already local and activation occurs inside their lifecycle. +There is no data migration and no dynamic uninstall operation. OSEP approval authorizes implementation planning; it does not mean the feature is implemented. GitHub issue #1289 remains open until this proposal is approved diff --git a/oseps/README.md b/oseps/README.md index f72cf98d3..de02c850e 100644 --- a/oseps/README.md +++ b/oseps/README.md @@ -20,4 +20,4 @@ This is the complete list of OpenSandbox Enhancement Proposals: | [OSEP-0012](0012-credential-vault.md) | Credential Vault and Credential Proxy | implementing | 2026-06-10 | | [OSEP-0013](0013-isolated-execution-api.md) | Isolated Execution API | implementing | 2026-06-23 | | [OSEP-0014](0014-multi-tenancy.md) | Multi-Tenancy Support for Kubernetes Runtime | draft | 2026-05-07 | -| [OSEP-0015](0015-declarative-sandbox-features.md) | Declarative Sandbox Features | draft | 2026-07-16 | +| [OSEP-0015](0015-declarative-sandbox-features.md) | Declarative Sandbox Features | draft | 2026-07-17 | From 7748857bc1f9d3f89931e2adfaf7c9781f899904 Mon Sep 17 00:00:00 2001 From: Xuehao Xu Date: Fri, 17 Jul 2026 14:37:01 +0800 Subject: [PATCH 4/5] docs: define feature snapshot guard --- oseps/0015-declarative-sandbox-features.md | 33 ++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/oseps/0015-declarative-sandbox-features.md b/oseps/0015-declarative-sandbox-features.md index d899fa5ee..c11fa870d 100644 --- a/oseps/0015-declarative-sandbox-features.md +++ b/oseps/0015-declarative-sandbox-features.md @@ -108,7 +108,8 @@ therefore must not be encoded in `extensions`. - Transitive feature dependency declaration or dependency solving. - Adding, removing, or upgrading a feature after sandbox creation. - Pool-backed creation through `extensions.poolRef`. -- Snapshot restore or including feature state in a snapshot contract. +- Snapshot creation from a feature-enabled sandbox, snapshot restore with + `features`, or including feature state in a snapshot contract. - Windows feature images or activation. - Tool denial, command-name filtering, `PATH` hiding, or a `deny_tools` API. - Treating a feature as a sandbox isolation or privilege boundary. @@ -156,6 +157,9 @@ therefore must not be encoded in `extensions`. all current Kubernetes workload providers. - Version 1 MUST reject pool mode, snapshot restore, and Windows when `features` is non-empty. +- Version 1 MUST reject snapshot creation when the source sandbox has non-empty + persisted `resolvedFeatures`. The check MUST occur before any filesystem + commit, snapshot CR, or other provider side effect. ## Proposal @@ -598,6 +602,7 @@ uses the existing workload deletion and PVC cleanup path. | Linux image-based standard creation | Yes | Yes | Supported | | `extensions.poolRef` / pool mode | No | No | Reject non-empty `features` | | Snapshot restore | No | No | Reject non-empty `features` | +| Snapshot creation from feature-enabled sandbox | No | No | Reject when persisted `resolvedFeatures` is non-empty | | Windows | No | No | Reject non-empty `features` | | Add/remove/upgrade after create | No | No | No API; immutable for sandbox lifetime | | Caller-supplied OCI artifact | No | No | Rejected by schema and resolver | @@ -609,7 +614,12 @@ effects whenever possible. The server never drops `features` and proceeds with an unmodified sandbox. Requests that omit `features` retain current behavior on Linux, Windows, image, -snapshot, and pool flows. An empty array is equivalent to omission. +pool, and snapshot restore flows for snapshots that do not contain managed +feature state. An empty array is equivalent to omission. Snapshot creation is +the sole additional guard: a feature-enabled source is rejected even though the +snapshot request itself has no `features` field, because current providers may +capture the injected payload and compiled activation without a corresponding +snapshot-level `resolvedFeatures` contract. ### Security Boundary @@ -663,6 +673,9 @@ separate proposal and implementation. latency. Version 1 does not add separate feature quotas. - The feature root is immutable by lifecycle API contract, but it is not necessarily read-only inside every sandbox. Runtime hardening is separate. +- Feature-enabled sandboxes cannot be snapshotted in version 1. Persisting and + restoring their payload plus immutable resolution requires a later extension + to the snapshot contract. - Registry reload, garbage collection, signature policy, and catalog discovery are deliberately deferred. Static startup loading keeps the first trust and consistency model reviewable. @@ -786,6 +799,7 @@ implementation introduces stable machine-readable reasons such as: | Unknown name | 400 | `FEATURE_NOT_FOUND` | | Unknown exact version | 400 | `FEATURE_VERSION_NOT_FOUND` | | Unsupported pool/snapshot/Windows combination | 400 | `FEATURES_UNSUPPORTED_FOR_CREATION_MODE` | +| Snapshot requested for feature-enabled sandbox | 400 | `FEATURE_SNAPSHOT_UNSUPPORTED` | | No compatible platform | 400 | `FEATURE_PLATFORM_UNSUPPORTED` | | Activation collision | 400 | `FEATURE_ACTIVATION_CONFLICT` | | Approved OCI registry unavailable | 503 | `FEATURE_IMAGE_UNAVAILABLE` | @@ -817,6 +831,8 @@ Implementation proceeds only after this OSEP is approved: - Add feature-registry configuration, startup validation, resolution, and OCI artifact validation. - Implement Docker extraction/cache/pre-start injection and cleanup. +- Reject snapshot creation from a source with persisted resolved features before + invoking Docker commit or creating a Kubernetes snapshot resource. - Implement ordered Kubernetes feature init containers with isolated staging volumes, a trusted merger into runtime assets, activation compilation, scheduling constraints, persistence, and cleanup for every existing non-pool @@ -874,6 +890,8 @@ Phase 1 must include focused tests at each ownership boundary. preserve order. - Pool, snapshot, and Windows combinations return the documented explicit error rather than dropping the field. +- Snapshot creation schema/service tests reject a source with non-empty + `resolvedFeatures` before calling any provider snapshot operation. **Registry and resolver unit tests** @@ -913,6 +931,8 @@ Phase 1 must include focused tests at each ownership boundary. partial sandbox and preserve existing sidecar/managed-volume cleanup. - Create followed by get returns byte-equivalent resolved feature metadata from the Docker label even after registry defaults change. +- Snapshot creation from a feature-enabled Docker sandbox returns + `FEATURE_SNAPSHOT_UNSUPPORTED` and never invokes image commit. **Kubernetes provider tests** @@ -934,6 +954,8 @@ Phase 1 must include focused tests at each ownership boundary. timeout delete the workload and follow existing managed-PVC cleanup rules. - Create followed by get returns the persisted annotation rather than resolving the current registry. +- Snapshot creation from a feature-enabled Kubernetes sandbox fails before a + snapshot CR or commit workload is created. **Phase 2 and Phase 3 tests** @@ -1055,6 +1077,13 @@ registry version does not rewrite or invalidate a running sandbox, though it prevents new resolution of that version. Operators should retain old digests in their OCI registry until no supported workflow needs to recreate them. +After upgrading to a feature-aware server, snapshot requests for existing +feature-enabled sandboxes begin returning `FEATURE_SNAPSHOT_UNSUPPORTED`. This +is intentional protection against metadata drift, not a transient provider +failure. Users who need a reusable environment must pin the feature request and +create a new image-based sandbox; feature-aware snapshots require a future OSEP +extension. + Rollback first revokes feature capability for the whole base URL, then waits for in-flight capability checks and feature creates to drain before any older server is admitted. Removing the registry path alone is not a compatibility gate; a From 395eeb5b1dce63bff4693ad89aa75eb16eae1ed2 Mon Sep 17 00:00:00 2001 From: Xuehao Xu Date: Tue, 21 Jul 2026 15:23:22 +0800 Subject: [PATCH 5/5] docs: clarify feature workload scope --- oseps/0015-declarative-sandbox-features.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/oseps/0015-declarative-sandbox-features.md b/oseps/0015-declarative-sandbox-features.md index c11fa870d..e739ddba3 100644 --- a/oseps/0015-declarative-sandbox-features.md +++ b/oseps/0015-declarative-sandbox-features.md @@ -107,6 +107,9 @@ therefore must not be encoded in `extensions`. sandbox creation. - Transitive feature dependency declaration or dependency solving. - Adding, removing, or upgrading a feature after sandbox creation. +- Replacing pre-baked OCI images or VM templates for stable, high-throughput + workloads where cold-start latency, cache locality, and deterministic startup + are the primary constraints. - Pool-backed creation through `extensions.poolRef`. - Snapshot creation from a feature-enabled sandbox, snapshot restore with `features`, or including feature state in a snapshot contract. @@ -1020,11 +1023,13 @@ AppArmor, read-only filesystems, and isolated execution. ### Continue requiring custom sandbox images -This remains a valid optimization for stable, frequently used toolchains and -may have better cold-start performance. It is not sufficient as the only API: -it forces every user to operate an image build pipeline and does not give the +This may remain the preferred optimization for stable, frequently used +toolchains and high-throughput fleets where cold-start latency, cache locality, +and deterministic startup dominate. It is not sufficient as the only API: it +forces every user to operate an image build pipeline and does not give the lifecycle server a structured, persisted feature resolution. Declarative -features complement rather than replace custom images. +features target dynamic, per-request toolchains and complement rather than +replace pre-baked OCI images or VM templates. ### Encode features in `extensions`