Skip to content

Commit 89d6488

Browse files
RemiPellouxRemiPelloux
committed
feat: implement full container runtime across all platforms
Complete implementation of the Containust container runtime with 9 phases delivering a working cross-platform container engine. Core Runtime (Linux): - Namespace isolation (PID, Mount, Network, User, IPC, UTS) via unshare/setns - Cgroup v2 resource management (CPU, memory, I/O) - OverlayFS, pivot_root, essential filesystem mounts - Capability dropping via prctl(PR_CAPBSET_DROP) - All functions gated with #[cfg(target_os = "linux")] Image Management (cross-platform): - SHA-256 file hashing with sha2 crate - Source resolution: file://, tar://, https:// URI protocols - Layer extraction with tar + flate2 (supports .tar.gz) - JSON-backed image catalog with CRUD operations .ctst Parser (cross-platform): - Full nom-based lexer with 17 token types - Recursive-descent parser for 16+ component properties - Semantic validation (duplicates, undefined refs, missing image) - Topological sort via petgraph with cycle detection - Connection resolver with auto-wired env injection - Import resolver for local .ctst files Container Lifecycle: - Process spawning with chroot-based isolation - Container start/stop with SIGTERM/SIGKILL grace period - Exec via nsenter-based namespace joining - JSON state persistence with full container metadata - Cgroup-based metrics collection - Append/read log management Engine + CLI Wiring: - ContainerBackend trait for platform abstraction - LinuxNativeBackend with direct syscall implementation - Engine orchestrator: parse -> validate -> graph -> deploy - All 10 CLI commands wired (build, plan, run, ps, exec, stop, images, convert, logs, vm) VM Backend (macOS/Windows): - QEMU-based lightweight Alpine Linux VM - Platform acceleration: HVF (macOS), WHPX (Windows), TCG fallback - JSON-RPC protocol over TCP for container operations - Auto-detection with ~/.containust/vm/ asset management Testing: - 167 tests passing, 0 failures - 36 end-to-end integration tests - Zero clippy warnings, zero formatting issues - Zero todo!() macros remaining CI/CD: - 3-OS matrix (Linux, macOS, Windows) for check/clippy/test - 5-target release workflow with SHA-256 checksums - cargo-deny security auditing Documentation: - README with platform badges, support matrix, architecture - Updated ARCHITECTURE.md, CLI_REFERENCE.md, SPEC.md - Updated CONTRIBUTING.md, TUTORIALS.md - 8,409 lines of Rust across 83 source files
1 parent 0951238 commit 89d6488

80 files changed

Lines changed: 6484 additions & 287 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.cursor/rules/architecture.mdc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ Each crate has an explicit, closed set of allowed internal dependencies. Adding
2828
| containust-common | NONE (leaf crate) |
2929
| containust-core | containust-common |
3030
| containust-image | containust-common, containust-core |
31-
| containust-runtime | containust-common, containust-core, containust-ebpf |
31+
| containust-runtime | containust-common, containust-core, containust-ebpf, containust-image, containust-compose |
3232
| containust-compose | containust-common, containust-core |
3333
| containust-ebpf | containust-common |
3434
| containust-sdk | containust-common, containust-runtime, containust-image, containust-compose |

‎.github/workflows/ci.yml‎

Lines changed: 31 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -8,44 +8,60 @@ on:
88

99
env:
1010
CARGO_TERM_COLOR: always
11-
RUSTFLAGS: -D warnings
11+
RUSTFLAGS: "-D warnings"
1212

1313
jobs:
1414
check:
15-
name: Check
16-
runs-on: ubuntu-latest
15+
name: Check (${{ matrix.os }})
16+
runs-on: ${{ matrix.os }}
17+
strategy:
18+
fail-fast: false
19+
matrix:
20+
os: [ubuntu-latest, macos-latest, windows-latest]
1721
steps:
1822
- uses: actions/checkout@v4
1923
- uses: dtolnay/rust-toolchain@stable
24+
with:
25+
components: clippy, rustfmt
2026
- uses: Swatinem/rust-cache@v2
21-
- run: cargo check --workspace --all-targets
27+
- name: Check
28+
run: cargo check --workspace --all-targets
29+
- name: Clippy
30+
run: cargo clippy --workspace --all-targets -- -D warnings
2231

23-
fmt:
32+
format:
2433
name: Format
2534
runs-on: ubuntu-latest
2635
steps:
2736
- uses: actions/checkout@v4
2837
- uses: dtolnay/rust-toolchain@stable
2938
with:
3039
components: rustfmt
31-
- run: cargo fmt --all --check
40+
- name: Format check
41+
run: cargo fmt --all --check
3242

33-
clippy:
34-
name: Clippy
35-
runs-on: ubuntu-latest
43+
test:
44+
name: Test (${{ matrix.os }})
45+
runs-on: ${{ matrix.os }}
46+
strategy:
47+
fail-fast: false
48+
matrix:
49+
os: [ubuntu-latest, macos-latest, windows-latest]
3650
steps:
3751
- uses: actions/checkout@v4
3852
- uses: dtolnay/rust-toolchain@stable
39-
with:
40-
components: clippy
4153
- uses: Swatinem/rust-cache@v2
42-
- run: cargo clippy --workspace --all-targets -- -D warnings
54+
- name: Run tests
55+
run: cargo test --workspace
4356

44-
test:
45-
name: Test
57+
docs:
58+
name: Documentation
4659
runs-on: ubuntu-latest
4760
steps:
4861
- uses: actions/checkout@v4
4962
- uses: dtolnay/rust-toolchain@stable
5063
- uses: Swatinem/rust-cache@v2
51-
- run: cargo test --workspace
64+
- name: Build docs
65+
run: cargo doc --workspace --no-deps
66+
env:
67+
RUSTDOCFLAGS: "-D warnings"

‎.github/workflows/release.yml‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
name: Release
2+
3+
on:
4+
push:
5+
tags: ["v*"]
6+
7+
permissions:
8+
contents: write
9+
10+
env:
11+
CARGO_TERM_COLOR: always
12+
13+
jobs:
14+
build:
15+
name: Build (${{ matrix.target }})
16+
runs-on: ${{ matrix.os }}
17+
strategy:
18+
fail-fast: false
19+
matrix:
20+
include:
21+
- target: x86_64-unknown-linux-gnu
22+
os: ubuntu-latest
23+
artifact: ctst
24+
- target: aarch64-unknown-linux-gnu
25+
os: ubuntu-latest
26+
artifact: ctst
27+
cross: true
28+
- target: x86_64-apple-darwin
29+
os: macos-latest
30+
artifact: ctst
31+
- target: aarch64-apple-darwin
32+
os: macos-latest
33+
artifact: ctst
34+
- target: x86_64-pc-windows-msvc
35+
os: windows-latest
36+
artifact: ctst.exe
37+
steps:
38+
- uses: actions/checkout@v4
39+
40+
- uses: dtolnay/rust-toolchain@stable
41+
with:
42+
targets: ${{ matrix.target }}
43+
44+
- uses: Swatinem/rust-cache@v2
45+
with:
46+
key: ${{ matrix.target }}
47+
48+
- name: Install cross
49+
if: matrix.cross
50+
run: cargo install cross --git https://github.com/cross-rs/cross
51+
52+
- name: Build (cross)
53+
if: matrix.cross
54+
run: cross build --release --target ${{ matrix.target }} --package containust-cli
55+
56+
- name: Build (native)
57+
if: "!matrix.cross"
58+
run: cargo build --release --target ${{ matrix.target }} --package containust-cli
59+
60+
- name: Package (Unix)
61+
if: runner.os != 'Windows'
62+
run: |
63+
cd target/${{ matrix.target }}/release
64+
tar czf ../../../ctst-${{ matrix.target }}.tar.gz ${{ matrix.artifact }}
65+
cd ../../..
66+
sha256sum ctst-${{ matrix.target }}.tar.gz > ctst-${{ matrix.target }}.tar.gz.sha256
67+
68+
- name: Package (Windows)
69+
if: runner.os == 'Windows'
70+
shell: pwsh
71+
run: |
72+
Compress-Archive -Path "target/${{ matrix.target }}/release/${{ matrix.artifact }}" -DestinationPath "ctst-${{ matrix.target }}.zip"
73+
(Get-FileHash "ctst-${{ matrix.target }}.zip" -Algorithm SHA256).Hash.ToLower() + " ctst-${{ matrix.target }}.zip" | Out-File "ctst-${{ matrix.target }}.zip.sha256" -Encoding ASCII
74+
75+
- name: Upload artifact
76+
uses: actions/upload-artifact@v4
77+
with:
78+
name: ctst-${{ matrix.target }}
79+
path: ctst-${{ matrix.target }}.*
80+
81+
release:
82+
name: Create Release
83+
needs: build
84+
runs-on: ubuntu-latest
85+
steps:
86+
- uses: actions/checkout@v4
87+
88+
- name: Download all artifacts
89+
uses: actions/download-artifact@v4
90+
with:
91+
path: artifacts
92+
93+
- name: Create GitHub Release
94+
uses: softprops/action-gh-release@v2
95+
with:
96+
generate_release_notes: true
97+
files: artifacts/**/*
98+
env:
99+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎.github/workflows/security.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,9 @@ on:
88
schedule:
99
- cron: '0 6 * * 1'
1010

11+
permissions:
12+
contents: read
13+
1114
jobs:
1215
audit:
1316
name: Dependency Audit

‎ARCHITECTURE.md‎

Lines changed: 93 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -53,10 +53,39 @@ Image and layer lifecycle:
5353
#### `containust-runtime`
5454
Container lifecycle management:
5555
- Container struct with state machine (Created → Running → Stopped → Failed).
56-
- Process spawning inside isolated namespaces.
56+
- Process spawning inside isolated namespaces with chroot isolation.
5757
- Persistent state index (`state.json`) for daemon-less management.
58-
- Namespace joining for `exec` operations.
59-
- Real-time metrics collection from cgroup stat files.
58+
- Namespace joining for `exec` operations via `nsenter`.
59+
- Real-time metrics collection from cgroup v2 stat files.
60+
- Container log management with append-only log files (`logs.rs`).
61+
- Platform-agnostic backend trait (`ContainerBackend`) with Linux native and VM implementations (`backend/`).
62+
- Runtime engine that orchestrates `.ctst` deployments through the compose layer (`engine.rs`).
63+
64+
##### `ContainerBackend` Trait
65+
66+
The `ContainerBackend` trait abstracts all platform-specific container operations:
67+
68+
```rust
69+
pub trait ContainerBackend: Send + Sync {
70+
fn create(&self, config: &ContainerConfig) -> Result<ContainerId>;
71+
fn start(&self, id: &ContainerId) -> Result<u32>;
72+
fn stop(&self, id: &ContainerId) -> Result<()>;
73+
fn exec(&self, id: &ContainerId, cmd: &[String]) -> Result<ExecOutput>;
74+
fn remove(&self, id: &ContainerId) -> Result<()>;
75+
fn logs(&self, id: &ContainerId) -> Result<String>;
76+
fn list(&self) -> Result<Vec<ContainerInfo>>;
77+
fn is_available(&self) -> bool;
78+
}
79+
```
80+
81+
Two implementations exist:
82+
83+
| Implementation | Module | Platform | Mechanism |
84+
|---|---|---|---|
85+
| `LinuxNativeBackend` | `backend/linux.rs` | Linux | Direct syscalls: `clone(2)`, `unshare(2)`, cgroups v2, OverlayFS |
86+
| `VMBackend` | `backend/vm.rs` | macOS, Windows | QEMU VM with JSON-RPC over TCP to a Linux guest agent |
87+
88+
Backend selection is automatic via `detect_backend()`, which uses compile-time `#[cfg(target_os)]` on Linux and runtime QEMU detection on other platforms.
6089

6190
#### `containust-compose`
6291
`.ctst` language processing:
@@ -86,7 +115,7 @@ Public facade for using Containust as a library:
86115
### CLI Layer
87116

88117
#### `containust-cli`
89-
The `ctst` binary with subcommands: `build`, `plan`, `run`, `ps`, `exec`, `stop`, `images`.
118+
The `ctst` binary with subcommands: `build`, `plan`, `run`, `ps`, `exec`, `stop`, `logs`, `images`, `convert`, `vm`.
90119
Uses `clap` for argument parsing and `anyhow` for error reporting.
91120

92121
#### `containust-tui`
@@ -104,7 +133,7 @@ Dependencies flow strictly downward through the layers. The complete allowed-dep
104133
| `containust-common` | None |
105134
| `containust-core` | `containust-common` |
106135
| `containust-image` | `containust-common`, `containust-core` |
107-
| `containust-runtime` | `containust-common`, `containust-core`, `containust-ebpf` |
136+
| `containust-runtime` | `containust-common`, `containust-core`, `containust-ebpf`, `containust-image`, `containust-compose` |
108137
| `containust-compose` | `containust-common`, `containust-core` |
109138
| `containust-ebpf` | `containust-common` |
110139
| `containust-sdk` | `containust-common`, `containust-runtime`, `containust-image`, `containust-compose` |
@@ -126,3 +155,62 @@ OverlayFS enables efficient layer caching and copy-on-write semantics without du
126155

127156
### Why feature-gated eBPF?
128157
eBPF requires a modern Linux kernel and BPF support. Feature-gating it with `ebpf` allows the core runtime to work on systems without BPF, including development on macOS via cross-compilation.
158+
159+
### Why a VM backend on macOS/Windows?
160+
Linux containers require Linux kernel primitives (namespaces, cgroups, OverlayFS). On non-Linux platforms, Containust boots a lightweight Alpine Linux VM (~50MB) via QEMU with hardware acceleration (HVF on macOS, Hyper-V/WHPX on Windows). Container operations are forwarded to the native Linux backend inside the VM via JSON-RPC over TCP, achieving sub-2s boot time and near-native performance.
161+
162+
## Platform Backend Architecture
163+
164+
```mermaid
165+
graph TB
166+
CLI[ctst CLI] --> SDK[containust-sdk]
167+
SDK --> RT[containust-runtime]
168+
RT --> |detect_backend| DETECT{Platform?}
169+
DETECT --> |Linux| NATIVE[LinuxNativeBackend]
170+
DETECT --> |macOS/Windows| VM[VMBackend]
171+
172+
NATIVE --> NS[Namespaces]
173+
NATIVE --> CG[Cgroups v2]
174+
NATIVE --> OFS[OverlayFS]
175+
176+
VM --> QEMU[QEMU VM]
177+
QEMU --> |JSON-RPC/TCP| AGENT[VM Agent]
178+
AGENT --> NS2[Namespaces]
179+
AGENT --> CG2[Cgroups v2]
180+
AGENT --> OFS2[OverlayFS]
181+
```
182+
183+
## VM Lifecycle
184+
185+
The VM backend follows this lifecycle on macOS and Windows:
186+
187+
1. **Boot** — `ctst vm start` (or auto-start on first container operation) launches QEMU with hardware acceleration. The Alpine Linux VM image (~50MB) boots in under 2 seconds.
188+
2. **Connection** — The CLI establishes a JSON-RPC connection over TCP to the VM agent running inside the guest.
189+
3. **Container Operations** — All `ContainerBackend` trait calls (`create`, `start`, `stop`, `exec`, `logs`, `list`) are serialized as JSON-RPC requests and forwarded to the VM agent, which delegates to the `LinuxNativeBackend` inside the VM.
190+
4. **Shutdown** — `ctst vm stop` gracefully shuts down the VM. The VM is also stopped automatically when no containers are running and the CLI exits.
191+
192+
## Module Dependency Graph (Mermaid)
193+
194+
```mermaid
195+
graph TD
196+
CLI[containust-cli] --> SDK[containust-sdk]
197+
CLI --> TUI[containust-tui]
198+
CLI --> COMMON[containust-common]
199+
TUI --> SDK
200+
TUI --> COMMON
201+
SDK --> RUNTIME[containust-runtime]
202+
SDK --> IMAGE[containust-image]
203+
SDK --> COMPOSE[containust-compose]
204+
SDK --> COMMON
205+
RUNTIME --> CORE[containust-core]
206+
RUNTIME --> IMAGE
207+
RUNTIME --> COMPOSE
208+
RUNTIME --> EBPF[containust-ebpf]
209+
RUNTIME --> COMMON
210+
IMAGE --> CORE
211+
IMAGE --> COMMON
212+
COMPOSE --> CORE
213+
COMPOSE --> COMMON
214+
EBPF --> COMMON
215+
CORE --> COMMON
216+
```

0 commit comments

Comments
 (0)