Skip to content

Commit f6b2000

Browse files
author
Roy Lin
committed
feat(sdk): parallel pipeline fan-out + typed JSON report with metrics
Base::run_parallel runs steps as concurrent copy-on-write MicroVM forks (bounded, collect-all, input-ordered) and returns a dependency-free JSON Report. StepResult gains separated stdout/stderr, duration_ms, and metrics parsed from `::metric <key>=<value>` guest-stdout lines (a scoring channel for matrix/selection workloads). Steps now take &self (atomic fork counter), so fan-out no longer needs hand-rolled threads. RAII removes each fork on every path and the base snapshot on Drop (--force). Box/snapshot names carry per-process+instance entropy to prevent cross-pipeline name collisions; output is capped to bound report memory under large fan-out; infra failures use a distinct sentinel. 15 unit tests + doctest, clippy clean; validated end-to-end on a real /dev/kvm host (real boot, CoW fork-per-step, metrics, JSON report, 0 leaks).
1 parent 70e6285 commit f6b2000

4 files changed

Lines changed: 561 additions & 80 deletions

File tree

CHANGELOG.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,26 @@ All notable changes to A3S Box will be documented in this file.
44

55
## [Unreleased]
66

7+
### Added
8+
9+
- **Programmable-CI pipeline: parallel fan-out + typed JSON report (`a3s-box-sdk`).**
10+
`Base::run_parallel(steps, max_concurrency)` runs steps concurrently as isolated
11+
copy-on-write MicroVM forks (bounded, collect-all, results in input order) and returns a
12+
`Report` with a dependency-free `to_json()`. `StepResult` now carries separated
13+
`stdout`/`stderr`, `duration_ms`, and `metrics` parsed from `::metric <key>=<value>`
14+
guest-stdout lines (a machine-readable scoring channel for matrix/selection workloads).
15+
Steps run via `&self` (atomic fork counter), so fan-out no longer needs hand-rolled
16+
threads. The base auto-removes its snapshot on `Drop` (`--force`), and each fork is
17+
removed on every path (including panic). Box/snapshot names now carry per-process +
18+
per-instance entropy, so concurrent pipelines from the same image+setup can no longer
19+
collide and tear down each other's boxes. Validated end-to-end on a real `/dev/kvm` host.
20+
21+
### Changed
22+
23+
- **`StepResult.logs` is replaced by separated `stdout` / `stderr` fields**
24+
(use `StepResult::combined()` for the old concatenated view). Breaking for
25+
direct `.logs` field access on the `a3s-box-sdk` pipeline API.
26+
727
## [2.6.0] — 2026-06-26
828

929
### Added

src/sdk/README.md

Lines changed: 28 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -11,28 +11,48 @@ state). Set `A3S_BOX` if `a3s-box` is not on `PATH`.
1111

1212
## Pipelines
1313

14-
Warm a base box **once** (clone + install deps), snapshot it, fork per step:
14+
Warm a base box **once** (clone + install deps), snapshot it, fork per step.
15+
Run steps **sequentially** (fail-fast) or **in parallel** (collect-all → a typed
16+
report):
1517

1618
```rust
1719
use a3s_box_sdk::pipeline::{warm_base, WarmBase, FileCache, Step};
1820

1921
fn main() -> Result<(), a3s_box_sdk::pipeline::PipelineError> {
2022
let cache = FileCache::new(".ci-cache")?; // skip a step when its inputs are unchanged
21-
let mut base = warm_base(
23+
let base = warm_base(
2224
WarmBase::new("node:20", "git clone $REPO /w && cd /w && npm ci") // runs ONCE
2325
.env("REPO", "https://github.com/me/app")
2426
.cache(&cache),
2527
)?;
28+
29+
// Sequential, fail-fast: a non-zero exit returns Err.
2630
base.step(Step::new("lint", "cd /w && npm run lint"))?;
27-
base.step(Step::new("test", "cd /w && npm test"))?; // nonzero exit -> Err (fail-fast)
28-
base.step(Step::new("build", "cd /w && npm run build"))?;
29-
base.dispose(); // drops the snapshot
30-
Ok(())
31+
32+
// Parallel, collect-all: each step is an isolated CoW fork; <=4 at a time.
33+
let report = base.run_parallel(vec![
34+
Step::new("test", "cd /w && npm test"),
35+
Step::new("build", "cd /w && npm run build"),
36+
], 4);
37+
38+
println!("{}", report.to_json()); // {"passed":..,"total_ms":..,"steps":[..]}
39+
if !report.passed { /* inspect report.failures() */ }
40+
Ok(()) // `base` drops here -> snapshot auto-removed (or call base.dispose())
3141
}
3242
```
3343

34-
`Step::allow_failure()` keeps the pipeline going on a non-zero exit; `Step::input(..)`
35-
adds extra cache-key parts. Parallel steps = spawn threads (each `step` is blocking).
44+
- `run_parallel` is the way to use a3s-box's cheap (~ms) CoW fork at scale — a
45+
matrix / evolution-style batch — without hand-rolling threads (every method
46+
takes `&self`).
47+
- A step reports a metric by printing `::metric <key>=<number>` to stdout; it
48+
surfaces as `StepResult::metrics` (the scoring channel for a selection loop).
49+
- `StepResult` carries separated `stdout`/`stderr`, `duration_ms`, and `cached`;
50+
`Report::to_json()` is the machine-readable handoff to an agent/scorer.
51+
- `Step::allow_failure()` keeps a non-zero step from failing the run; `Step::input(..)`
52+
adds extra cache-key parts.
53+
54+
The base **auto-disposes** its snapshot on drop, and each per-step box is removed
55+
on every path (including a panic), so a long-running batch doesn't leak.
3656

3757
## Why forking is cheap
3858

src/sdk/examples/pipeline.rs

Lines changed: 24 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -12,25 +12,36 @@ fn main() -> Result<(), a3s_box_sdk::pipeline::PipelineError> {
1212
let cache = FileCache::new("/tmp/.a3s-ci-demo")?;
1313

1414
// Warm the base once (here: write a marker = "deps installed"), snapshot it.
15-
let mut base = warm_base(WarmBase::new(image, "echo DEPS-INSTALLED > /warmed").cache(&cache))?;
15+
let base = warm_base(WarmBase::new(image, "echo DEPS-INSTALLED > /warmed").cache(&cache))?;
1616

17+
// Sequential, fail-fast: a non-zero exit returns Err.
1718
let r = base.step(Step::new("read", "cat /warmed"))?;
18-
println!(
19-
"read -> code={} out={:?} cached={}",
20-
r.exit_code,
21-
r.logs.trim(),
22-
r.cached
23-
);
24-
19+
println!("read -> code={} out={:?}", r.exit_code, r.stdout.trim());
2520
let r2 = base.step(Step::new("read", "cat /warmed"))?; // identical -> cache hit
2621
println!("read#2 -> cached={}", r2.cached);
2722

28-
match base.step(Step::new("fail", "exit 7")) {
29-
Err(e) => println!("fail-fast ok: {e}"),
30-
Ok(_) => println!("ERROR: fail step did not error"),
31-
}
23+
// Parallel matrix, collect-all: each step is an isolated CoW fork of the base.
24+
// A step reports a metric by printing `::metric key=value`.
25+
let report = base.run_parallel(
26+
vec![
27+
Step::new("ok", "echo fine"),
28+
Step::new("perf", "echo '::metric duration_ms=12.5'"),
29+
Step::new("fail", "exit 7"), // collected, not fatal
30+
],
31+
4,
32+
);
33+
println!("report -> {}", report.to_json());
34+
println!(
35+
"passed={} failures={:?}",
36+
report.passed,
37+
report
38+
.failures()
39+
.iter()
40+
.map(|s| &s.name)
41+
.collect::<Vec<_>>()
42+
);
3243

33-
base.dispose();
44+
// base.dispose() is optional — the snapshot is removed when `base` drops.
3445
println!("demo ok");
3546
Ok(())
3647
}

0 commit comments

Comments
 (0)