Skip to content

Commit d55ca0d

Browse files
ZhiXiao-LinRoy Lin
andauthored
feat(runtime): exhaustively inject durable lifecycle faults (#11)
Co-authored-by: Roy Lin <roylin@a3s.box>
1 parent d0520a8 commit d55ca0d

18 files changed

Lines changed: 1846 additions & 77 deletions

File tree

.github/workflows/ci.yml

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@ jobs:
2424
- uses: Swatinem/rust-cache@v2
2525
- run: cargo fmt --all --check
2626
- run: cargo clippy --workspace --all-targets -- -D warnings
27+
- name: Exhaustive durable-state and host-driver fault matrix
28+
run: cargo test -p a3s-oci-runtime fault_matrix
2729
- run: cargo test --workspace --all-targets
2830
- run: cargo run -p a3s-oci-cli -- features
2931
- if: runner.os == 'macOS'
@@ -599,6 +601,12 @@ jobs:
599601
for target in x86_64-unknown-linux-musl aarch64-unknown-linux-musl; do
600602
agent="target/$target/release/a3s-oci-agent"
601603
file "$agent"
602-
! readelf --program-headers "$agent" | grep --quiet INTERP
603-
! readelf --dynamic "$agent" | grep --quiet NEEDED
604+
if readelf --program-headers "$agent" | grep --quiet INTERP; then
605+
echo "$agent contains a dynamic interpreter" >&2
606+
exit 1
607+
fi
608+
if readelf --dynamic "$agent" | grep --quiet NEEDED; then
609+
echo "$agent contains a dynamic dependency" >&2
610+
exit 1
611+
fi
604612
done

README.md

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,8 @@ Workload calls require an explicitly supplied launch-ready `RuntimeDriver`.
8181
state mutation
8282
- **Durable Lifecycle**: Persist create, state, start, kill, and delete with
8383
monotonic generations, operation IDs, replay, fencing, reconciliation, and
84-
quarantine
84+
quarantine, backed by exhaustive durable-write and driver-boundary fault
85+
matrices
8586
- **Shared Linux Executor**: Reuse one fail-closed namespace, mount, process,
8687
and cleanup implementation directly on Linux and through the guest agent
8788
- **Cross-Platform Drivers**: Inspect native Linux, KVM, HVF, and WHPX
@@ -471,8 +472,9 @@ RFC 2119 keyword occurrences from the normative specification. CI verifies:
471472

472473
This is not yet a claim of full OCI conformance. Remaining normative entries,
473474
complete enforcement, hooks, descriptor-relative filesystem operations,
474-
recovery fault injection, upstream lifecycle suites, and platform security
475-
certification must pass before a driver becomes `supported`.
475+
utility-VM host/agent transition fault injection, upstream lifecycle suites,
476+
and platform security certification must pass before a driver becomes
477+
`supported`.
476478

477479
Security-sensitive platform controls include:
478480

@@ -523,6 +525,8 @@ cargo clippy \
523525

524526
Platform CI covers:
525527

528+
- the 237-point durable commit matrix and all 12 `RuntimeDriver` call
529+
boundaries on Linux, macOS, and Windows;
526530
- Ubuntu x86_64 native lifecycle and three-phase no-delete cleanup without KVM;
527531
- Ubuntu aarch64 native lifecycle and three-phase no-delete cleanup without KVM;
528532
- macOS HVF, isolated libkrun context, guest-marker, authenticated-agent,

ROADMAP.md

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -115,14 +115,16 @@ Completed:
115115
terminal failure replay, crash reconciliation, and quarantine;
116116
- async `RuntimeDriver` integration plus a tested host implementation of
117117
`create`, `state`, `start`, `kill`, and `delete`;
118+
- typed, exhaustive recovery injection at all 237 registered durable commit
119+
stages and all 12 before/after `RuntimeDriver` method boundaries;
118120
- runtime-owned Windows state paths with protected DACLs limited to the
119121
runtime principal and LocalSystem, inheritance disabled, and every applied
120122
owner and ACL verified;
121123
- Windows, Linux, and macOS CI.
122124

123125
Not yet complete:
124126

125-
- fault injection at every durable write and host/driver boundary;
127+
- fault injection inside every utility-VM host/agent transport transition;
126128
- descriptor-relative path resolution;
127129
- complete shared guest OCI executor;
128130
- a production workload driver;
@@ -181,11 +183,14 @@ enforce it. No property is silently ignored.
181183
- [x] Preserve the exact create/start barrier in the durable host/driver
182184
contract.
183185
- [x] Verify the barrier against the real Linux guest bootstrap executor.
186+
- [x] Fault-inject every registered core-lifecycle durable commit stage and
187+
every `RuntimeDriver` method boundary, then reopen and replay.
184188
- [ ] Implement all OCI hook phases and error behavior.
185189
- [ ] Implement `run` as a client composition, not a second lifecycle.
186190

187191
Exit gate: lifecycle tests pass under fault injection at every durable write
188-
and host/agent transition.
192+
and host/agent transition. The durable-write and `RuntimeDriver` portions pass;
193+
the utility-VM host/agent transport portion remains open.
189194

190195
### R2 — Windows WHPX Utility VM
191196

crates/runtime/src/fault.rs

Lines changed: 294 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,294 @@
1+
use std::fmt;
2+
3+
use a3s_oci_sdk::Result;
4+
5+
/// One semantic mutation of runtime-owned durable state.
6+
///
7+
/// Every file replacement and directory move in the lifecycle store must carry
8+
/// exactly one of these identities. The test matrix treats `ALL` as the
9+
/// auditable coverage registry.
10+
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
11+
pub(crate) enum DurableMutation {
12+
RuntimeRootMarker,
13+
AllocateGeneration,
14+
PrepareCreateOperation,
15+
StoreCreateConfig,
16+
StoreCreatingContainer,
17+
ClaimCreateOperation,
18+
CompleteCreateContainer,
19+
CompleteCreateOperation,
20+
PrepareStartOperation,
21+
ClaimStartOperation,
22+
ReconcileStartContainer,
23+
ReconcileStartOperation,
24+
CompleteStartContainer,
25+
CompleteStartOperation,
26+
PrepareKillOperation,
27+
ClaimKillOperation,
28+
ReconcileKillContainer,
29+
ReconcileKillOperation,
30+
CompleteKillContainer,
31+
CompleteKillOperation,
32+
PrepareDeleteOperation,
33+
ClaimDeleteOperation,
34+
ReconcileDeleteOperation,
35+
MoveDeleteTombstone,
36+
CompleteDeleteOperation,
37+
RecordCreateFailure,
38+
MoveFailedCreateTombstone,
39+
ReleaseFailedStartClaim,
40+
RecordStartFailure,
41+
ReleaseFailedKillClaim,
42+
RecordKillFailure,
43+
ReleaseFailedDeleteClaim,
44+
RecordDeleteFailure,
45+
ObserveContainer,
46+
CompleteObservedOperation,
47+
}
48+
49+
impl DurableMutation {
50+
#[cfg(test)]
51+
pub(crate) const ALL: [Self; 35] = [
52+
Self::RuntimeRootMarker,
53+
Self::AllocateGeneration,
54+
Self::PrepareCreateOperation,
55+
Self::StoreCreateConfig,
56+
Self::StoreCreatingContainer,
57+
Self::ClaimCreateOperation,
58+
Self::CompleteCreateContainer,
59+
Self::CompleteCreateOperation,
60+
Self::PrepareStartOperation,
61+
Self::ClaimStartOperation,
62+
Self::ReconcileStartContainer,
63+
Self::ReconcileStartOperation,
64+
Self::CompleteStartContainer,
65+
Self::CompleteStartOperation,
66+
Self::PrepareKillOperation,
67+
Self::ClaimKillOperation,
68+
Self::ReconcileKillContainer,
69+
Self::ReconcileKillOperation,
70+
Self::CompleteKillContainer,
71+
Self::CompleteKillOperation,
72+
Self::PrepareDeleteOperation,
73+
Self::ClaimDeleteOperation,
74+
Self::ReconcileDeleteOperation,
75+
Self::MoveDeleteTombstone,
76+
Self::CompleteDeleteOperation,
77+
Self::RecordCreateFailure,
78+
Self::MoveFailedCreateTombstone,
79+
Self::ReleaseFailedStartClaim,
80+
Self::RecordStartFailure,
81+
Self::ReleaseFailedKillClaim,
82+
Self::RecordKillFailure,
83+
Self::ReleaseFailedDeleteClaim,
84+
Self::RecordDeleteFailure,
85+
Self::ObserveContainer,
86+
Self::CompleteObservedOperation,
87+
];
88+
89+
#[must_use]
90+
pub(crate) const fn is_directory_move(self) -> bool {
91+
matches!(
92+
self,
93+
Self::MoveDeleteTombstone | Self::MoveFailedCreateTombstone
94+
)
95+
}
96+
}
97+
98+
/// Crash boundary within one atomic file replacement.
99+
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
100+
pub(crate) enum FileCommitStage {
101+
TemporaryFileCreated,
102+
TemporaryFileProtected,
103+
DataWritten,
104+
DataFlushed,
105+
FileSynced,
106+
FileReplaced,
107+
ParentDirectorySynced,
108+
}
109+
110+
impl FileCommitStage {
111+
#[cfg(test)]
112+
pub(crate) const ALL: [Self; 7] = [
113+
Self::TemporaryFileCreated,
114+
Self::TemporaryFileProtected,
115+
Self::DataWritten,
116+
Self::DataFlushed,
117+
Self::FileSynced,
118+
Self::FileReplaced,
119+
Self::ParentDirectorySynced,
120+
];
121+
}
122+
123+
/// Crash boundary within one atomic directory move.
124+
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
125+
pub(crate) enum DirectoryCommitStage {
126+
DirectoryMoved,
127+
SourceParentSynced,
128+
DestinationParentSynced,
129+
}
130+
131+
impl DirectoryCommitStage {
132+
#[cfg(test)]
133+
pub(crate) const ALL: [Self; 3] = [
134+
Self::DirectoryMoved,
135+
Self::SourceParentSynced,
136+
Self::DestinationParentSynced,
137+
];
138+
}
139+
140+
/// Runtime driver method crossed by host orchestration.
141+
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
142+
pub(crate) enum DriverOperation {
143+
Capability,
144+
Create,
145+
State,
146+
Start,
147+
Kill,
148+
Delete,
149+
}
150+
151+
impl DriverOperation {
152+
#[cfg(test)]
153+
pub(crate) const ALL: [Self; 6] = [
154+
Self::Capability,
155+
Self::Create,
156+
Self::State,
157+
Self::Start,
158+
Self::Kill,
159+
Self::Delete,
160+
];
161+
}
162+
163+
/// Side of one host/driver call boundary.
164+
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
165+
pub(crate) enum DriverBoundaryStage {
166+
BeforeCall,
167+
AfterCall,
168+
}
169+
170+
impl DriverBoundaryStage {
171+
#[cfg(test)]
172+
pub(crate) const ALL: [Self; 2] = [Self::BeforeCall, Self::AfterCall];
173+
}
174+
175+
/// Typed, enumerable fault identity used by deterministic recovery tests.
176+
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
177+
pub(crate) enum FaultPoint {
178+
DurableFile {
179+
mutation: DurableMutation,
180+
stage: FileCommitStage,
181+
},
182+
DurableDirectory {
183+
mutation: DurableMutation,
184+
stage: DirectoryCommitStage,
185+
},
186+
DriverBoundary {
187+
operation: DriverOperation,
188+
stage: DriverBoundaryStage,
189+
},
190+
}
191+
192+
impl fmt::Display for FaultPoint {
193+
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
194+
write!(formatter, "{self:?}")
195+
}
196+
}
197+
198+
#[cfg(test)]
199+
impl FaultPoint {
200+
pub(crate) fn durable_registry() -> Vec<Self> {
201+
DurableMutation::ALL
202+
.into_iter()
203+
.flat_map(|mutation| {
204+
if mutation.is_directory_move() {
205+
DirectoryCommitStage::ALL
206+
.into_iter()
207+
.map(move |stage| Self::DurableDirectory { mutation, stage })
208+
.collect::<Vec<_>>()
209+
} else {
210+
FileCommitStage::ALL
211+
.into_iter()
212+
.map(move |stage| Self::DurableFile { mutation, stage })
213+
.collect::<Vec<_>>()
214+
}
215+
})
216+
.collect()
217+
}
218+
219+
pub(crate) fn driver_registry() -> Vec<Self> {
220+
DriverOperation::ALL
221+
.into_iter()
222+
.flat_map(|operation| {
223+
DriverBoundaryStage::ALL
224+
.into_iter()
225+
.map(move |stage| Self::DriverBoundary { operation, stage })
226+
})
227+
.collect()
228+
}
229+
}
230+
231+
/// Synchronous checkpoint called immediately after durable commit stages and
232+
/// immediately before or after host/driver calls.
233+
pub(crate) trait FaultInjector: fmt::Debug + Send + Sync {
234+
fn check(&self, point: FaultPoint) -> Result<()>;
235+
}
236+
237+
#[derive(Debug, Default)]
238+
pub(crate) struct NoFaultInjector;
239+
240+
impl FaultInjector for NoFaultInjector {
241+
fn check(&self, _point: FaultPoint) -> Result<()> {
242+
Ok(())
243+
}
244+
}
245+
246+
#[cfg(test)]
247+
pub(crate) mod testing {
248+
use std::sync::atomic::{AtomicBool, Ordering};
249+
use std::sync::Mutex;
250+
251+
use a3s_oci_sdk::{Error, ErrorCode, Result};
252+
253+
use super::{FaultInjector, FaultPoint};
254+
255+
#[derive(Debug)]
256+
pub(crate) struct RecordingFaultInjector {
257+
target: FaultPoint,
258+
fired: AtomicBool,
259+
events: Mutex<Vec<FaultPoint>>,
260+
}
261+
262+
impl RecordingFaultInjector {
263+
pub(crate) fn fail_once(target: FaultPoint) -> Self {
264+
Self {
265+
target,
266+
fired: AtomicBool::new(false),
267+
events: Mutex::new(Vec::new()),
268+
}
269+
}
270+
271+
pub(crate) fn fired(&self) -> bool {
272+
self.fired.load(Ordering::SeqCst)
273+
}
274+
275+
pub(crate) fn events(&self) -> Vec<FaultPoint> {
276+
self.events.lock().expect("fault event lock").clone()
277+
}
278+
}
279+
280+
impl FaultInjector for RecordingFaultInjector {
281+
fn check(&self, point: FaultPoint) -> Result<()> {
282+
self.events.lock().expect("fault event lock").push(point);
283+
if self.target == point && !self.fired.swap(true, Ordering::SeqCst) {
284+
return Err(Error::new(
285+
ErrorCode::Unavailable,
286+
format!("injected fault at {point}"),
287+
)
288+
.for_operation("fault-injection")
289+
.retryable(true));
290+
}
291+
Ok(())
292+
}
293+
}
294+
}

crates/runtime/src/lib.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ mod agent_smoke_process;
1717
mod agent_socket;
1818
mod cleanup_report;
1919
mod driver;
20+
mod fault;
2021
#[cfg(all(target_os = "macos", target_arch = "aarch64"))]
2122
mod host_cleanup;
2223
#[cfg(target_os = "linux")]

0 commit comments

Comments
 (0)