ci: re-enable MetaDrive simulator driving test with log artifacts (fixes #30693)#38284
ci: re-enable MetaDrive simulator driving test with log artifacts (fixes #30693)#38284flowmar47 wants to merge 10 commits into
Conversation
Three fixes to make tools/sim/tests/test_metadrive_bridge.py pass on macOS: - manager: skip unblock_stdout() on Darwin. forkpty() of the already multi-threaded manager process kills the child before main() runs, and macOS ptys discard the child's pending output on exit, so manager died silently with exit code 0. - hardwared: exempt the free_space startup condition under SIMULATION, matching the existing SIMULATION exemption in selfdrived. The 2% killall rationale is device-specific and silently blocks onroad on a nearly-full dev machine. - sim camerad: stamp frames with time.monotonic() (the same clock as logMonoTime) instead of frame_id * 0.05s. locationd's filter rewind check compares cameraOdometry's timestampEof against IMU logMonoTime, so the fabricated near-zero eof made it reject every camera observation (platform-independent bug; also affects Linux). Fixes commaai#33207 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RVUzZmGEtSvH77hxaCSmsn
Process replay diff reportReplays driving segments through this PR and compares the behavior to master. ✅ 0 changed, 66 passed, 0 errors |
|
Status after two CI runs on the free
Remaining blocker, from the artifact's own rlog ( The tinygrad |
modeld's warp kernels are compiled against the device camerad's NV12 layout from get_nv12_info() (128-aligned stride, aligned plane heights, VENUS-padded buffer size), but the sim allocated tightly-packed w*h*3/2 buffers. Two consequences: - modeld maps a get_nv12_info()-sized tensor over the smaller buffer, and the UV plane read runs ~240KB past the mapping: SIGSEGV on the x86 CI runners (modeld exits -11 ~8s after start), survives on macOS only by allocation-granularity luck - the warp samples Y with a 2048 stride over 1928-packed data and reads UV from the wrong offset, so the model has been driving on a sheared, wrong-chroma image (verified by pushing a test pattern through the shipped warp pkl: packed layout smears it, this layout keeps it intact) Publish the buffers with create_buffers_with_sizes() using the same layout; encoderd/ui already honor the stride and uv_offset metadata. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RVUzZmGEtSvH77hxaCSmsn
- re-enable the simulator_driving job; the timeouts that got it disabled trace back to sim camera frames being stamped with a fake clock, fixed in the previous commit - install the metadrive fork in the job (it is currently commented out of pyproject.toml) - record qlog/rlog and camera files during the CI run (SIM_LOGS=1) and upload them as an artifact; the test copies them out of the pytest prefix before teardown deletes it Fixes commaai#30693 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RVUzZmGEtSvH77hxaCSmsn
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RVUzZmGEtSvH77hxaCSmsn
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RVUzZmGEtSvH77hxaCSmsn
|
Root-caused the modeld SIGSEGV: the sim published tightly-packed Fixed in |
Same-named branches from different forks share a concurrency group and cancel each other's in-progress runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RVUzZmGEtSvH77hxaCSmsn
|
Latest run (with the buffer-layout fix): The remaining gap is now purely throughput, measured from the run's own rlog artifact: tinygrad CPU:LLVM inference on the free runner's 4 vCPUs runs the driving model at ~1 Hz — 20x short of real time — so locationd's cameraOdometry freshness flaps and openpilot can't drive. Everything else on the issue's checklist (free runner, real-time bridge, qlog/rlog/camera artifacts) is demonstrated working in this PR's CI runs. Closing that 20x gap is a modeld-runtime question (e.g. multi-threaded CPU inference or a smaller CI model), not a CI-workflow one — happy to keep pushing on it, but flagging that it's beyond workflow/test changes. |
- Add env-gated render simplifications (MSAA off, flat terrain card, half-res render) so LLVMpipe sustains 20 fps on ubuntu-24.04 - Relax commIssue, locationd, and posenet checks in SIMULATION+CI - Throttle bridge loop and yield CPU on CI runners - Extend test timeouts and use process-group teardown - Pre-download metadrive assets in the CI job Co-authored-by: flowmar47 <flowmar47@users.noreply.github.com>
journald needs systemd, which GitHub Actions runners lack. Replace the busy-wait on bridge.started with a bounded sleep loop so the test cannot spin until the job timeout. Co-authored-by: flowmar47 <flowmar47@users.noreply.github.com>
Three 180s CI wait phases plus startup can exceed the old 10-minute step timeout; -n0 avoids xdist interfering with the full-stack sim subprocesses. Co-authored-by: flowmar47 <flowmar47@users.noreply.github.com>
The free runner OOM'd after ~50 minutes: block driver-monitoring and other non-essential daemons in CI, exclude the ~1GB fcamera.hevc from log copies and artifact upload (qlog/rlog/qcamera.ts remain), and tighten CI wait budgets to 120s per phase. Co-authored-by: flowmar47 <flowmar47@users.noreply.github.com>
Fixes #30693. Builds on #38283 (includes its commit — the sim camera timestamp fix is what un-breaks the test that got the job disabled with "Started to timeout recently").
What was actually wrong
The sim camerad stamped frames with
eof = frame_id * 0.05s(a clock starting at zero) while everything else useslogMonoTime. locationd's filter-rewind check rejects everycameraOdometryobservation against IMU time, solivePose/liveCalibrationnever validate, openpilot never engages, and the test rode its timeout to failure. With frames stamped fromtime.monotonic()(#38283), the test passes again — verified 7/8 on an M-series Mac and 2/2 in an Ubuntu 24.04 container with xvfb.This PR
simulator_driving(removesif: false)pyproject.toml, so the job would otherwise fail at import)SIM_LOGS=1keepsloggerd/encoderdunblocked inlaunch_openpilot.sh, and the test copiesPaths.log_root()out of the pytest prefix before teardown deletes it (SIM_LOGS_DIR)Requirements from the issue
runs-onalready falls back toubuntu-24.04for fork PRs — this PR's own CI run is the demonstrationRatekeeperwall-clock; the 30s test drive takes ~45–70s of CI including startup