Update tests by vcharraut · Pull Request #462 · Emerge-Lab/PufferDrive

vcharraut · 2026-05-30T23:31:34Z

Restructure test suite & consolidate CI workflows

Summary

Reorganizes the test suite into a clear unit_tests/ vs smoke_tests/ split, collapses the fragmented CI workflows into a single ci.yml, and adds a deterministic CPU smoke test for the training pipeline. Net: −685 / +473 lines, far fewer workflow files, and tests that pass from any working directory.

Test layout

tests/
├── unit_tests/          # fast, isolated
│   ├── test_drive_config.py        (moved from tests/)
│   ├── test_drive_map_types.py     (moved)
│   ├── test_eval_manager.py        (moved)
│   ├── test_map_cache.py           (moved)
│   ├── test_single_agent_yaml.py   (moved)
│   ├── test_geometry.py            (moved from pufferlib/ocean/benchmark/)
│   ├── test_map_metrics.py         (moved from benchmark/)
│   ├── test_road_edges.py          (moved from benchmark/)
│   └── test_ttc.py                 (moved from benchmark/)
└── smoke_tests/         # end-to-end pipeline
    ├── test_drive_train.py         (new: deterministic golden-based smoke)
    ├── test_drive_eval.py          (new: stub, "to fill")
    ├── test_validation_replay_html.py (moved)
    ├── test_simulator_perf.py      (moved)
    └── data/drive_smoke_golden.json (committed golden)

Moved the geometry/metric tests out of pufferlib/ocean/benchmark/ into tests/unit_tests/ so all tests live under tests/.
New test_drive_train.py — runs the real load_config → load_env → load_policy → PuffeRL pipeline for 5 CPU epochs and compares PPO + env metrics against a committed golden (np.isclose, tunable via SMOKE_RTOL/SMOKE_ATOL). Replaces the old tests/test_drive_train.py.
Path fixes: every relocated test computed repo-root by walking up a fixed number of levels from __file__; bumped each by one level so fixtures (pufferlib/resources/drive/binaries, launcher YAML, golden) resolve from the new depth.
test_simulator_perf.py: changed the hardcoded cwd-relative map_dir="resources/drive/binaries" to an absolute __file__-based path pointing at binaries/carla (top-level dir holds no .bin files since maps were split into subdirs). It now runs under bare pytest from the repo root.

CI workflows

Replaced 5 overlapping workflows (utest.yml, smoke-train-test.yml, train-ci.yml, perf-ci.yml, training-test.yml) and the dead reusable _cleanup.yml with a single ci.yml containing three parallel jobs:

Job	Runs
`unit-tests`	`pytest tests/unit_tests`
`smoke-tests`	`test_drive_train.py` + `test_validation_replay_html.py`
`perf-tests`	`test_simulator_perf.py`

Dropped stale tests/ini_parser/ C tests and the deleted test_drive_scenario_length.py from CI.
Removed df -h debug noise; bumped actions/* to v4/v5; TMPDIR/PIP_NO_CACHE_DIR scoped to the install step (runner.temp isn't valid in job-level env).
install.yml slimmed to a pure cross-platform install matrix (dropped its duplicate pre-commit job, now py3.12/3.11/3.10).
pre-commit.yml bumped to Python 3.13.

Known trade-off / follow-ups

The three CI jobs each repeat pip install -e . + build_ext (no shared filesystem across jobs). Can be cut to one build via a build job + artifact if desired.
tests/smoke_tests/test_drive_eval.py is a stub ("to fill").

🤖 Generated with Claude Code

…formance

…rf-ci.yml file

Copilot

Pull request overview

This PR reorganizes and substantially expands the test suite. New unit tests cover interaction features (TTC/geometry), map-metric road-edge distances, the EvalManager config/dispatch/aggregation logic, and Python load_config behavior. Smoke coverage is moved under tests/smoke_tests/ and a deterministic CPU smoke training test with a committed golden file is added. CI workflows are consolidated into ci.yml plus a separate perf-ci.yml, and the old C INI parser harness and several legacy workflows/tests are removed. Path resolution in existing tests is corrected for the new tests/unit_tests/ depth.

Changes:

New unit tests for benchmark interaction/map metrics and EvalManager; new smoke training test with golden file; new simulator perf smoke.
Workflow consolidation (ci.yml, simplified install.yml/perf-ci.yml) and removal of utest.yml, train-ci.yml, training-test.yml, _cleanup.yml.
Removal of legacy tests (test_drive_train.py, test_drive_scenario_length.py) and the C INI parser test harness; parents[1]/two-dirname REPO_ROOT bumped to the new depth.

Reviewed changes

Copilot reviewed 20 out of 26 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/unit_tests/test_ttc.py	New TTC tests using 3-timestep central-difference setup
tests/unit_tests/test_geometry.py	New box signed-distance + invalid/multi-rollout tests
tests/unit_tests/test_map_metrics.py	New road-edge signed distance and full-pipeline tests
tests/unit_tests/test_road_edges.py	Manual visualization script without pytest test functions
tests/unit_tests/test_eval_manager.py	Extensive EvalManager parsing/dispatch/rollout/render tests
tests/unit_tests/test_drive_config.py	Python `load_config` tests (with inline-comment test skipped)
tests/unit_tests/test_single_agent_yaml.py	REPO_ROOT updated for new path depth
tests/unit_tests/test_map_cache.py	REPO_ROOT updated for new path depth
tests/unit_tests/test_drive_map_types.py	REPO_ROOT updated for new path depth
tests/smoke_tests/test_drive_train.py	New deterministic smoke training test with golden comparison
tests/smoke_tests/test_simulator_perf.py	New CI perf smoke (passes wrong `map_dir`)
tests/smoke_tests/test_validation_replay_html.py	REPO_ROOT updated for new path depth
tests/smoke_tests/data/drive_smoke_golden.json	Committed golden metrics for the smoke train test
tests/test_drive_train.py / tests/test_drive_scenario_length.py	Removed in favor of new smoke/map-type tests
tests/ini_parser/*	C INI parser test harness removed
.github/workflows/ci.yml	New consolidated unit + smoke job
.github/workflows/perf-ci.yml	Reworked to run new perf smoke
.github/workflows/install.yml	Simplified install matrix; pre-commit job dropped
.github/workflows/{utest,train-ci,training-test,_cleanup}.yml	Removed

Comments suppressed due to low confidence (2)

tests/smoke_tests/test_simulator_perf.py:17

The map_dir here points to resources/drive/binaries, but (a) the path is missing the pufferlib/ prefix that the bundled fixtures live under, and (b) that directory contains only sub-directories (carla/, nuplan/, …), not .bin files. Drive.__init__ resolves map_dir with os.listdir(...) filtering on .endswith(".bin"), so self.map_files will be empty and the constructor will fail (or the run will produce no maps). The perf CI job runs this file directly and will fail. Point map_dir at a concrete fixture directory such as pufferlib/resources/drive/binaries/carla.
tests/smoke_tests/test_simulator_perf.py:5
json, warnings, and Path are imported but never used. They can be removed to keep the smoke test minimal.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…ters

…; add pytest to dependencies

…ut test The train smoke golden drifted between machines because the test ran on async Multiprocessing (8 envs / batch 4, half in-flight): recv order depends on host core-count/scheduling, which reorders the PPO batch and flips sampled actions -> metrics diverge. The Docker pins fix float rounding but not the scheduler, so CI still mismatched. Switch the train test to the synchronous Serial backend, which is bit-identical across machines (verified twice in-image). Add test_drive_rollout.py: the training-free twin (same 5 x 64 env steps, seeded random actions, no policy/PPO) for a pure-simulator determinism check. Both goldens regenerated inside the pinned image; CMD runs both. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Even with Serial + BLAS/ISA pins, goldens still drifted across CPUs: torch CPU kernels are not bit-identical (2x train drift) and glibc libm uses IFUNC CPU dispatch (~6% drift on accumulated sim metrics). Fix by running the smoke tests inside the image under QEMU emulating a fixed CPU (-cpu Haswell): every host and dev box sees the same CPUID, so libm dispatch and torch kernels are identical and the golden is bit-reproducible everywhere. torch training is ~30x slower emulated, so shrink test_drive_train to 2 epochs / 32-wide net / 4 envs (was 5 / 64 / 8); both tests run in ~4 min under QEMU. Goldens regenerated under emulation; CI smoke timeout 20 -> 30 min. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

QEMU (-cpu Haswell) collapses the chaotic train divergence (200% -> <6%) and makes all rate/count metrics exact across hosts, but qemu-user still runs FP on the host CPU and glibc libm dispatches on the host AT_HWCAP, so accumulation metrics (loss sums, episode_return) keep ~5% host residue. 8% absorbs that while still catching real regressions; literal bit-exactness needs identical hardware. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…tors) CI showed that under QEMU only value_loss, ema_max and episode_return drift >1% across hosts; every other metric is <1%. Replace the blanket 8% with two tiers: 2% for the stable rate/count metrics, 8% only for those float-accumulation keys. Also add tests/smoke_tests/update_goldens.sh to regenerate goldens in-image. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

eugenevinitsky · 2026-05-31T21:10:13Z

+# Bit-exact CPU image for the PufferDrive smoke golden test.
+#
+# Option A: freeze every source of float divergence so the golden metrics in
+# data/drive_smoke_golden.json reproduce on ANY AVX2-capable x86-64 host
+# (CI runner or dev box). What is pinned:
+#   - compiler        : the gcc shipped in python:3.11-slim-bookworm
+#   - torch / numpy   : exact CPU wheels (no per-host wheel/kernel skew)
+#   - kernel ISA      : oneDNN + MKL/OpenBLAS capped to a fixed ISA / kernel
+#   - C sim float math: built baseline-ISA + -ffp-contract=off (see setup.py)
+#
+# Regenerate the golden (after an intentional pipeline change) INSIDE this image:
+#   docker build -f tests/smoke_tests/Dockerfile -t pufferdrive-smoke .
+#   docker run --rm -e SMOKE_UPDATE_GOLDEN=1 \
+#       -v "$PWD/tests/smoke_tests/data:/app/tests/smoke_tests/data" \
+#       pufferdrive-smoke
+# then commit tests/smoke_tests/data/drive_smoke_golden.json.


Claude narrative comment, just explain why it is the way it is

eugenevinitsky

Seems like it just moves the tests and add the golden tests? Sg

vcharraut added 2 commits May 31, 2026 01:23

Update tests

d6b8c93

Fix tests

35dff0e

Copilot AI review requested due to automatic review settings May 30, 2026 23:31

Copilot started reviewing on behalf of vcharraut May 30, 2026 23:31 View session

vcharraut added 2 commits May 31, 2026 01:34

Update CI workflows and test simulator configuration for improved per…

fdc0f3d

…formance

Consolidate performance tests into CI workflow and remove separate pe…

fe25a6f

…rf-ci.yml file

Copilot AI reviewed May 30, 2026

View reviewed changes

Comment thread tests/smoke_tests/test_drive_train.py Outdated

vcharraut and others added 12 commits May 31, 2026 01:38

Update Python version matrix in install workflow

daf80c2

Potential fix for pull request finding

4ae27dc

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Update Python version in pre-commit workflow to 3.13

fa04603

Reorder push and pull_request triggers in installation workflow

58bd6c9

Add dummy CPU smoke test for PufferDrive evaluation pipeline

37a1c7f

Update drive smoke test data and adjust training configuration parame…

08d0cee

…ters

Add smoke test workflow and Docker setup for reproducibility

ad3cbd3

Update Python version in installation workflow and Dockerfile to 3.14…

227b379

…; add pytest to dependencies

eugenevinitsky reviewed May 31, 2026

View reviewed changes

eugenevinitsky approved these changes May 31, 2026

View reviewed changes

eugenevinitsky merged commit 8725401 into emerge/temp_training May 31, 2026
11 checks passed

vcharraut deleted the vcha/smoke-tests branch June 16, 2026 00:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update tests#462

Update tests#462
eugenevinitsky merged 16 commits into
emerge/temp_trainingfrom
vcha/smoke-tests

vcharraut commented May 30, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

eugenevinitsky May 31, 2026

Uh oh!

eugenevinitsky left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

vcharraut commented May 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Restructure test suite & consolidate CI workflows

Summary

Test layout

CI workflows

Known trade-off / follow-ups

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

eugenevinitsky May 31, 2026

Choose a reason for hiding this comment

Uh oh!

eugenevinitsky left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vcharraut commented May 30, 2026 •

edited

Loading