You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(usage): add Open3D RANSAC baseline (§6) + format polish (#99)
* docs(usage): add Open3D RANSAC baseline (§6) + format polish
- python/examples/evaluate_ransac_in_semantickitti.py: new eval driver
built on open3d.geometry.PointCloud.segment_plane. Same metric
definitions and --eval_protocol flag as evaluate_semantickitti.py,
with --distance_threshold / --num_iterations knobs and an optional
--sweep_thresholds / --sweep_iterations grid. Per-frame median ms
is reported alongside P/R/F1.
- USAGE.md §6: full 6×5 grid (thr ∈ {0.10..0.50}, iter ∈ {100..10000})
on KITTI seq 00 — F1 saturates between iter=500 and iter=1000 (the
highest-iter cell only buys +0.07 F1 anywhere in the table), F1 ridge
is at thr=0.15. Best config (thr=0.15, iter=1000) evaluated on full
KITTI 00-10 gives macro P=94.18 / R=82.03 / F1=87.11 at 19.5 ms
median per frame — +9.18 F1 behind Patchwork++ on the macro and
-25.88 F1 on the worst sequence (seq 10, rolling rural).
- USAGE.md top: full README-style centered header block with badges,
demo gif, pip-install banner (matches README.md for consistency).
- USAGE.md section headings now use ## :emoji: N. ... form per the
format-readme template; existing 70-underscore dividers retained.
No algorithmic change; new script + docs only.
* style: apply black to evaluate_ransac_in_semantickitti.py
[Patchwork++][arxivlink], an extension of [Patchwork][patchworklink], is **a fast, robust, and self-adaptive ground segmentation algorithm** on 3D point cloud.
The Patchwork and Patchwork++ papers use **different** ground-truth definitions on SemanticKITTI. The eval driver `python/examples/evaluate_semantickitti.py` supports both via `--eval_protocol {patchwork, patchworkpp}`.
16
56
@@ -54,7 +94,7 @@ Same Patchwork++ inference, KITTI 00–10 macro average, two protocols:
If results look wrong on a new sensor (Velodyne 16/32, Ouster 64/128, Livox, etc.), tune in roughly this order. Defaults are in `cpp/patchworkpp/include/patchwork/patchworkpp.h` (Patchwork++) and `cpp/patchwork/include/patchwork/patchwork.h` (classic Patchwork).
60
100
@@ -109,7 +149,7 @@ Rule of thumb: scale them ∝ `expected_terrain_undulation / 1.723 m` if your se
KITTI 00-10 full sweep, **23,201 frames**, macro-average across the eleven sequences. All numbers are produced by `python/examples/evaluate_semantickitti.py` on current `master` (v1.3.1) with paper-matched parameters (the script already sets `uprightness_thr=0.707` and `using_global_thr=false` for `--method patchwork`; `--method patchworkpp` uses library defaults).
All numbers below are produced by `python/examples/evaluate_semantickitti.py` on v1.3.1 (current `master`), KITTI 00-10, paper-matched parameters. Use them to debug per-sequence regressions: if seq 05 looks fine but seq 10 is 3 F1 below the table, you have a parameter problem, not a code problem.
214
254
@@ -288,8 +328,116 @@ All numbers below are produced by `python/examples/evaluate_semantickitti.py` on
A common first instinct on a new dataset is to fit a single plane with RANSAC and call the inliers "ground". `python/examples/evaluate_ransac_in_semantickitti.py` does exactly that, on top of Open3D's `segment_plane`, with the same metric definitions and `--eval_protocol` flag as `evaluate_semantickitti.py`, so the numbers drop directly into the same comparison frame as §5.
334
+
335
+
```bash
336
+
# Single (thr, iter) point — defaults to thr=0.15, iter=500
`distance_threshold` (rows) is the max point-to-plane distance counted as inlier (metres). `num_iterations` (columns) is the RANSAC hypothesis cap; Open3D's `segment_plane` early-terminates when a hypothesis crosses an internal confidence bound, so this is a **maximum** not an exact iteration count. `ransac_n=3` throughout (plane). Cell value is **F1 (%)**; second line is the **median wall-clock ms** of `segment_plane` per frame.
Wall-clock numbers are median per-frame ms of `segment_plane` on an i7-12700; the 24-thread parallel default of Open3D is used for iter ≤ 1000, and 8 threads (`OMP_NUM_THREADS=8`) for iter ≥ 5000 (the 24-thread iter=10000 run exhausted system memory). Compare F1 numbers across columns freely; absolute ms across iter≤1000 and iter≥5000 columns are not directly comparable.
364
+
365
+
### Reading the grid
366
+
367
+
-**`distance_threshold` is the dominant knob, and the F1 column has a clear inverted-U.** Tight thresholds (0.10 m) over-reject — precision saturates near 96.7 but recall caps at 83. Loose thresholds (0.40–0.50 m) over-accept — precision falls below 81. The F1 ridge sits firmly at **thr=0.15**, no matter how many iterations RANSAC is allowed.
368
+
-**`num_iterations` saturates between 500 and 1000.** Going from 100 → 500 buys 3–6 F1; 500 → 1000 buys 0.0–0.6; 1000 → 10000 buys at most **+0.07 F1** anywhere in the table — well inside run-to-run noise. Open3D's early-termination is the cause: for thr ≥ 0.15 the wall-clock barely moves between iter=1000 and iter=10000, confirming that the inner loop stops on its own well before the cap. Only thr=0.10 keeps the loop running to the cap (37.5 → 56.7 ms going 1000 → 10000), and even there F1 changes by 0.02.
369
+
-**The dominant-plane assumption is the ceiling.** The best cell on the entire grid is `thr=0.15, iter=10000 → F1=93.35`, indistinguishable from `thr=0.15, iter=1000 → F1=93.28`. Practically there is no high-iter config that meaningfully improves on the cheap one; the algorithmic ceiling is set by the single-plane model, not by RANSAC's iteration budget.
370
+
371
+
### Best config on the full KITTI 00–10 sweep
372
+
373
+
Picking `thr=0.15, iter=1000` (ties the highest-iter F1 at this threshold, runs faster) and evaluating on all 23,201 frames under the Patchwork++ paper protocol:
**Patchwork++ wins by +9.18 F1 on the macro average** and roughly **matches** RANSAC on wall-clock per frame (~18 ms vs. ~19.5 ms), even though Patchwork++ is currently single-threaded on v1.4.0 (TBB intentionally disabled; see #96) while Open3D's `segment_plane` is using all 24 cores. The recall column is where the gap concentrates: RANSAC's 82.03 vs. Patchwork++'s 97.16 — a single global plane simply cannot cover the multiple ground patches that the concentric-zone partition handles natively.
403
+
404
+
### Per-sequence gap to Patchwork++
405
+
406
+
The macro gap is not uniform; it is dragged down by the hard sequences:
| 10 | rough rural / rolling roads |**67.75**|**93.63**|**-25.88**|
421
+
422
+
Sequences with a gap below 5 F1 (00, 01, 04, 05, 07) are essentially flat with a single dominant ground plane — exactly where the single-plane assumption holds. Sequences with a gap above 10 F1 (02, 03, 06, 08, 10) all have rolling shoulders, multi-tier sidewalks, or rough off-road terrain — multiple ground patches that one plane cannot represent. Seq 10 is the extreme case: rolling rural terrain where one global plane is so wrong RANSAC drops below 70 F1 while Patchwork++ stays above 93 F1.
423
+
424
+
### Takeaway
425
+
426
+
RANSAC is the obvious sanity-check baseline for ground segmentation. On KITTI it is **9 F1 behind the macro Patchwork++ row, 26 F1 behind on the worst sequence, and no improvement at higher iteration counts can close that gap** — the bottleneck is the model, not the optimiser. The concentric-zone partition that Patchwork and Patchwork++ both use turns this from a hard problem (one plane for the whole scan) into many easy ones (one plane per patch, with per-patch flatness and elevation gates), which is what closes the gap.
427
+
428
+
### Caveats
429
+
430
+
-`ransac_n=3` (plane) is the only value tested. Higher `ransac_n` fits higher-order surfaces and is out of scope here.
431
+
- The grid timing uses Open3D's default thread pool at iter ≤ 1000 and an 8-thread cap at iter ≥ 5000 (memory pressure at 24 threads × iter=10000 forced the cap). F1 numbers are insensitive to thread count; wall-clock numbers between low-iter and high-iter columns are **not** directly comparable. The full-KITTI Patchwork++ comparison row uses 24 threads on both sides.
432
+
- The Patchwork++ wall-clock row above (~18 ms median, single-threaded) is conservative. Enabling TBB on the Patchwork++ side (currently disabled — see #96) is expected to roughly halve it and widen the Hz gap further. Classic Patchwork (v1.4.0) is already TBB-parallel and runs at ~9 ms median on this machine.
0 commit comments