support Scan4D semantic labeling

[mwfarb]

We should consider adding the ARKit furniture/objects labeling system. In that way some of our overlay views on mobile may render for rough understanding of active labeling. More importantly perhaps, our exports could include this semantic labeling for free for downstream pipeline consumption.

Per slack: @anthonyrowe @hi-liang @sagar-bharadwaj-ks

[mwfarb]

Issue #2 — Semantic Labeling via ARKit Mesh Classification

Verdict: ✅ Very feasible — low-to-medium effort

ARKit provides ARMeshClassification out of the box on LiDAR devices. The app already uses .mesh scene reconstruction; switching to .meshWithClassification is a one-line config change per call site. The classification data then arrives automatically on every ARMeshAnchor.

What ARKit gives you for free

Per-face labels for 8 categories: floor, wall, ceiling, table, seat, door, window, none. This is a per-triangle UInt8 buffer on ARMeshGeometry.classification, available at the same cadence as the mesh geometry the app already processes.

Integration points in the current codebase

Touch point	What changes	Effort
ARCoverageView.swift — config sites (lines 252, 336, 1337)	Change .mesh → .meshWithClassification	Trivial
ARCoverageView.swift — buildWireframeForAnchor (line 610)	Read geometry.classification buffer alongside vertices/faces. Color wireframe edges by category	Medium
ARCoverageView.swift — exportMeshOBJ (line 1207)	Emit per-face classification as OBJ groups (g wall, g floor, ...) or a sidecar JSON	Medium
FrameCaptureSession.swift — writeScan4DMetadata (line 672)	Include a summary of classified area (e.g. wall m², floor m²) in scan4d_metadata.json	Low

Risk assessment

Note

.meshWithClassification requires the same LiDAR hardware as .mesh. The app already gates on supportsLiDAR, so no new hardware requirement.

Warning

Performance: Apple docs note that classification adds a small per-anchor cost. The existing 0.5s wireframe throttle (wireframeThrottleInterval) and background geometry extraction pattern should absorb it, but worth profiling on a large scan.

What it doesn't give you

ARKit classification is limited to architectural elements + furniture. It does not provide:

• Object-level instance segmentation (e.g. "this is lamp support live Scan4D coverage suggestions #3")
• Fine-grained furniture taxonomy (e.g. "sofa" vs "armchair" — both are .seat)
• Outdoor terrain labels (grass, pavement, etc.)

For richer semantics you'd need Vision framework's VNGeneratePersonSegmentationRequest (already used for privacy) or a custom CoreML model, which is a much larger effort and probably a separate issue.