[BUG] Gaussians' means is not properly calculated.

[scpoppin1207]

Bug Report: Inconsistent Depth Representation in GaussianAdapter Leading to Distorted Gaussians

Summary

Issues #236 and #100 report distortion in Gaussian representations produced by GaussianAdapter. After investigation, the root cause appears to be inconsistent handling of depth representations (z-depth vs. ray distance), along with a potentially misbehaving depth_offset.

---

Observations

1. Mismatch in Depth Representation

   • GaussianAdapter.forward takes output.depth as input.
   • This depth is z-depth (camera coordinate), but later computations treat it as ray distance when deriving Gaussian means.
   • This inconsistency leads to incorrect spatial positioning.

2. Incorrect Use of depth_offset

   • depth_offset = raw_gaussians[..., -1] is directly added to gs_depths.
   • Evidence suggests depth_offset is likely defined in ray distance space, causing further inconsistency when combined with z-depth.

3. Empirical Validation
   Several fixes were tested:

   • Converting depth_offset into z-depth offset before adding → still incorrect
   • Converting output.depth to ray distance before adding depth_offset → still incorrect

   These results suggest:

   • The issue is not only representation mismatch
   • depth_offset itself is likely poorly learned or unstable, contributing significantly to the distortion

---

Proposed Fix

1. Use z-depth for unprojection

   • Compute Gaussian means via z-depth-based unprojection, ensuring consistency with the original depth definition.

2. Disable depth_offset

   • Given its unstable behavior and lack of reliable contribution, removing or disabling depth_offset improves correctness.

---

Additional Notes

• The problem likely stems from mixed assumptions about depth parameterization during training vs. inference.
• A more robust solution may require revisiting how depth_offset is supervised or parameterized.

---

Expected Outcome

Applying the above fixes should eliminate the observed Gaussian distortion and produce geometrically consistent Gaussian means.

---

Please let me know if further experiments or ablations are needed.

[scpoppin1207]

The Gaussians AFTER debug

The Gaussians BEFORE debug

[scpoppin1207]

4.16.2.mp4

[lbh666]

我尝试过微调DA3的3DGS，loss后期一直会震荡，我微调anyspslat和worldmirror就不会有问题，他们结构都是类似的，差别比较大的就是depth_offset的处理。可视化DA3的3DGS点云也能看到很多漂浮的点。

[scpoppin1207]

我尝试过微调DA3的3DGS，loss后期一直会震荡，我微调anyspslat和worldmirror就不会有问题，他们结构都是类似的，差别比较大的就是depth_offset的处理。可视化DA3的3DGS点云也能看到很多漂浮的点。

似乎确实是offset自由度太多，导致DA3的GS头光拟合外去了