Could this be helpfull?

[noobtoob4lyfe]

This looks like it could be promising for your purposes https://github.com/Eyeline-Labs/Vista4D. Especially if you use as much of the point cloud/original pixels as possible for the synthetic view point. In other words, only use the generative pixels for the occluded areas. They even have a clever solution for long duration shots. Would love to know what you think.

[calledit]

Vista4D looks amazing. But running it on anything less than a A100 would be hard.

A good thing is that the manual work of masking dynamic objects in the scene that vista4D requires could be excluded from the Vista4d pipeline if it was used for a stereo 3D senario.

But the main issue is probably this:

Vista4D has the same type of issue that inspatio-world has where things in the render and the infilled output wont align due to how they both reposition dynamic objects to compensate for inaccurate depth from the depth model, you can see it in the examples if you play them frame by frame (Vista4D is actually even worse than inspatio-world when it comes to this aspect). I spent quite a bit of time trying to counteract this issue in inspatio_world_infill.py but to get anything good it really has to be solved in the model.

This type of alignment issue makes both models quite bad for Stereo3D where you need both eyes to actually see the same scene. If you move an object in the left frame and move it differently in the right things looks very unpleasant.

At the moment most of my efforts are going towards training my own infill model https://github.com/calledit/one_shot_stereo

[noobtoob4lyfe]

Thanks for the info. Looking forward to seeing more of your infill model!