GT 16 bits image support + lossless 16bits cache for HDR training [VOLINGA]

[AlbertoValenzuelaRedondo]

This set of commits introduces support for 16-bit and FP32 image workflows, making the image pipeline more flexible and better prepared for HDR and high-precision image inputs.

The loading path has been improved to support u16 and FP32 images, with internal templating added to load_image_t to make the implementation cleaner and easier to extend. A new CUDA kernel was also added to convert FP32 images from HWC to CHW layout directly on the GPU.

JPEG2000 support has been added for lossless encoding, with the correct encoding selected depending on the active configuration. The decoding path has also been advanced, including work on 16-bit decoding and support for float tensors during Lanczos resize.

A new --use_8bit_color flag has been introduced to easily switch between standard 8-bit input and HDR 16-bit image input by disabling it with 0.

To build and compile these changes correctly, JPEG2000 development support needs to be installed on the target machine, including the required libraries and headers.

We merged this based on an old commit, please doble check that we are not breaking any new functionality added in between.

[MrNeRF]

Thank you. I will review it carefully.

There is nvimagecodec as dependency in this repo and it supports JPEG2000. However, I have stripped down the repo aggressively so I need to tests if that's still working.

[MrNeRF]

I have tested it. In general it works but the reconstruction is very poor. Do you have any images you can share where it outperforms the standard training path.

[AlbertoValenzuelaRedondo]

Hi Janusch,
First, let's make sure that the training pipeline with 16-bit reading and 16-bit lossless caching is functioning correctly.

Although the jpeg2k dependency is listed as a dependency for nv_image_codec, it fails silently if the dynamic libraries are not found at runtime. As a result, training runs slower because it bypasses the cache and reads directly from the disk in the original format. Please note that the compilation process does not copy the required DLLs (like nvjpeg2k_0.dll) next to the LFS executable by default (though the install process does).

Steps to test on Windows:

1. Install nv_jpeg2k v0.9.0 on the system (v0.10.0 causes compilation failures with the current version of nv_image_codec in the repo).
2. In CMake, specify the path for the nvjpeg2k headers and library (I have attached an image with the extra configuration).
3. Run Configure and Generate in CMake.
4. Compile
5. Ensure that the dynamic library (nvjpeg2k_0.dll next to the .exe) and the extension for nv_image_codec (nvjpeg2k_ext_0.dll in the extensions subfolder) are visible to the executable. If not, copy them manually, as their absence causes a silent failure.
6. Run LFS_executable.exe --use_8bit_color 0

For training, this changes the pipeline from:
Load EXR -> Convert to 8-bit JPEG -> Cache -> Convert to FP32 for the training GT image
to:
Load EXR -> Convert to 16-bit lossless JPEG2K -> Cache -> Convert to FP32 for the training GT image

Better quality is preserved in the step from the cache to the training GT image, meaning the training results will better represent the original EXR values. This comparison cannot be made using the LOSS, since LOSS compares the render vs. the GT, and the GT has already lost information during the conversion process from the cache.

In our internal testing using MCMC method, the original EXR color is better preserved, although training times are affected, with performance dropping to nearly half.

If necessary, we can provide a set of EXR training images with the appropriate encoding through another channel.

Let us know if you have any questions.

Thank you very much.
Regards,
Alberto.

[MrNeRF]

@AlbertoValenzuelaRedondo Ok, thank you. I will merge this shortly and try to fix the jpeg so that it works based on the external dependency that is shipped.

I will also switch the flag so it will be enabled with an explicit flag something like --use-16bit

Maybe I will ping you later for a dataset if possible.