Skip to content

README_DATA.md

Latest commit

 

History

History
179 lines (139 loc) · 8.14 KB

README_DATA.md

File metadata and controls

179 lines (139 loc) · 8.14 KB
Raw

Dataset preparation

This repo is evaluated on three datasets. For each one the flow is the same:

  1. Obtain the raw data (download the official release, or our packaged zip).
  2. Convert it to our format — a per-scene cameras.json plus resized images (scripts/preprocess/<dataset>.py).
  3. Predict per-view G-buffers (base color / metallic / normal / roughness) with DiffusionRenderer, which writes the *_single folders the inverse-rendering stage consumes (scripts/<dataset>/<dataset>_single_img_process.sh).

After step 3 a scene is ready for Stage 2 (neural surface reconstruction) and Stage 3 (PBIR) as described in the top-level README.md.

Dataset Source Raw → our format G-buffers
DTC-Synthetic (ours) zip on our GitHub release (already in our format) scripts/dtc/dtc_single_img_process.sh
Stanford-ORB Stanford-ORB (ground_truth.tar.gz, blender_HDR.tar.gz) scripts/preprocess/stanford_orb.py scripts/stanford_orb/stanford_orb_single_img_process.sh
Synthetic4Relight InvRender scripts/preprocess/mii.py scripts/mii/mii_single_img_process.sh

Naming note. Synthetic4Relight is referred to as mii throughout the code and scripts (its data/ folder is data/Synthetic4Relight/, but the driver scripts live under scripts/mii/). The two names refer to the same dataset.

Every command below is run from the repository root.

Our data format

Each scene is a directory containing a cameras.json and the referenced images:

<scene_name>
├── cameras.json
└── ... images referenced by cameras.json ...

cameras.json holds the intrinsics, the foreground AABB, the train/test split, and the per-frame camera-to-world matrices:

{
    "fx": 723.36, "fy": 723.36, "cx": 450.0, "cy": 337.5,
    "aabb": [[-1.5, -1.5, -1.5], [1.5, 1.5, 1.5]],
    "split": { "train": [1, 2, 3, 5, 6, 7, 9], "test": [0, 4, 8] },
    "frames": [
        {
            "path": "images/im_0001.exr",
            "mask": "optional/path/to/mask.png",
            "mask_from_alpha": false,
            "to_world": [[ "... 4x4 ..." ]]
        }
    ]
}
  • fx, fy, cx, cy — pinhole intrinsics.
  • aabb — axis-aligned box enclosing the foreground object; everything outside is modeled by the background field.
  • frames — per-view relative image path + optional mask + to_world (camera-to-world).

Step A — DiffusionRenderer (G-buffer prediction)

G-buffers are predicted by DiffusionRenderer (NVIDIA Cosmos DiffusionRenderer), vendored as a submodule at diffusion_reg/deps/cosmos1-diffusion-renderer.

# 1. Fetch the submodule (skip if you cloned with --recursive)
git submodule update --init diffusion_reg/deps/cosmos1-diffusion-renderer

# 2. Create its conda env and download the 7B checkpoints, following the submodule's
#    own instructions: diffusion_reg/deps/cosmos1-diffusion-renderer/INSTALL.md
#    (creates a "cosmos-predict1" env; requires a Hugging Face token for the weights)
cd diffusion_reg/deps/cosmos1-diffusion-renderer
conda env create --file cosmos-predict1.yaml         # see INSTALL.md for the full recipe
CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) \
    python scripts/download_diffusion_renderer_checkpoints.py --checkpoint_dir checkpoints
cd ../../../..

Our per-dataset *_single_img_process.sh scripts wrap the three sub-steps: export_train_images.py (stage the train views into the submodule) → run_inverse_renderer_image_sample.sh (run the inverse renderer) → dtc_diffrender_single.py (assemble the basecolor/metallic/normal/roughness _single folders). They invoke the inverse renderer with conda run -n cosmos-predict1.1; if you named the env differently (e.g. cosmos-predict1 from INSTALL.md), set COSMOS_ENV:

COSMOS_ENV=cosmos-predict1 bash scripts/mii/mii_single_img_process.sh <scene>

The inverse renderer needs a CUDA GPU (~27 GB peak). Running it on all training views of a scene takes on the order of an hour on a single A40/A100.

Step B — per-dataset preparation

Run the python scripts/preprocess/*.py conversions in the neural-pbir conda env (the Stage-2 env from the top-level README.md). The *_single_img_process.sh scripts manage their own conda envs internally (neural-pbir for the wrappers, the cosmos env for the inverse renderer) and should be launched from the repository root.

DTC-Synthetic (our dataset)

Download the packaged case(s) from our GitHub release and extract under data/dtc/adt_dslr/. The release ships each case already in our format (cameras.json, synthetic_exrs_800/, mask_800/, mesh_transformed.obj + material.mtl for the GT shape, env.exr for the GT illumination, and relit/ — the ground-truth relit images the relight-metric evaluation compares against), so no raw conversion is needed — just predict the G-buffers.

We recommend starting with the example case TeaPot_B094FQW6Q4_EmeraldGoldTop_scene002:

mkdir -p data/dtc/adt_dslr
cd data/dtc/adt_dslr
wget https://github.com/gerwang/DiffReg-PBIR/releases/download/dtc-synthetic-v1/DTC-Synthetic_TeaPot_B094FQW6Q4_EmeraldGoldTop_scene002.zip
unzip DTC-Synthetic_TeaPot_B094FQW6Q4_EmeraldGoldTop_scene002.zip
cd ../../..

# predict the G-buffers for the extracted case
bash scripts/dtc/dtc_single_img_process.sh TeaPot_B094FQW6Q4_EmeraldGoldTop_scene002

All seven scenes used in the paper are available on the dtc-synthetic-v1 release; download any other case the same way and pass its scene name to the scripts.

Stanford-ORB

Download ground_truth.tar.gz and blender_HDR.tar.gz from the Stanford-ORB release. You only need the object(s) you want to evaluate; relighting requires all lighting scenes of an object (e.g. the cup object's cup_scene00N captures). Extract just those scenes:

mkdir -p data/stanford_orb
# The cup object has lighting scenes 003, 006, 007 (scene numbers are not contiguous and differ
# per object — list an object's scenes with `tar tzf ground_truth.tar.gz | grep cup_scene`).
tar xzf blender_HDR.tar.gz  -C data/stanford_orb  blender_HDR/cup_scene00{3,6,7}
tar xzf ground_truth.tar.gz -C data/stanford_orb  ground_truth/cup_scene00{3,6,7}

The raw blender_HDR/<scene> holds transforms_{train,test,novel}.json and the train/ test/ train_mask/ test_mask/ frames; ground_truth/<scene> holds the GT env maps, mesh, normals and depth used for evaluation. Convert to our format and predict G-buffers per scene:

python -m scripts.preprocess.stanford_orb data/stanford_orb/blender_HDR/cup_scene006
bash   scripts/stanford_orb/stanford_orb_single_img_process.sh cup_scene006
# Only the scene you train (cup_scene006) needs preprocessing + G-buffers. Its sibling lighting
# scenes (cup_scene003, cup_scene007) just supply the novel-light ground-truth images the
# relighting metric compares against — extracting them (above) is enough.

Synthetic4Relight (mii)

Download the Synthetic4Relight scenes (air_baloons, chair, hotdog, jugs) from InvRender and extract under data/Synthetic4Relight/:

data/Synthetic4Relight/<scene>
├── transforms_train.json
├── transforms_test.json
├── train/
├── test/
└── test_rli/        # relighting test views

Convert to our format and predict G-buffers:

python -m scripts.preprocess.mii data/Synthetic4Relight/hotdog
bash   scripts/mii/mii_single_img_process.sh hotdog

Worked examples

data/ is not tracked in git — a fresh clone ships it empty. Obtain the data with the download commands above and extract it under data/ in the layout shown. The commands use one worked example per dataset (the paper's teaser cases):

  • data/dtc/adt_dslr/TeaPot_B094FQW6Q4_EmeraldGoldTop_scene002/ — DTC-Synthetic, our format.
  • data/stanford_orb/{blender_HDR,ground_truth}/cup_scene00{3,6,7}/ — Stanford-ORB, raw.
  • data/Synthetic4Relight/hotdog/ — Synthetic4Relight, raw.