Depth Supervision

[reynoldscem]

I saw in the recent commits that depth supervision was added. Could you please inform me how to go about using this or provide an example?

[Karbo123]

I have the same question, and I don't know whether the function of depth supervision have been finished. I have tried several time, and I just found that if we combine RGB image supervision with Depth supervision, the visualzation results will be even worse. My depth images are rendered by enabling testbed.render_mode = ngp.RenderMode.Depth, and saved as .npy format. I'm not sure if something is wrong for my depth maps. I used depth_scale=1.

[SSground]

@JasonLSC If you have depth images already, and if you put the depth images appropriately under a folder, and load these by providing the dataset config json file, you should probably set integer_depth_scale; but if you load them by calling Testbed::Nerf::Training::set_image and loading them from numpy array, then you should set depth_scale. I would highly encourage you to check the source codes, and figure out by yourself how it loads depth images into the CUDA/C++ backend.

If I have depth values from colmap, then do I need to normalize the depth map and add the depth_scale? I think the depth_scale should be tuned to match the sampling interval of depths, but I have no idea how the sampling interval of instant_ngp is determined.

Can you tell me how to get the depth values ​​that colmap finds. Are there any relevant script.

[chl2]

Does the depth information you are talking about mean that each image must have a depth image? Can I use colmap's 3d point, which is the overall depth information?

[Karbo123]

@chl2 Yes, for sparse depth supervision, maybe you can take a look at DSNeRF which uses sparse point cloud from COLMAP. For instant-ngp, you should provide depth images (you can render the point cloud into multi-view depth images).

[ddkats]

Hi guys,
When I enable depth supervision, the result seems to be worse (visualize black spots and blurry outcome). I think that this has to do to with the format of the depth images, I mean as far as I can understand ingp expects a float 16bit format in pixels, normalized at [0,1]. Is that correct? Can anyone provide an example depth map to see the format used? @Karbo123

(*the depth maps I use are generated by a stereo camera and I have corresponding depth and rgb images)

[kafei123456]

I set the transforms.json like that , but it have not loaded depth map. How to load depth map ?

[CherniGhassen]

Did you manage to add depth supervision? There's no explanation anywhere on how to add it

[CherniGhassen]

Thank you very much, this is helpful

[poyodiaz]

Did you manage to add depth supervision? There's no explanation anywhere on how to add it

Hi i added :

"enable_depth_loading": true,
"integer_depth_scale": 1,
"depth_path": "./depth/0002-standard-scale-4_00x.png",

to my transforms json file but it doesnt load at all could you please help me to understand what am i missing?

[ddkats]

Did you manage to add depth supervision? There's no explanation anywhere on how to add it

Hi i added :

"enable_depth_loading": true, "integer_depth_scale": 1, "depth_path": "./depth/0002-standard-scale-4_00x.png",

to my transforms json file but it doesnt load at all could you please help me to understand what am i missing?

You need to have corresponding depth map image for each input image in the transform.json. Then, when you open the gui (or headless) I think it's necessary to adjust depth_supervision value.

[poyodiaz]

ok i see , thanks ! so the json must contain the info on each frame along the whole file ?

[ddkats]

I think so...Also I'm waiting for a reply at this thread here:
#1095