Depth Data Visualization

Depth images are beautiful but also problematic. Visualization is key to seeing what is in the scene, and certain methods can make the process faster. If you can try your eyes to understand depth images with extra information, failure analysis could be done more efficiently. This data is from the NYU Depth V1 Dataset (Silberman, N., Fergus, R.: Indoor scene segmentation using a structured light sensor. In: ICCV Workshop on 3D Representation and Recognition. (2011) 

Left: RGB Gray , Right: Depth Image

I inserted a more complex 3D object, which is a face from my dissertation work. The quality of the depth of the face is quite bad (can't recognize the person), but it still is useful to show how the complexity of that depth image could be shown as well with a depth image with a large variance of depth measurements. In this case, the gray representation of depth shows the face as a flat object.

Left: RGB , Right: Gray Depth Image with highlighted low quality 3D face

Let's go to Jet because the blue to red color scheme makes some of the data easier to see

Left: Gray Depth , Right: Jet colormap (courtesy to Matlab)

Let's take a look at shifting the different pieces of information. It looks like the data in the back is unreliable because it it showing the data is close up, but in the RGB image, we know the depth is far away. A bit more detail can be seen on the chairs and the face.

Left: Focus on the foreground , Right: Focus on the background

How about an HSV color mapping instead of Jet, just to see.

Left: Jet colormap , Right:HSV colormap

Why not use the same colormap for multiple depth boxes? This can be done by using a mod colormap. Color mod is done in the lines below. It causes the colormap to be used multiple times in the N interval of points. This allows more depth information to be shown in the same image.

N = 30000; %Max is uint16 (65,535)

depthMod = mod(depth,N)/N;

Left: Jet , Right: Mod Jet

The main issue with this is that at the transition points, the color jumps from blue to red, which is not appealing to the eye. If we modify the colormap to go from blue to red to blue (Jet+), that problem is solved (below):

Left: Mod Jet , Right: Mod Jet+

Let's change N form 30,000 to 20,000 for a little more clarity:

Left: Mod Jet+ at N=30,000 , Right: Mod Jet+ N=20,000

Still, there are some background pixels that aren't helpful. These are either too close (<1500 in this case) or too far away (== 65,535). If we set these to black, it gives it another view:

Left: Mod Jet+ at N=20,000 , Right: Mod Jet+ N=20,000 dropped useless pixels

We can even scale the colors by the original depth to help make a different between two things that are the same color but different depths.

N1=0.2; Scaled = JetImage.*((original.^N1));

Left: Mod Jet+ at N=20,000 dropped useless pixels , Right: Mod Jet+ N=20,000 dropped useless pixels Scaled

This brings out that added face artifact, and the scene is a little easier to interpret. Here is the final comparison before and after. I love Jet if it wasn't clear already.

Left: Original Gray , Right: Mod Jet+ N=20,000 dropped useless pixels Scaled