TSDF Integration#

Truncated Signed Distance Function (TSDF) integration is the key of dense volumetric scene reconstruction. It receives relatively noisy depth images from RGB-D sensors such as Kinect and RealSense, and integrates depth readings into the Voxel Block Grid given known camera poses. TSDF integration reduces noise and generates smooth surfaces.

The integration process mainly consists of two steps, (sparse) block selection and activation, and (dense) voxel value integration. An example can be found at examples/python/t_reconstruction_system/integrate.py.

Activation#

In the activation step, we first locate blocks that contain points unprojected from the current depth image. In other words, it finds active blocks in the current viewing frustum. Internally, this is achieved by a frustum hash map that produces duplication-free block coordinates, and a block hash map that activates and query such block coordinates.

51# examples/python/t_reconstruction_system/integrate.py
52        frustum_block_coords = vbg.compute_unique_block_coordinates(
53            depth, depth_intrinsic, extrinsic, config.depth_scale,
54            config.depth_max)

Integration#

Now we can process the voxels in the blocks at frustum_block_coords. This is done by projecting all such related voxels to the input images and perform a weighted average, which is a pure geometric process without hash map operations.

We may use optimized functions, along with raw depth images with calibration parameters to activate and perform TSDF integration, optionally with colors:

55# examples/python/t_reconstruction_system/integrate.py
56        if config.integrate_color:
57            color = o3d.t.io.read_image(color_file_names[i]).to(device)
58            vbg.integrate(frustum_block_coords, depth, color, depth_intrinsic,
59                          color_intrinsic, extrinsic, config.depth_scale,
60                          config.depth_max)
61        else:
62            vbg.integrate(frustum_block_coords, depth, depth_intrinsic,
63                          extrinsic, config.depth_scale, config.depth_max)

Currently, to use our optimized function, we assume the below combinations of data types, in the order of depth image, color image, tsdf in voxel grid, weight in voxel grid, color in voxel grid in CPU

212template void IntegrateCPU<uint16_t, uint8_t, float, uint16_t, uint16_t>(
213        FN_ARGUMENTS);
214template void IntegrateCPU<uint16_t, uint8_t, float, float, float>(
215        FN_ARGUMENTS);
216template void IntegrateCPU<float, float, float, uint16_t, uint16_t>(
217        FN_ARGUMENTS);
218template void IntegrateCPU<float, float, float, float, float>(FN_ARGUMENTS);

and CUDA

238template void IntegrateCUDA<uint16_t, uint8_t, float, uint16_t, uint16_t>(
239        FN_ARGUMENTS);
240template void IntegrateCUDA<uint16_t, uint8_t, float, float, float>(
241        FN_ARGUMENTS);
242template void IntegrateCUDA<float, float, float, uint16_t, uint16_t>(
243        FN_ARGUMENTS);
244template void IntegrateCUDA<float, float, float, float, float>(FN_ARGUMENTS);

For more generalized functionalities, you may extend the macros and/or the kernel functions and compile Open3D from scratch to achieve the maximal performance (~100Hz on a GTX 1070), or follow Customized Integration and implement a fast prototype (~25Hz).

Surface extraction#

You may use the provided APIs to extract surface points.

105# examples/python/t_reconstruction_system/integrate.py
106    pcd = vbg.extract_point_cloud()
107    o3d.visualization.draw([pcd])
108
109    mesh = vbg.extract_triangle_mesh()
110    o3d.visualization.draw([mesh.to_legacy()])

Note extract_triangle_mesh applies marching cubes and generates mesh. extract_point_cloud uses a similar algorithm, but skips the triangle face generation step.

Save and load#

The voxel block grids can be saved to and loaded from .npz files that are accessible via numpy.

47# examples/python/t_reconstruction_system/integrate.py
48    for i in tqdm(range(n_files)):
49

The .npz file of the aforementioned voxel block grid contains the following entries:

  • attr_name_tsdf: stores the value buffer index: 0

  • attr_name_weight: stores the value buffer index: 1

  • attr_name_color: stores the value buffer index: 2

  • value_000: the tsdf value buffer

  • value_001: the weight value buffer

  • value_002: the color value buffer

  • key: all the active keys

  • block_resolution: 8

  • voxel_size: 0.0059 = 3.0 / 512

  • CUDA:0: the device