TSDF Integration

Truncated Signed Distance Function (TSDF) integration is the key of dense volumetric scene reconstruction. It receives relatively noisy depth images from RGB-D sensors such as Kinect and RealSense, and integrates depth readings into the Voxel Block Grid given known camera poses. TSDF integration reduces noise and generates smooth surfaces.

The integration process mainly consists of two steps, (sparse) block selection and activation, and (dense) voxel value integration. An example can be found at examples/python/t_reconstruction_system/integrate.py.

Activation

In the activation step, we first locate blocks that contain points unprojected from the current depth image. In other words, it finds active blocks in the current viewing frustum. Internally, this is achieved by a frustum hash map that produces duplication-free block coordinates, and a block hash map that activates and query such block coordinates.

82
83
84
85
# examples/python/t_reconstruction_system/integrate.py
            frustum_block_coords = vbg.compute_unique_block_coordinates(
                depth, depth_intrinsic, extrinsic, config.depth_scale,
                config.depth_max)

Integration

Now we can process the voxels in the blocks at frustum_block_coords. This is done by projecting all such related voxels to the input images and perform a weighted average, which is a pure geometric process without hash map operations.

We may use optimized functions, along with raw depth images with calibration parameters to activate and perform TSDF integration, optionally with colors:

86
87
88
89
90
91
92
93
# examples/python/t_reconstruction_system/integrate.py
            if config.integrate_color:
                color = o3d.t.io.read_image(color_file_names[i]).to(device)
                vbg.integrate(frustum_block_coords, depth, color,
                              depth_intrinsic, color_intrinsic, extrinsic,
                              config.depth_scale, config.depth_max)
            else:
                vbg.integrate(frustum_block_coords, depth, depth_intrinsic,

Currently, to use our optimized function, we assume the below combinations of data types, in the order of depth image, color image, tsdf in voxel grid, weight in voxel grid, color in voxel grid in CPU

229
230
231
232
233
234
template void IntegrateCPU<uint16_t, uint8_t, float, uint16_t, uint16_t>(
        FN_ARGUMENTS);
template void IntegrateCPU<uint16_t, uint8_t, float, float, float>(
        FN_ARGUMENTS);
template void IntegrateCPU<float, float, float, uint16_t, uint16_t>(
        FN_ARGUMENTS);

and CUDA

255
256
257
258
259
260
template void IntegrateCUDA<uint16_t, uint8_t, float, uint16_t, uint16_t>(
        FN_ARGUMENTS);
template void IntegrateCUDA<uint16_t, uint8_t, float, float, float>(
        FN_ARGUMENTS);
template void IntegrateCUDA<float, float, float, uint16_t, uint16_t>(
        FN_ARGUMENTS);

For more generalized functionalities, you may extend the macros and/or the kernel functions and compile Open3D from scratch achieve the maximal performance (~100Hz on a GTX 1070), or follow Customized Integration and implement a fast prototype (~25Hz).

Surface extraction

You may use the provided APIs to extract surface points.

135
136
137
138
139
140
# examples/python/t_reconstruction_system/integrate.py
    color_intrinsic = load_intrinsic(config, 'color')

    extrinsics = load_extrinsics(config.path_trajectory, config)
    vbg = integrate(depth_file_names, color_file_names, depth_intrinsic,
                    color_intrinsic, extrinsics, config)

Note extract_triangle_mesh applies marching cubes and generate mesh. extract_point_cloud uses the similar algorithm, but skips the triangle face generation step.

Save and load

The voxel block grids can be saved to and loaded from .npz files that are accessible via numpy.

47
48
49
# examples/python/t_reconstruction_system/integrate.py
        vbg = o3d.t.geometry.VoxelBlockGrid.load(config.path_npz)
        print('Saving to {}...'.format(config.path_npz))

The .npz file of the aforementioned voxel block grid contains the following entries:

  • attr_name_tsdf: stores the value buffer index: 0

  • attr_name_weight: stores the value buffer index: 1

  • attr_name_color: stores the value buffer index: 2

  • value_000: the tsdf value buffer

  • value_001: the weight value buffer

  • value_002: the color value buffer

  • key: all the active keys

  • block_resolution: 8

  • voxel_size: 0.0059 = 3.0 / 512

  • CUDA:0: the device