Ray Casting in a Voxel Block Grid

Note

This is NOT ray casting for triangle meshes. Please refer to open3d.t.geometry.RaycastingScene for that use case.

Ray casting can be performed in a voxel block grid to generate depth and color images at specific view points without extracting the entire surface. It is useful for frame-to-model tracking, and for differentiable volume rendering.

We provide optimized conventional rendering, and basic support for customized rendering that may be used in differentiable rendering. An example can be found at examples/python/t_reconstruction_system/ray_casting.py.

Conventional rendering

From a reconstructed voxel block grid from TSDF Integration, we can efficiently render the scene given the input depth as a rough range estimate.

76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
# examples/python/t_reconstruction_system/ray_casting.py
        result = vbg.ray_cast(block_coords=frustum_block_coords,
                              intrinsic=intrinsic,
                              extrinsic=extrinsic,
                              width=depth.columns,
                              height=depth.rows,
                              render_attributes=[
                                  'depth', 'normal', 'color', 'index',
                                  'interp_ratio'
                              ],
                              depth_scale=config.depth_scale,
                              depth_min=config.depth_min,
                              depth_max=config.depth_max,
                              weight_threshold=1)

        fig, axs = plt.subplots(2, 2)

The results could be directly obtained and visualized by

 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
# examples/python/t_reconstruction_system/ray_casting.py
        fig, axs = plt.subplots(2, 2)
        # Colorized depth
        colorized_depth = o3d.t.geometry.Image(result['depth']).colorize_depth(
            config.depth_scale, config.depth_min, config.depth_max)
        axs[0, 0].imshow(colorized_depth.as_tensor().cpu().numpy())
        axs[0, 0].set_title('depth')

        axs[0, 1].imshow(result['normal'].cpu().numpy())
        axs[0, 1].set_title('normal')

        axs[1, 0].imshow(result['color'].cpu().numpy())
        axs[1, 0].set_title('color via kernel')

Customized rendering

In customized rendering, we manually perform trilinear-interpolation by accessing properties at 8 nearest neighbor voxels with respect to the found surface point per pixel:

 97
 98
 99
100
101
102
103
104
105
# examples/python/t_reconstruction_system/ray_casting.py
        vbg_color = vbg.attribute('color').reshape((-1, 3))
        nb_indices = result['index'].reshape((-1))
        nb_interp_ratio = result['interp_ratio'].reshape((-1, 1))
        nb_colors = vbg_color[nb_indices] * nb_interp_ratio
        sum_colors = nb_colors.reshape((depth.rows, depth.columns, 8, 3)).sum(
            (2)) / 255.0
        axs[1, 1].imshow(sum_colors.cpu().numpy())
        axs[1, 1].set_title('color via indexing')

Since the output is rendered via indices, the rendering process could be rewritten in differentiable engines like PyTorch seamlessly via PyTorch I/O with DLPack memory map.