Customized Integration

You can prototype a new RGB-D volumetric reconstruction algorithm with additional properties (e.g. semantic labels) while maintaining a reasonable performance. An example can be found at examples/python/t_reconstruction_system/integrate_custom.py.

Activation

The frustum block selection remains the same, but then we manually activate these blocks and obtain their buffer indices in the Hash map:

78
79
80
81
82
83
84
85
86
87
# examples/python/t_reconstruction_system/ray_casting.py
            # Get active frustum block coordinates from input
            frustum_block_coords = vbg.compute_unique_block_coordinates(
                depth, intrinsic, extrinsic, config.depth_scale,
                config.depth_max)
            # Activate them in the underlying hash map (may have been inserted)
            vbg.hashmap().activate(frustum_block_coords)

            # Find buf indices in the underlying engine
            buf_indices, masks = vbg.hashmap().find(frustum_block_coords)

Voxel Indices

We can then unroll voxel indices in these blocks into a flattened array, along with their corresponding voxel coordinates.

91
92
93
# examples/python/t_reconstruction_system/ray_casting.py
            voxel_coords, voxel_indices = vbg.voxel_coordinates_and_flattened_indices(
                buf_indices)

Up to now we have finished preparation. Then we can perform customized geometry transformation in the Tensor interface, with the same fashion as we conduct in numpy or pytorch.

Geometry transformation

We first transform the voxel coordinates to the frame’s coordinate system, project them to the image space, and filter out-of-bound correspondences:

 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
# examples/python/t_reconstruction_system/ray_casting.py
            extrinsic_dev = extrinsic.to(device, o3c.float32)
            xyz = extrinsic_dev[:3, :3] @ voxel_coords.T() + extrinsic_dev[:3,
                                                                           3:]

            intrinsic_dev = intrinsic.to(device, o3c.float32)
            uvd = intrinsic_dev @ xyz
            d = uvd[2]
            u = (uvd[0] / d).round().to(o3c.int64)
            v = (uvd[1] / d).round().to(o3c.int64)
            o3d.core.cuda.synchronize()
            end = time.time()

            start = time.time()
            mask_proj = (d > 0) & (u >= 0) & (v >= 0) & (u < depth.columns) & (
                v < depth.rows)

            v_proj = v[mask_proj]
            u_proj = u[mask_proj]
            d_proj = d[mask_proj]

Customized integration

With the data association, we are able to conduct integration. In this example, we show the conventional TSDF integration written in vectorized Python code:

  • Read the associated RGB-D properties from the color/depth images at the associated u, v indices;

  • Read the voxels from the voxel buffer arrays (vbg.attribute) at masked voxel_indices;

  • Perform in-place modification

118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
# examples/python/t_reconstruction_system/ray_casting.py
            depth_readings = depth.as_tensor()[v_proj, u_proj, 0].to(
                o3c.float32) / config.depth_scale
            sdf = depth_readings - d_proj

            mask_inlier = (depth_readings > 0) \
                & (depth_readings < config.depth_max) \
                & (sdf >= -trunc)

            sdf[sdf >= trunc] = trunc
            sdf = sdf / trunc
            weight = vbg.attribute('weight').reshape((-1, 1))
            tsdf = vbg.attribute('tsdf').reshape((-1, 1))

            valid_voxel_indices = voxel_indices[mask_proj][mask_inlier]
            w = weight[valid_voxel_indices]
            wp = w + 1

            tsdf[valid_voxel_indices] \
                = (tsdf[valid_voxel_indices] * w +
                   sdf[mask_inlier].reshape(w.shape)) / (wp)
            if config.integrate_color:
                color = o3d.t.io.read_image(color_file_names[i]).to(device)
                color_readings = color.as_tensor()[v_proj,
                                                   u_proj].to(o3c.float32)

                color = vbg.attribute('color').reshape((-1, 3))
                color[valid_voxel_indices] \
                    = (color[valid_voxel_indices] * w +
                             color_readings[mask_inlier]) / (wp)

You may follow the example and adapt it to your customized properties. Open3D supports conversion from and to PyTorch tensors without memory any copy, see PyTorch I/O with DLPack memory map. This can be use to leverage PyTorch’s capabilities such as automatic differentiation and other operators.