Open3D 0.14 is full of new features

Open3D 0.14 Release Notes

We are excited to present the new Open3D version 0.14!

In this release, you will find:

  • TensorBoard visualization
  • Upgraded GUI API
  • New RayCastingScene class
  • Improved Tensor performance
  • Improved I/O performance
  • New 3D learning models in Open3D-ML: Point Transformer and PVCNN
  • Improved interoperability with Numpy

Changes in Installation and Build system

  • Open3D now works with Python 3.9. We release Open3D pre-compiled Python packages in Python 3.6, 3.7 3.8, and 3.9.
  • Open3D 0.14 is the last version that supports conda installation. Starting from version 0.15, users will need to install Open3D with pip install open3d. We recommend installing Open3D with pip inside a conda virtual environment.
  • Git submodules are no longer required in Open3D. You can now clone Open3D with git clone https://github.com/isl-org/Open3D.git without the --recursive flag. Also please note the updated Github URL.
  • Open3D will now build in Release mode by default if CMAKE_BUILD_TYPE is not specified. Python is no longer required for building Open3D for C++ users.
  • Open3D-ML is now recommended to be used along with PyTorch 1.8.2 and/or Tensorflow 2.5.2. Checkout Open3D-ML for more information.

Tensorboard Integration

Now you can use Open3D within Tensorboard for interactive 3D visualization! At a glance, you can:

  • Save and visualize geometry sequences and their properties. This enables interactive visualization and debugging of 3D data and 3DML model training.
  • Visualize 3D semantic segmentation and object detection with input data, ground truth, and predictions. In addition, any custom properties for a PointCloud, from scalar to vector, can be easily visualized.
  • Synchronize time steps and viewpoints during different runs. This helps debug and monitor the effect of parameter tuning.

Rich PBR materials

tensorboard_demo_scene

Object detection

tensorboard_objdet_full_2_vp9 webm

Semantic segmentation

tensorboard_sync_view_vp9

To get started, write some sample geometry data to a TensorBoard summary with this snippet:

from torch.utils.tensorboard import SummaryWriter  # TensorFlow also works, see docs.
import open3d as o3d
from open3d.visualization.tensorboard_plugin import summary
from open3d.visualization.tensorboard_plugin.util import to_dict_batch
writer = SummaryWriter("demo_logs/")
cube = o3d.geometry.TriangleMesh.create_box(1, 2, 4)
cube.compute_vertex_normals()
colors = [(1.0, 0.0, 0.0), (0.0, 1.0, 0.0), (0.0, 0.0, 1.0)]
for step in range(3):
    cube.paint_uniform_color(colors[step])
    writer.add_3d('cube', to_dict_batch([cube]), step=step)

Now you can visualize this in TensorBoard with tensorboard --logdir demo_logs. For more details on how to use TensorBoard with Open3D, check out this tutorial.

Visualizer

Further enhancements have been added to the GUI viewer. Now you can:

  • Directly visualize tensor-based geometry classes including PointCloud, TriangleMesh, and LineSet.
  • Use physically based rendering (PBR) materials that deliver appealing appearance.
  • New default lighting environment and skybox improves visual appeal

  • Use all the functionality in Tensorboard!

    img

import open3d as o3d
import open3d.visualization as vis
a_sphere = o3d.geometry.TriangleMesh.create_sphere(2.5, create_uv_map=True)
a_sphere.compute_vertex_normals()
a_sphere = o3d.t.geometry.TriangleMesh.from_legacy(a_sphere)
# Compare this...
vis.draw(a_sphere)
a_sphere.material = vis.Material('defaultLit')
a_sphere.material.texture_maps['albedo'] =
    o3d.t.io.read_image('examples/test_data/demo_scene_assets/Tiles074_Color.jpg')
a_sphere.material.texture_maps['roughness'] =
    o3d.t.io.read_image('examples/test_data/demo_scene_assets/Tiles074_Roughness.jpg')
a_sphere.material.texture_maps['normal'] =
    o3d.t.io.read_image('examples/test_data/demo_scene_assets/Tiles074_NormalDX.jpg')
# With this!
vis.draw(a_sphere)

A complete, complex demo scene can be found at examples/python/gui/demo-scene.py

Core

  • The Open3D Tensor class received a major performance boost with the help of Intel ISPC compiler and optimization for the contiguous code path.
    img
    (See python/benchmarks/core for the benchmark scripts. For each operation, the geometric mean of run times with different data types is reported. The time is measured with an Intel i9-10980XE CPU.)
  • A major upgrade of Parallel HashMap is done. Now you can choose from multi-valued HashMap and HashSet depending your value types. A comprehensive tutorial is also available.
  • Linear Algebra performance have been optimized for small matrices, especially on 4x4 transformations.
  • Semantics for tensor and tensor-based geometry have been improved, especially on device selection.

  • Functions expecting a Tensor now accept Numpy arrays and Python lists. For example:

    import open3d as o3d
import numpy as np

mesh = o3d.t.geometry.TriangleMesh()
mesh.vertex['positions'] = np.array([[0, 0, 0], [1, 0, 0], [1, 1, 0]], dtype=np.float32)
mesh.vertex['colors'] = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]], dtype=np.float32)
mesh.triangle['indices'] = [[0, 1, 2]]
o3d.visualization.draw(mesh)

I/O

  • We now support I/O from/to Numpy compatible .npz .npy formats for Open3D tensors and tensor maps. It is now easier to convert between Open3D geometry classes and Numpy properties.
  • We have improved I/O performance for tensor-based point cloud and triangle-mesh file-formats, including .ply, .pcd, .pts. Geometry loading time is hence improved for the stand-alone visualizer app.
  • We added support for material data to the MessagePack serialization format of the RPC module.

Geometry

  • We introduce a new class RaycastingScene with basic ray intersections functions and distance transform for meshes, utilizing the award-winning Intel Embree library.

    Example code for rendering a depth map:

    import open3d as o3d
import matplotlib.pyplot as plt

# Create scene and add a cube
cube = o3d.t.geometry.TriangleMesh.from_legacy(o3d.geometry.TriangleMesh.create_box())
scene = o3d.t.geometry.RaycastingScene()
scene.add_triangles(cube)

# Use a helper function to create rays for a pinhole camera.
rays = scene.create_rays_pinhole(fov_deg=60, center=[0.5,0.5,0.5], eye=[-1,-1,-1], up=[0,0,1],
                               width_px=320, height_px=240)

# Compute the ray intersections and visualize the hit distance (depth)
ans = scene.cast_rays(rays)
plt.imshow(ans['t_hit'].numpy())

    Distance transform generated with RaycastingScene:

    See the tutorials for more information (Ray casting, Distance queries).

  • Normal estimation for tensor PointCloud is supported with the tensor-compatible nearest neighbor search modules.

  • Customizable tensor-based TriangleMesh, VoxelBlockGrid, and LineSet are implemented that allow user-defined properties. For example:

    import open3d as o3d
import open3d.core as o3c

mesh = o3d.t.geometry.TriangleMesh()
mesh.vertex["positions"] = o3c.Tensor([[0.0, 0.0, 1.0],
                                     [0.0, 1.0, 0.0],
                                     [1.0, 0.0, 0.0],
                                     [1.0, 1.0, 1.0]], dtype=o3c.float32)
mesh.vertex["my_custom_labels"] = o3c.Tensor([0, 1, 2, 4], dtype=o3c.int32)
mesh.triangle["indices"] = o3c.Tensor([[0, 1, 2], 
                                     [1, 2, 3]], dtype=o3c.int32)

Pipelines

  • We have enhanced point cloud registration (ICP) with a tensor interface:
    • Float64 (double) precision point cloud is supported for a higher numerical stability
    • Robust Kernels, including Huber, Tukey, and GM losses are supported for robust registration.
    • Colored-ICP is now supported in the unified tensor-based API.
    • See this tutorial for more details.
  • We also provide an initial tensor-based reconstruction system in Python, including
    • Customizable volumetric RGB-D integration;
    • Dense RGB-D SLAM with a GUI;
    • See this tutorial for more details.

Open3D-ML

The Open3D-ML library welcomes more state-of-the-art models and operators that are ready to use for advanced 3D perception, especially semantic segmentation, including

  • New state-of-the-art Point Transformer for Semantic Segmentation.
    img
  • Highly Efficient Point-Voxel Convolution for Semantic Segmentation.
    img
  • RaggedTensor integration that enables batch SparseConvolution and SparseConvolutionTranspose along with PyTorch.
  • Batched voxelization for fast point-voxel conversions.

Refer to the tutorial for training and inference on new models. (PyTorch TensorFlow).

Expreimental ARM64 builds

We hope you find Open3D 0.14.0 exciting and useful. Happy coding!

Remember that you can reach out with questions, requests, or feedback through the following channels:

Acknowledgment

We thank all the community contributors for this release!

(alphabetical order)
@cclauss
@chrockey
@chunibyo-wly
@cosama
@forrestjgq
@gsakkis
@junha-l
@ktsujister
@leomariga
@li6in9muyou
@marcov868
@michaelbeale-IL
@muskie82
@nachovizzo
@NobuoTsukamoto
@plusk01
@reyanshsolis
@ShreyanshDarshan
@ShubhamAgarwal12
@SoftwareApe
@stanleyshly
@stotko
@theNded
@zhengminxu