Open3D better than ever in our newest 0.13 release

Open3D 0.13.0 Release Notes

We welcome you to the 0.13.0 release of Open3D. This release is full of exciting new features with a strong emphasis in real-time pipelines, but also full of bug fixes and usability improvements. The big highlights of this release are as follows:

  • New real-time 3D reconstruction pipeline, featuring GPU and CPU support based on VoxelHashing.
  • New real-time point cloud registration algorithm, featuring a high-performance version of Iterative Closest Point (ICP).
  • New Neighbor Search module, introducing your favorite search algorithms such as KNN and RadiusSearch, with support for GPU and CPU devices through a common interface.
  • New web visualizer, which enables users to access the advanced rendering and visualization features of Open3D in your favourite web environments (remote and locally!), including Jupyter notebooks, Jupyter lab, and standalone web applications.
  • New 3D machine learning models and datasets, featuring PointRCNN for 3D object detection, SparseConvNets for point cloud semantic segmentation, and support for ScanNet and SunRGBD.
  • Upgraded GUI module, providing improved and more versatile versions of existing widgets, and new ones: ImageWidget and ToggleSwitch.
  • Upgraded build system, adding support for CUDA 11.

Real-time 3D reconstruction

We introduce a new CUDA accelerated pipeline including RGBD odometry, frame-to-model tracking, and volumetric integration.

image
Figure 1. Example of 3D reconstruction from an RGB-D sensor.

Odometry

We introduce the tensor based real-time RGBD Odometry pipeline. In addition to the legacy Hybrid and Intensity based methods, we support the popular point-to-plane method.

TSDFVoxelGrid

We further accelerate volumetric integration and introduce fast ray casting for rendering.

VoxelHashing

Based on the accelerated RGBD odometry and raycasting, we present the fully functional VoxelHashing system. It performs dense volumetric reconstruction with fast frame-to-model tracking. We present an easy-to-use GUI that also shows real-time interactable surface reconstruction.

SLAC

We have further enhanced our legacy offline reconstruction system by introducing the Simultaneous Localization and Calibration (SLAC) algorithm. This algorithm applies advanced dense multi-way registration along with non-rigid deformation to create highly-accurate reconstructions.

Real-time point cloud registration

We present a high-performance implementation of ICP using Open3D’ Tensor library. This module is one of the first on leveraging the new Neighbor search module and the newly crafted parallel kernels. This implementation brings support for multi-scale ICP, which allows us to do iterations on different resolutions in order to accelerate convergence while keeping computation low.

image
Figure 2. ICP registration of multiple point clouds from a driving dataset.

New Neighbor Search module

Neighbor search is at the core of many 3D algorithms. Therefore, it is critical to have access to a fast implementation able to execute a large number of queries in a fraction of a second. After months of development, the Open3D team is proud to present the new Neighbor Search module!

This module brings support for core search algorithms, such as KNN, Radius search, and Hybrid search. All these algorithms are provided with support for both CPU and GPU, through a common and easy-to-use interface. Write your code once and support multiple devices! Moreover, we have not sacrificed a single flop of computation, making this module one of the fastest neighbor search libraries ever created.

Web visualizer

The need for visualizing complex 3D data in web environments has surged considerably in the past few years, in part thanks to the proliferation of sensors like LIDAR and RGBD cameras. New use cases, such as online dataset inspection and remote visualization are now an integral part of many tasks, requiring the crafting of ad-hoc tools, which often are cumbersome to use.

image
Figure 3. Standalone visualization of a semantic segmentation model in a browser.

In order to improve this situation, we introduce our new web-based visualization module, which enables 3D visualization from any browsers and any location. This module lets users run advanced rendering and visualization pipelines, both remote and locally through your web browser. All the power of Open3D’ rendering engine --including support for PBR materials, multiple lighting systems, 3D ML visualization, and many other features--, are now supported in your browser. This module also includes a Jupyter extension for interactive web-based visualization! This new feature allows you to run compute-intensive 3D processing in a dedicated server while visualizing the results remotely on any device through your browser.

image
Figure 4. Visualization of a 3D model on a Jupyter notebook.

3D Machine Learning

Architectures

In this release, we introduce a new point cloud semantic segmentation architecture based on a Sparse Convolution-based UNet model. This architecture leverages the new sparse convolution operators provided by Open3D, and achieves state of art performance for Semantic Segmentation on the ScanNet dataset. We have also added support for PointRCNN for the task of 3D object detection. To enable PointRCNN we have added new operators like furthest_point_sampling, three_interpolate, ball_query, which are available through Open3D for TensorFlow and Pytorch.

image
Figure 5. Example of 3D semantic segmentation using a SparseConvUNet model on ScanNet.

image
Figure 6. Example of 3D object detection using a PointRCNN on KITTI.

All these models are provided with their respective training and inference pipelines with support for TensorFlow and Pytorch. Pre-trained models are also provided (check out the following link).

Datasets

This release brings the following datasets: Scannet and SunRGBD.

Frameworks

We now support all models on newer versions of TensorFlow (2.4.1) and PyTorch (1.7.1), on CUDA 11.0.

GUI and rendering modules

Open3D 0.13.0 brings a cascade of improvements and fixes to the renderer and GUI modules.

  • The camera can now be controlled with respect to a target object, and the pan-in/pan-out actions are smoother.
  • The render supports render targets.
  • The black screen issue in MacOS systems is now solved.
  • Support for a video widget
  • The background of the 3D scene widget can be updated with an image.
  • Camera objects can be visualized in the 3D scene.
  • Camera objects support intrinsic and extrinsic parameters.
  • Support for multiple fonts.
  • Support for a new ToggleSwitch widget.
  • New GUI/rendering examples for common tasks, such as data acquisition from RGBD sensors, 3D reconstruction, and registration, among others.

Build System

Our pip packages now include support for CUDA 11.0, PyTorch 1.7.1, and TensorFlow 2.4.1 to enable RTX 3000 series devices. Please, notice that we provide custom PyTorch wheels for Linux to work around an incompatibility between CUDA 11, PyTorch, and extension modules such as Open3D-ML.

This release also brings new improved support for CUDA on Windows. Users can now build CUDA accelerated Python wheels for Windows. Open3D is now built with security options enabled by default.

We hope you find Open3D 0.13.0 exciting and useful. Happy coding!

Remember that you can reach out with questions, requests, or feedback through the following channels:

The Open3D team

Changelog:

Deprecating

  • All visualization tools, such as draw_geometries will be deprecated in Open3D 0.14 in favor of the new O3DViewer.

Changes to Open3D

  • add ml deps for conda build (#2826)
  • tensorkey refactoring: pimpl and optional arguments (#2813)
  • Add file and line for error message (#2829)
  • color map optimization refactoring (#2832)
  • remove openmp macro guards for color map optimization (#2835)
  • easy way of initializing c++ tensor (#2796)
  • Add numerical operations to Tensor (#2848)
  • Implement np style arange for Tensor (#2849)
  • Refactor TSDF kernels (#2854)
  • core::Scalar class and refactor arange (#2857)
  • octree updates (#2833)
  • numpy-compatible tensor IO (#2859)
  • Upgrade zlib and libpng, use ExteralProject (#2878)
  • update mkl download links (#2882)
  • Python RSBagReader.open() crash on Ubuntu (#2876)
  • refactoring: add pipelines/kernel folder (#2875)
  • Standarize To() and Copy() (#2877)
  • compile 3rd-party library in a separate step for arm64 CI (#2891)
  • Hardened security flags for building Open3D (#2883)
  • fix incompatible CUB include dir with CUDA 11 and later (#2913)
  • Jsoncpp with ExternalProject (#2907)
  • Python example of using a texture on a Material (#2935)
  • Add rendering for VoxelGrid and Octree to new renderer (#2932)
  • Enable RSUSB backend only on Linux. (#2923)
  • Added C++ example OffscreenRendering (#2914)
  • Lazy load renderer when creating window in case Python crashes before window becomes visible (#2819)
  • Fixed trackpad scrolling to be buttery smooth at slow speeds on macOS (instead of being very discrete-seeming) (#2964)
  • add tunable parameter to filter outliers in surface extraction (#2931)
  • Added close callback to window (#2958)
  • Mode For Light To Follow Camera (#2944)
  • Image processing Ops sample with IPP and NPP (#2898)
  • Added ability to set camera in open3d.visualization.draw() (#2936)
  • Mouse movement by spherical coordinates (#2962)
  • Refactored RPC interface to be more amenable to variation in visualizers (fixes add not updating) (#2970)
  • Optimised kernel for ICP (#2917)
  • Add support for FPS-style mouselook to visualizer (#2551)
  • maximize build space for ubuntu (#2986)
  • Minor typo in repr for OBB (#2987)
  • function for casting the tensor data pointer to a datatype (#2934)
  • Undo lazy creation of renderer on Windows; it fixed the hang on exit when a python exception is generated before the run() happens, but broke normal windows (#2996)
  • Add Image IO (#2955)
  • Implement ray casting for TSDF voxel grids (#2941)
  • Procedural Ground Plane (#2980)
  • Mousewheel dollies based on distance from center of rotation rather than size of model (#3011)
  • Update Mac build to use latest 1.9.11 Filament to fix illegal instruction error (#3013)
  • Fix arm64 CI out-of-time (#3020)
  • Add Ground Plane support to O3DViewer (#3014)
  • Recenter rotation to point on double-click (#2905)
  • Implement depth buffer read back (#3028)
  • Remove unnecessary include causing unwanted compile dependencies (#3030)
  • Update faiss_build.cmake (#2979)
  • Added python bindings for rendering::Camera (#3033)
  • PointRCNN ops (#3021)
  • add int16 dtype and tests (#3035)
  • enable ccache for cuda with CMAKE_CUDA_COMPILER_LAUNCHER (#3043)
  • Support updating tpointcloud with the same or fewer vertices as original (#3048)
  • Use RenderTargets For Scene Caching (#3031)
  • Add init pybind for Label3D (#3045)
  • Fix linking issue of Torch/TF Ops (#2957)
  • GCE CUDA Python tests (#3063)
  • Rearrange constructor so input missing "points" attribute reaches LogError (#3062
  • Make sure bounding box is non-empty (#3061)
  • Replace CMake GLOB and GLOB_RECURSE with file listing (#3059)
  • Add pybind param names to docstring for 'TSDFVoxelGrid.integrate' (#3058)
  • Support USE_SYSTEM_LIBREALSENSE flag. (#3057)
  • Faster FixedRadiusSearch and HybridSearch (#2821)
  • IPP/NPP image operators (#2972)
  • Add function to randomly downsample from pointcloud (#3050)
  • maximize build space (#3071)
  • Add 'd' suffix to non-system libpng compiled library discovery for non-MSVC Debug builds (#3067)
  • disable incompatible clang flags for now (#3080)
  • [FileSystem] Fix GetPathComponents and unit tests on Windows. (#3070)
  • Minor spell fix (#3081)
  • Add disconnection function to AzureKinectSensor (#3044) (#3082)
  • PointRCNN ops tf (#3075)
  • Set post-processing coniguration when copying View settings (#3083)
  • Fix crash if posted Window callback executes after Window is destroyed (#3084)
  • style-guide link update (#3086)
  • Update contribution_recipes.rst (#3088)
  • changed Registration unit-test to use RegistrationPermuteDevices (#3095)
  • LU, LU_with_ipiv, Det, Triu, Tril, Triul Ops (#2879)
  • Set position in constructor (#3096)
  • memory efficient hybrid search (#3073)
  • Implemented ability to render to a bitmap window and interact with UI (#3072)
  • Better error message if you don't initialize gui::Application before creating a window (#3103)
  • Added pybind for Camera::CopyFrom() (#3104)
  • Enable builds on Fedora. (#3102)
  • [CUDA] Fix memory leaks. (#3099)
  • Use UnownedPointer so Python doesn't retain a valid shared_ptr (#3106)
  • Tensor nested initializer list with recursion (#3055)
  • Fix window resize when Scene caching is enabled. (#3107)
  • delete some ssh keys to keep the profiles small (#3110)
  • Fixed crash when referencing submenus (#3114)
  • Enable builds on Fedora (FAISS, MKL, TBB) and fix output result of CMAKE. (#3118)
  • Added the double-click code from Window.cpp in #2905 that got clobbered by the WindowSystem refactor (#3119)
  • TransformConverter supports Float32 and Float64 (#3098)
  • Fix one frame blinking in Open3DViewer when layout changed or dialog box was closed (#3121)
  • benchmark links 3rd-party library (#3124)
  • Set byproducts for external projects. (#3138)
  • Open3D C++ example projects as separete repos (#3128)
  • gce disk size set to 48GB (#3169)
  • download test datasets from intel-isl/open3d_downloads (#3105)
  • Add ability to set center of rotation on SceneWidget, which is necessary for copying from a camera. (#3147)
  • Minor(?) fix (#3175)
  • Clone from specific commit instead of release branch (#3179)
  • fix docstrings for ML modules (#3160)
  • Add multi-window C++ and Python examples (#3167)
  • Expose rendering to a depth image to Python, makes resulting depth image range from 0 (near plane) to 1 (far plane) instead of simply returning the raw values from Filament's buffer (which are reversed), fixes widgets that aren't SceneWidget or a Layout not working if they are a top-level child. (#3178)
  • Fix incorrect coordinates for mouse events on Linux (#3177)
  • Added gui::ToggleSwitch (#3182)
  • Give Material::shader default value of "defaultUnlit" (#3184)
  • Add support for textures in TriangleMesh::operator+= (#3170)
  • TensorToEigenMatrix Ops. (#3163)
  • assigning scalar to r-value tensor of arbitrary shape (#3137)
  • t::PointCloud from RGBD image (#3113)
  • openblas x86_64 docker (#3195)
  • Updated copyrights to 2021 for gui (#3189)
  • Added an example for line width (#3194)
  • disable CreateFromRGBDImage for arm64 test (#3206)
  • Don't allow empty geometry to be added to scene (#3203)
  • upgrade scipy and cmake version (#3199)
  • Added pybinds for color grading enums, also changed property open3d.visualization.rendering.Open3DScene.get_view to view (it is a property, not a function) (#3193)
  • Fix for CreateFromRGBDImage test (#3207)
  • speed up arm64 build with docker and ccache (#3205)
  • [Windows] Fix linker errors and warnings. (#3151)
  • Fix ModuleNotFoundError when build with GCC10. (#3142)
  • Tensor-based RGBD Odometry (#3085)
  • t-ICP ComputeTransform major performance improvements. (#3019)
  • UV-Map for Box, Tetrahedron, Octahedron, Icosahedron (#3097)
  • Init voxeldownsample (#3144)
  • Fix compilation on Windows when building with CUDA. (#3036)
  • Added mouse events + convert point to world coordinates example (#3192)
  • Optimising Tensor RGBD Odometry. (#3213)
  • handle the invalid tetra warning (#3201)
  • SplitString returns tokens (#3223)
  • clean up ci_utils for arm64 (#3224)
  • less verbose tar print for arm docker (#3226)
  • optimised hybrid search, clear tree from master (#3218)
  • Added 3D labels to O3DVisualizer, added example (#3233)
  • Add ability to set layout minor axis size, moved render target drawing from Open3DScene to SceneWidget which is a little more straightforward and also puts the image in the proper z-level so that SceneWidget can be used in layouts (#3212)
  • Improves progress bar in Open3DViewer app, also fixes unlit material instead of lit (#3238)
  • refactor logging (#3241)
  • Adds ImageWidget and ability to efficiently change Scene background image (#3002)
  • Update to filament v1.9.19 for mac for multi-window fixes (#3251)
  • Fix spurious warning if a triangle mesh with no normals uses "default Unlit". Happens if you pick points (#3256)
  • Fix camera matrix getting pybinds not returning something that can be converted to a Python object (#3257)
  • Hashmap refactor (#3211)
  • Added camera visualization to LineSet (#3255)
  • Added header files for TriangleMeshModel to Open3D.h.in (#3261)
  • Update stdgpu dependency to latest upstream (#3265)
  • Use more sensible default color for point clouds with no per-vertex color (#3278)
  • T-Registration pybinds and python-tests. (#3112)
  • Fix memory leak in stdgpu hashmap backend (#3281)
  • Upgrade Tensor-based RGBD odometry (#3229)
  • UV Maps for Cone, Cylinder, Sphere added. (#3275)
  • get_is_open() for CollapsableVert widget (#3286)
  • Remove use of CMAKE_BUILD_TYPE. (#3164)
  • Support TGeometry Image At Low Level and In Image Widget (#3277)
  • simplify MANIFEST.in with graft (#3288)
  • Optimised CopyCPU and CopyCUDA for case when dst is contiguous and src has single element. [Fill] (#3136)
  • Added polygon picking (#3252)
  • Add post_redraw() to O3DVisualizer so that it can be used non-blocking (#3262)
  • Fixes crash/hang on exit if rendered to an image (#3273)
  • voxel_down_sample pybind for t::pointcloud (#3293)
  • Added italic, bold, and bold italic styles (#3298)
  • Tensor IsInf, IsNan, IsFinite (#3302)
  • Add remaining fixed-size integer dtypes (#3320)
  • Port to CMAKE_CUDA_ARCHITECTURES (#3316)
  • Tensor Multi Scale ICP (#3285)
  • Fix many minor C++ documentation issues (#3326)
  • update docs-build dependencies (#3335)
  • Added tooltips (#3334)
  • Defer layout of dialog to ensure ImGUI context is valid for CalcPreferredSize (#3339)
  • Optimize TSDF-related operations (#3314)
  • Unify dispatch macros from CoreUtil.h and Dispatch.h (#3295)
  • Fix error message when no correct arguments are provided (#3292)
  • Add material property for supplying vertex colors in sRGB (#3340)
  • CUDA, TF, Pytorch version upgrade (#3049)
  • fix header include (#3344)
  • pass cuda inc dir to stdgpu to find thrust if cuda is not in a standard (#3343)
  • fix conditions for showing warning. drop pytorch version test. (#3355)
  • Remove attributes from pointcloud and trianglemesh (#3299)
  • fixed int64_t issue on mac (#3291)
  • Adjusted odd characters which were giving unicode charmap error (#3336)
  • Improve performance of stdgpu hashmap backend (#3346)
  • Add pybind for T PointCloud and fix associated bugs (#3345)
  • Shubhamagarwal12/tensor mesh io (#3221)
  • Update Linux and Windows platforms to Filament v1.9.19 (#3271)
  • Fix multiple VectorEdits in a UI not working (#3360)
  • Properly scale UInt8 colors when converting to Legacy Point Cloud (#3350)
  • Make tooltips prettier (#3348)
  • fix scalar boolean handling in pybind (#3362)
  • Fixed identically named Checkboxes and ToogleSwitches not working, removed use of stringstream in the rest (#3369)
  • Unqueue dead window events (#3370)
  • Make sure ResizeRenderer gets called on BitmapWindow (#3374)
  • Further improve performance of stdgpu hashmap backend (#3377)
  • simplify jsoncpp cmake (#3174)
  • [WebRTC] WebRTC library build scripts (#3349)
  • Move vertexmap/normalmap related functions to t::Image (#3318)
  • Revert "simplify jsoncpp cmake (#3174)" (#3386)
  • Don't add menus in Jupyter that are not meaningful (#3385)
  • Added ability to setup camera with extrinsic and intrinsic in new visualizer (#3338)
  • Increase warning level (#3357)
  • Clarify C++ standard in documentation (#3384)
  • Optimize RGBD odometry (#3323)
  • Python docs fix (#3373)
  • Add Docker minimal example to getting_started docs. Fix #3388 (#3396)
  • Introducing SLAC (#3108)
  • Voxel hashing GUI (#3363)
  • Fix screen turning black on resize in standalone viewer (#3394)
  • Tensor Registration Update (#3352)
  • SLAC Pybinds, Unit Tests, Documentation. (#3393)
  • add template parameters for input and output tensors in the cconv op (#3353)
  • RGBD Odometry Pybinds (#3359)
  • Tensor::Clip() simple implementation (#3266)
  • Fix filament byproducts and build from source. (#3375)
  • Fix CUDA builds on Windows. (#3380)
  • Replace templates with Scalar class (#3391)