Rendering is the core of computer graphics, which can transform 3D model into 2D image. This is also a common way to bridge between 3D scene properties and 2D image pixels. However, traditional rendering engines cannot be differentiated, so they cannot be incorporated into the deep learning work pipeline. Pytorch3d has a built-in modular differentiable renderer, which can be used to process differentiable 3D data.

Facebook recently opened up a library of functions to use Py Torch for 3D deep learning PyTorch3DThis is a highly modular and optimized library with unique functionality designed to simplify 3D deep learning through PyTorch. PyTorch3D provides a common set of 3D operators and fast and differentiatable loss functions (lossfunction) for 3D data, as well as a modular differential rendering API. with the above functions, researchers can immediately import these functions into the current state-of-the-art deep learning system.

researchers and engineers can use pytorch 3d for various 3d deep learning studies (whether 3d reconstruction, cluster adjustment, or even 3d reasoning), as well as improved recognition tasks in two-dimensional space.

The cognition of three-dimensional space plays an important role in the interaction between artificial intelligence and the real world. For example, robot navigation in physical space, improvement of virtual reality experience, recognition of occluded objects in 2D content and so on. But even Facebook, which has rich deep learning technology, will still be bothered in the face of 3D deep learning problems. Facebook said that the reason why deep learning technology is less used in 3D scenes is the lack of sufficient tools and resources to support the complexity of the combination of neural network and 3D data. This kind of scene requires more memory and higher computing power, unlike 2D images, which can be displayed by tensor table, and many traditional graphic operators are not differentiable, so the research of 3D deep learning technology The investigation is limited.

To this end, Facebook built a pytorch3d function library to promote 3D deep learning research. Like pytorch3d, which provides a highly optimized function library for 2D recognition tasks, pytorch3d optimizes training and reasoning by providing batch processing and support for 3D operators and loss functions. In order to simplify the complexity of 3D model batch processing,Facebook created the meshes formatThis is a heterogeneous grid model data structure designed for deep learning applications for batch processing.

This data structure allows researchers to easily convert the underlying grid model data into different views to match the operator to the most efficient representation of the data. more importantly, pytorch3d provides researchers and engineers with the ability to flexibly switch between different representation views and access different grid properties.

Rendering is the core of computer graphics, which converts 3D models to 2D images. This is also a regular means of bridging between 3D scene properties and 2D image pixels. However, traditional rendering engines cannot differentiate, so they cannot be incorporated into the deep learning workflow. So Facebook has built-in a highly modular differentiator at PyTorch 3D to handle differential 3D data. The implementation of this feature consists of combineable units that allow users to easily extend the renderer to support custom lighting or shadowing effects.

Facebook packages these functions into a toolkit, and provides operators, heterogeneous batch processing functions and modular differential rendering API to help researchers with complex 3D neural network application research.

View PyTorch 3d documentation: https://pytorch3d.org/docs/why_pytorch3d.htm