Figure above shows three D of sofas, chairs and bathtubs created by Microsoft model

Many organizations have tried to transform 2D images into 3D forms, including AI research labs of Facebook, NVIDIA and other companies, or start-ups like threeddy.ai. Recently, a research team from Microsoft also published a preprint paper, demonstrating its ability to generate 3D shape images based on unstructured 2D images.

Generally speaking, the framework of training needs to use raster processing to perform differential step rendering. Therefore, in the past, researchers in this field have focused on the development of customized rendering models. However, the image processed by this kind of model will not be real and natural, and it is not suitable for generating industrial renderings of game and graphics industry.

Microsoft researchers have made new breakthroughs this time

Specifically, researchers are trying to use a full-featured industrial renderer, which can generate images based on display data. To do this, researchers trained 3D shape generation models to render shapes and generate images that match the 2D dataset distribution. The generator model uses random input vectors (representing the values of data set features) and generates a continuous voxel representation of 3D objects (values on the mesh in 3D space), then inputs voxels into the non differential rendering process, and reduces the threshold value to discrete value before rendering with the existing renderer.

That is to say, this is a novel way of rendering the continuum mesh generated by 3D shape generation model directly by proxy neural renderer. As the researchers explained, given the 3D mesh input, it needs to be trained to match the rendering output of the off the shelf renderer.

The results of Gans in producing 2D image data are impressive. Many visual applications, such as games, need 3D models as input, not just images. However, to directly extend the existing Gan model to 3D, it is necessary to obtain 3d training data.

Note to Lei Feng: the above picture is the 3D mushroom image generated by Microsoft model

the research team adopted a 3 D convolutional GAN architecture for the above generators (GAN is a two-part AI model that includes generators that generate synthetic examples from random noise using distributed sampling and feed these examples into the discriminator along with real examples in the training dataset to try to distinguish the two). data sets generated based on D 3 model and real data sets can synthesize images from different object categories and render from different angles throughout the training.

The researchers also said their framework would extract lighting and shadow information from images, enabling them to extract more meaningful data from each training sample and produce better results based on that. After training the data set of natural images, the framework can generate realistic samples. In addition, the frame can detect the internal structure of concave objects successfully by using the exposure difference between surfaces, so as to accurately capture the concave degree and hollow space.

Combine information about colors, materials, and lighting into the system, and in the future, this information will be available with more

Lei Feng net note: This article is compiled fromVentureBeat