DIB-R (Interpolation-based Differentiable Renderer), because that is the name of the Nvidia system, was created with the help of PyTorch software. This is an Python open source programming library for machine learning. Thus, DIB-R can instantly create three-dimensional models based on 2D images thanks to artificial intelligence.
How does DIB-R work? First, the system analyzes the given photo, and then based on it creates a 3D model quite well reflecting how the presented object would actually look in three dimensions. This model has the right shape, texture and color, and is properly illuminated. Nvidia described the process in detail here.
Nvidia trained its system with the help of many data sets. These sets included, for example, photos that were previously turned into three-dimensional models, three-dimensional models presented from many sides, and photos showing different sides of individual objects. It takes about two days for artificial intelligence to learn to render three-dimensional models of data objects, e.g. birds (on which Nvidia has focused) based on individual images, but after this period the system is able to generate 3D models based on previously unseen pictures in less than 100 milliseconds.
"In fact, for the first time, significant 3D properties can be predicted based on any 2D image." said Jun Gao, co-author work describing the DIB-R system.
The creators of this artificial intelligence believe that it could be used in many interesting ways – for example, enabling autonomous robots to perceive depth. If the machines understand the three-dimensionality of their surroundings, they will be able to navigate more efficiently and carry out the tasks assigned to them.