Nvidia’s new software is at the prototype stage but already the transformation of doodles into ‘art’ has received positive reviews. The software uses an artificial intelligence called a generative adversarial network (or what Nvidia term a “smart paint brush”) to turn a basic outline of a scene into a realistic image. Reviewed by The Verge, the software took a drawing of a tree on a hill and quickly filled in the rough sketch with natural textures including grass, clouds, forests, or rocks.
Generative Adversarial Networks
A generative adversarial network (GAN) a type of machine learning systems (a form of unsupervised learning). Two neural networks contest with each other in a zero-sum game framework, and this creates a technique to generate photographs that look authentic to human observers. In other areas, GANs can be taught to create worlds eerily similar to our own in any domain: images, music, speech, prose.
Nvidia’s GauGAN
The new software is called GauGAN after post-Impressionist painter Paul Gauguin, and generative adversarial networks. The software works like a simple digital painting tool. However, in place of the usual palette of colors, the software offers a collection of natural elements, like sand, snow, and sky, each represented by a different hue, to choose from and use in basic artwork.
The power of GauGAN is demonstrated with its ability to maintain a realistic image by dynamically adjusting parts of the picture to match changes. An example from DP Review is when transforming a grassy field to a snow-covered landscape this triggers an automatic sky change, ensuring the two elements of the render are compatible and realistic.
The machine learning was trained using around one million images. This enables the deep learning model to automatically fill in the landscape to suit what the user is intending. If the user draws a pond, the nearby elements – like trees and rocks – transform and appear as reflections in the water. Alternatively, switching from “grass” to “snow” leads to entire image changes, from a spring to a winter scene, for example, and what was a leafy tree turning barren.
Nvidia hopes to add GauGAN to its AI Playground suite in the coming weeks.
