Nvidia's GauGAN turns sketches into photos

Nvidia has released a demo of a sketch-to-landscape tool based on generative adversarial networks (GANs) - accelerated, naturally, by the company's graphics processing products.

Nvidia is currently on a heavy push to promote the use of artificial intelligence in a variety of fields. As well as building AI acceleration cores into its latest Turing-based RTX graphics products and launching a single-board computer aimed at makers and educators as part of its Jetson embedded AI family, the company has been churning out a number of demonstrations of how accelerated AI can be used in fields as far apart as gaming and radiology.

Its latest demo, though, straddles the line between practical tool and interesting toy: A generative adversarial network (GAN) which can turn a hand-drawn sketch into a photorealistic landscape in mere seconds.

Named GauGAN, for the painter Paul Gauguin, the effect is impressive to watch - but Nvidia claims it is more than just a neat demo. 'It’s much easier to brainstorm designs with simple sketches,' explains Nvidia's vice president for applied deep learning research Bryan Catanzaro of the project, 'and this technology is able to convert sketches into highly realistic images.

'It’s like a colouring book picture that describes where a tree is, where the sun is, where the sky is. And then the neural network is able to fill in all of the detail and texture, and the reflections, shadows and colours, based on what it has learned about real images. This technology is not just stitching together pieces of other images, or cutting and pasting textures. It's actually synthesising new images, very similar to how an artist would draw something.'

The current implementation of GauGAN focuses on natural elements - the land, the sky, sea and other water features, and the odd rock - but Nvidia claims it can be expanded to include man-made elements like buildings, roads, and even people when trained on the right image set. More information is available in the research paper, which is due to be presented at the Conference on Computer Vision and Pattern Recognition (CVPR) in June.