Nvidia demos clever image restoration technology

Nvidia's research and development team have come up with a new GPU-accelerated deep-learning system capable of restoring images seemingly damaged beyond repair, building on existing 'image inpainting' technologies - functionally similar to Adobe Photoshop's Content Aware Fill tool - and, it claims, surpassing them.

'Existing deep learning based image inpainting methods use a standard convolutional network over the corrupted image, using convolutional filter responses conditioned on both valid pixels as well as the substitute values in the masked holes (typically the mean value),' the researchers explain in the abstract for their paper, Image Inpainting for Irregular Holes using Partial Convolutions. 'This often leads to artefacts such as colour discrepancy and blurriness. Post-processing is usually used to reduce such artefacts, but are expensive and may fail.'

The solution, as the paper's title makes clear, is the use of partial convolutions 'where the convolution is masked and renormalised to be conditioned on only valid pixels.' Coupled with a mechanism which generates an updated mask for the next layer as part of the forward pass, the result is a system which can automatically restore missing parts of images - even when there is nothing left in the original image from which a sample can be taken, as in one of the example uses of replacing a celebrity's eyes.

The team's demonstration video, running on the company's Tesla V100 accelerator boards, showcases how the system operates, and it's undeniably impressive in use. While the in-filled content is, naturally, not an exact match for what was originally there - especially obvious in the demonstration where entire objects, like rocks and bridges, are masked from the image and disappear entirely post-processing - it's surprisingly convincing and, unlike techniques which require post-processing for colour matching and other tweaks, entirely automatic in operation.

Nvidia has published a blog post on the process, while the research paper is available for open access on arXiv.