Deep Shading

Convolutional Neural Networks for Screen-Space Shading

Oliver Nalbach1     Elena Arabadzhiyska1,2     Dushyant Mehta1     Hans-Peter Seidel1     Tobias Ritschel3

1 MPI Informatik    2 MMCI / Saarland University    3 University College London

An overview of Deep Shading.

Abstract

In computer vision, convolutional neural networks (CNNs) have recently achieved new levels of performance for several inverse problems where RGB pixel appearance is mapped to attributes such as positions, normals or reflectance. In computer graphics, screen-space shading has recently increased the visual quality in interactive image synthesis, where per-pixel attributes such as positions, normals or reflectance of a virtual 3D scene are converted into RGB pixel appearance, enabling effects like ambient occlusion, indirect light, scattering, depth-of-field, motion blur, or anti-aliasing. In this paper we consider the diagonal problem: synthesizing appearance from given per-pixel attributes using a CNN. The resulting Deep Shading simulates various screen-space effects at competitive quality and speed while not being programmed by human experts but learned from example images.

Materials

Citation

Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, Hans-Peter Seidel, Tobias Ritschel
Deep Shading: Convolutional Neural Networks for Screen-Space Shading
arXiv preprint

@misc{Nalbach2016,
	author		= {Oliver Nalbach and Elena Arabadzhiyska and Dushyant Mehta
					and Hans-Peter Seidel and Tobias Ritschel},
	title		= {Deep Shading: Convolutional Neural Networks for Screen-Space Shading},
	booktitle	= {arXiv preprint arXiv:1603.06078},
	year		= {2016}
 }

Interactive Comparison

This tool allows you to compare effects computed using DeepShading to ground truth images. First, select an effect from the list below. Then, move your mouse across the image to switch between different network architectures and the ground truth, respectively.

DoF AO SSS MB GI IBL


Data

What is available?
We provide network definitions, corresponding pre-trained networks as well as our datasets for download. As we use a few custom layers and EXR images, which are not supported by a vanilla Caffe at the present, we also provide a modified version of Caffe which is hosted on GitHub.
How can I use the pre-trained networks?
Our network definitions and snapshots (pre-trained networks) are to be used with our custom version of the Caffe deep learning framework. It is available here.
How are the datasets structured?
The datasets consist of about 61,000 pairs of network inputs (split into several EXR image files) and corresponding ground truth outputs. To get an impression of what these look like consider the comparison tool above which shows exemplary pairs.
How do I need to unpack the archives you provide?
There should be a main folder containing the network definition and solver as well as the image lists. The dataset itself should be organized into different sub-folders, one per scene. The different parts of the dataset below should already generate the correct folder structure.
What are the "dumped blobs" below?
They contain the weights of the trained network as simple text files. The first line specifies the shape of the blob, the rest of the file lists the actual values in Caffe's canonical order.

Depth-of-Field

Coming soon.