Introduction

• By optimizing volumetric scene function using sparse set of input views, we can synthesize novel views of complex scenes.
• Input : Scenes consisted of continuous spatial locations $(x, y, z)$ and viewing directions $(\theta, \phi)$
• Output : Volume density and view-dependent color at spatial location.
  • Density : differential opacity controlling the amount of radiance accumulated by a ray passing thru position.
• Model : MLP, without convolutional layers.
  • Because basic implementation does not converge to sufficient representation, we use the following;
    • We transform the input coordinates with positional encoding, to represent higher frequency functions.
    • We propose hierarchical sampling procedure to reduce number of queries.
• Using traditional volume rendering techniques, we can project the output into synthesized images.

Related Works : Neural 3D Shape representations.

• Implicit representation of 3D shapes as level sets
  • Method #1 : (x, y, z) coords to signed distance functions.
    • Curless, B : A volumetric method for building complex models from range images
    • Jiang, C : Local implicit grid representations for 3d scenes
    • Park, J : DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
  • Method #2 : (x, y, z) coords to occupancy fields.
    • Genova, K : Local Deep Implicit Functions for 3D Shape
    • Mescheder, L : Occupancy networks: Learning 3D reconstruction in function space
  • Limited by requirement of ground truth 3D geometry (ex : ShapeNet)
  • Recent : relax this requirement by functions that allow neural implicit shape representations to be optimized through 2D images.
    • Niemeyer, M : Differentiable volumetric rendering
    • Sitzmann, V : Scene representation networks
    • Still limited to simple shapes.

Related Works : View synthesis and image-based rendering.

• Light field sample interpolation
  • Levoy, M : Light field rendering

NeRF Representation

• Input :
  • 3D Location $\mathbf{x} = (x, y, z)$
  • 2D Viewing direction $\mathbf{d} = (\theta, \phi)$
• Output :
  • Emitted color $\mathbf{c} = (r, g, b)$
  • Volume density $\sigma$
• Network
  • $F_\Theta : (\mathbf{x}, \mathbf{d}) \rightarrow (\mathbf{c}, \sigma)$
  • In order to make the representation multiview consistent, restrict the network such that volume density $\sigma$ is only predicted by location $x$, regardless of direction.
    • $\mathbf{x} \rightarrow MLP(*8, 256) \rightarrow (\sigma, f.v)$
    • $Concat(f.v, \mathbf{d}) \rightarrow MLP(*1, 128) \rightarrow (\mathbf{c})$

Volume Rendering with Radiance Field

Written with StackEdit.