Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction


Guy Gafni1      Justus Thies1      Michael Zollhöfer2      Matthias Nießner1     

1Technical University of Munich       2Facebook Reality Labs


Introduction

We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face. Digitally modeling and reconstructing a talking human is a key building-block for a variety of applications. Especially, for telepresence applications in AR or VR, a faithful reproduction of the appearance including novel viewpoint or head-poses is required. In contrast to state-of-the-art approaches that model the geometry and material properties explicitly, or are purely image-based, we introduce an implicit representation of the head based on scene representation networks. To handle the dynamics of the face, we combine our scene representation network with a low-dimensional morphable model which provides explicit control over pose and expressions. We use volumetric rendering to generate images from this hybrid representation and demonstrate that such a dynamic neural scene representation can be learned from monocular input data only, without the need of a specialized capture setup. In our experiments, we show that this learned volumetric representation allows for photo-realistic image generation that surpasses the quality of state-of-the-art video-based reenactment methods.

Video

Publication

Paper - ArXiv - pdf (abs) | GitHub

If you find our work useful, please consider citing it:

        
        @misc{gafni2020dynamic,
              title={Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction}, 
              author={Guy Gafni and Justus Thies and Michael Zollhöfer and Matthias Nießner},
              year={2020},
              eprint={2012.03065},
              archivePrefix={arXiv},
              primaryClass={cs.CV}
        }
        

Dataset

If you would like to access our videos, in order to compare to our method, please contact us directly by email: guy.gafni at tum.de