CVPR 2021 Oral
NeRFace: Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction
TL;DR
Condition NeRF on 3DMM parameters.
Introduction
We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face.
Digitally modeling and reconstructing a talking human is a key building-block for a variety of applications. Especially, for telepresence applications in AR or VR, a faithful reproduction of the appearance including novel viewpoint or head-poses is required. In contrast to state-of-the-art approaches that model the geometry and material properties explicitly, or are purely image-based, we introduce an implicit representation of the head based on scene representation networks. To handle the dynamics of the face, we combine our scene representation network with a low-dimensional morphable model which provides explicit control over pose and expressions. We use volumetric rendering to generate images from this hybrid representation and demonstrate that such a dynamic neural scene representation can be learned from monocular input data only, without the need of a specialized capture setup. In our experiments, we show that this learned volumetric representation allows for photo-realistic image generation that surpasses the quality of state-of-the-art video-based reenactment methods.
Videos
Publication
If you find our work useful, please consider citing it:
@InProceedings{Gafni_2021_CVPR,
author = {Gafni, Guy and Thies, Justus and Zollh{\"o}fer, Michael and Nie{\ss}ner, Matthias},
title = {Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2021},
pages = {8649-8658}
}