3D-FM GAN: Towards 3D-Controllable Face Manipulation
Yuchen Liu1,2
Zhixin Shu2
Yijun Li2
Zhe Lin2
Richard Zhang2
S.Y. Kung1
1Princeton University
2Adobe Research
European Conference on Computer Vision (ECCV) 2022
[ArXiv]
[Paper]
[GitHub]
Given an input of existing face image, 3D-FM GAN produces photo-realistic controllable manipulations in pose, expression, and illumination with strong identity preservation.

Abstract

3D-controllable portrait synthesis has significantly advanced, thanks to breakthroughs in generative adversarial networks (GANs). However, it is still challenging to manipulate existing face images with precise 3D control. While concatenating GAN inversion and a 3D-aware, noise-to-image GAN is a straight-forward solution, it is inefficient and may lead to noticeable drop in editing quality. To fill this gap, we propose 3D-FM GAN, a novel conditional GAN framework designed specifically for 3D-controllable Face Manipulation, and does not require any tuning after the end-to-end learning phase. By carefully encoding both the input face image and a physically-based rendering of 3D edits into a StyleGAN’s latent spaces, our image generator provides high-quality, identity-preserved, 3D-controllable face manipulation. To effectively learn such novel framework, we develop two essential training strategies and a novel multiplicative co-modulation architecture that improves significantly upon naive schemes. With extensive evaluations, we show that our method outperforms the prior arts on various tasks, with better editability, stronger identity preservation, and higher photo-realism. In addition, we demonstrate a better generalizability of our design on large pose editing and out-of-domain images.


Methodology

3D-FM GAN takes a single photo as input. It first extracts the 3D parameter of the photo by face reconstruction and then renders identity-preserved 3D-manipulated edit faces. The input photo and the edit signals are later jointly encoded into a generator to synthesize various photo editings.


Supplemental Video


Code (Coming Soon!)



Paper and Supplementary Material

Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Richard Zhang, S.Y. Kung
3D-FM GAN: Towards 3D-Controllable Face Manipulation
In ECCV, 2022.


[Bibtex]