Doctoral Defense - Andrew Hou

About the Event

The Department of Computer Science & Engineering

Michigan State University

Ph.D. Dissertation Defense

March 18th, 2025 at 2:00pm EST

3105 Engineering Building and Zoom

Zoom Information: Upon Request from Vincent Mattison or Advisor

ABSTRACT

Face Modeling Under Diverse Illumination Conditions

By: Andrew Hou

Advisor: Dr. Xiaoming Liu

3D face modeling is a longstanding problem within the computer vision and computer graphics communities with widespread applications in AR/VR, movie production, and entertainment that involves a vast array of problems including modeling facial appearance, shape, expressions, lighting, and pose. One prominent issue in this domain is the inability to properly model non-Lambertian lighting effects, including hard shadows caused by directional lights. To this end, we propose two methods to better handle facial hard shadows. The first leverages physically-inspired supervision along the hard shadow boundaries to encourage the model to focus on properly synthesizing these regions. The second method pushes one step further and directly leverages the 3D face to physically constrain the generated hard shadows and guarantees shadow geometric consistency with respect to the face. However, both of these proposed methods do not have full control over facial illumination. For example, our directional relighting methods represent light as a direction and are not able to control light size, rendering them unable to model the associated lighting effects. We thus propose COMPOSE, the first single-image portrait shadow editing method with full control over all shadow attributes, including shadow intensity, shape, and position that also preserves other attributes of the portrait's original environmental lighting. This is achieved by decomposing the problem into an environment-preserving diffuse image estimation and subsequently estimating a shadowed image where the shadow parameters are varied. By performing image compositing between the diffuse and shadowed images, COMPOSE precisely controls all shadow attributes during illumination editing.
Moving beyond illumination editing, we investigate the remainder of the face modeling problem in editing appearance, shape, expressions, and pose. We first propose INFAMOUS-NeRF, an implicit face modeling method that introduces hypernetworks to estimate subject-specific model weights, thus alleviating the burden of the NeRF MLP to encode all subject information in a single shared set of weights. While improving representation power with subject-specific weights, INFAMOUS-NeRF also maintains editability by encouraging subjects with similar attributes (e.g. same expression) to share the same latent code for those attributes, which maintains semantic alignment of latent spaces. Finally, we investigate the tradeoff between representation power and efficiency. Gaussian splatting methods attempt to resolve this tradeoff by offering both high representation power and efficiency with real-time rendering. However, their efficiency nonetheless depends on the number of rasterized gaussians. Moreover, the time needed to optimize/prepare the avatar (enrollment time) is often high. We therefore propose EGGHead: a novel face modeling method that, given a fixed gaussian budget, efficiently assigns gaussians to different semantic regions of the avatar. Our algorithm reduces the total number of gaussians needed while minimizing the impact on representation power. Our pipeline operates in a single forward pass, and achieves SoTA reconstruction quality, novel view synthesis quality, and enrollment/rendering speeds compared to other single-image face modeling methods.