TLDR: StyleMM is a new framework that creates stylized 3D human faces from text descriptions. It uses text-guided image stylization to fine-tune 3D models, ensuring consistent facial structure, independent control over shape and texture, and a wide range of artistic styles. It outperforms previous methods in diversity and stylization quality, making it easier to generate animatable 3D avatars for games and animation.
Creating expressive and diverse 3D characters for movies, animations, and games often requires significant time and effort from artists. While traditional 3D Morphable Models (3DMMs) are excellent for generating realistic human faces, they fall short when it comes to producing stylized characters like those seen in Pixar films or fantasy worlds.
A new framework called StyleMM, developed by Seungmi Lee, Kwan Yun, and Junyong Noh from KAIST’s Visual Media Lab, aims to bridge this gap. StyleMM introduces a novel way to build stylized 3D Morphable Models directly from user-defined text descriptions, making the creation of unique 3D avatars more accessible and efficient.
The researchers identified three crucial requirements for a truly effective stylized 3DMM: first, maintaining consistent point-to-point correspondence across different faces, ensuring that features like eyes and mouths always align correctly. Second, offering disentangled control over shape and texture, meaning you can change a character’s face shape without altering its texture, and vice versa. Third, the ability to achieve expressive stylization that goes far beyond realistic human appearances.
Previous methods often struggled to meet all three criteria simultaneously. Some could stylize shapes but not textures, while others produced high-quality stylized faces but lacked the consistent structure needed for animation or easy editing. StyleMM addresses all these challenges by building upon existing 3DMMs designed for realistic faces and fine-tuning them with a clever text-driven approach.
The core of StyleMM lies in its use of text-guided image-to-image (i2i) translation with diffusion models. Imagine you have a realistic 3D face model. StyleMM renders this model into an image, then uses a text prompt (e.g., “Pixar child” or “green Orc”) to transform that image into a stylized version. These stylized images then serve as targets for training the 3D model.
A significant challenge with this image-based training is preventing unwanted changes to the character’s identity, facial alignment, or expressions during the stylization process. To overcome this, StyleMM introduces Explicit Attribute-preserving Stylization (EAS) and its component, the Explicit Attribute-preserving Module (EAM). EAS uses specific facial attributes like sparse landmarks (eyes, nose, lips), head rotation, and expression to guide the stylization, ensuring that the core facial structure and identity remain consistent even as the style changes dramatically.
The training process for StyleMM is divided into three stages to ensure robust results. First, a “geometry warm-up” phase focuses on establishing an accurate geometric foundation. This is followed by “joint fine-tuning” of both shape and texture, where the model learns to match the stylized images in terms of overall appearance and facial part layout. Finally, a “texture refinement” stage enhances the fine-grained details of the textures. A key innovation during this training is the Consistent Displacement Loss (CDL), which helps maintain the diversity of different identities within a chosen style, preventing all faces from converging to a similar look.
StyleMM has demonstrated impressive results, outperforming state-of-the-art methods in both identity-level facial diversity and stylization capability. Qualitative comparisons show that StyleMM can generate a wide range of text-driven styles while preserving the ability to control shape, expression, and texture independently, just like a traditional 3DMM. A user study further confirmed its superiority in diversity, style fidelity, and overall quality compared to other methods.
The applications of StyleMM are significant for digital content creation. It enables video-driven facial animation, allowing artists to easily transfer performances from real videos onto stylized 3D avatars without complex rigging. It also supports 3D face stylization from input images, transforming existing faces into stylized versions while maintaining their identity. Even details like eyeballs, initially excluded for simplicity, can be added back through a post-processing step, further enhancing realism and appeal.
While StyleMM represents a major leap forward, the researchers acknowledge some limitations. Extreme stylization can sometimes introduce minor misalignments, and strong mesh stabilization losses, while crucial for plausible geometry, might suppress very sharp stylistic details. Future work will focus on refining alignment strategies for highly artistic styles and incorporating multi-scale structural priors to better preserve fine details without compromising geometric consistency.
Also Read:
- Enhancing Visual Perception with Decoupled Learning: Introducing DeCLIP
- Enhancing MLLM Accuracy: A New Method for Controlled Image Captioning
StyleMM makes the creation of high-quality, animatable stylized 3D avatars more accessible, potentially becoming a standard tool for professional artists, small studios, and game developers alike. You can find more details about this research paper at https://arxiv.org/pdf/2508.11203.
