TLDR: FactoredScenes is a new AI framework that generates realistic 3D indoor scenes by decomposing the process into learning room structures (programs) and predicting object orientations (poses). It uses large language models to generate layouts based on a learned library of reusable patterns and a program-conditioned model to predict object poses hierarchically. The framework significantly outperforms prior methods and generates scenes that are difficult for humans to distinguish from real-world examples, addressing the challenge of limited real-world 3D data.
Creating realistic 3D indoor scenes that truly capture the essence of a lived-in space has long been a significant challenge in computer graphics and AI. Datasets of real-world scenes, such as ScanNet, are notoriously limited, making it difficult to generate diverse and believable environments with varied object arrangements and orientations. A new research paper, titled From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries, introduces an innovative framework called FactoredScenes that addresses this problem by breaking down scene generation into manageable, hierarchical components.
The team behind FactoredScenes, including Joy Hsu, Emily Jin, Jiajun Wu from Stanford University, and Niloy J. Mitra from University College London, recognized that despite the apparent randomness, real-world indoor scenes possess an underlying structure. Furniture is often arranged based on how people interact, and objects follow certain social norms and design principles. FactoredScenes leverages this insight by proposing a ‘factored representation’ that decomposes scene generation into two primary concepts: room programs and object poses.
How FactoredScenes Works
The framework operates in five distinct steps, allowing it to learn from various data sources and employ different methods for each component:
1. Learning a Program Library: Instead of relying on pre-defined rules, FactoredScenes learns a library of reusable layout patterns from a large dataset of synthetic, professionally designed indoor scenes called 3D-Front. This library contains functions like ‘align’ for arranging objects in a row, ‘grid’ for creating object grids, and ‘cluster_placement’ for grouping objects around a central point. This learning process uses a ‘wake-sleep’ framework with large language models (LLMs) to discover these structural relationships.
2. Generating Scene Programs: Once the library is learned, LLMs are used to generate high-level scene programs. These programs act like blueprints, encoding the underlying room structure. The LLMs are guided by the learned library and examples of real-world scene programs parsed from ScanNet, enabling them to create diverse and novel layouts that adhere to common-sense rules.
3. Executing the Program for Layouts: The generated program is then executed by a Python interpreter to produce axis-aligned layouts. These layouts define the basic placement and dimensions of objects within the scene.
4. Predicting Object Poses: This is where FactoredScenes captures the ‘lived-in’ quality. A specialized model, conditioned on the generated program and layout, predicts the precise orientation (pose) for each object. Crucially, it does this hierarchically: first predicting the poses of primary objects (like a table), and then predicting the poses of dependent objects (like chairs around that table) based on the primary object’s orientation. This step is trained on the limited real-world data from ScanNet, with the program acting as a regularization to ensure effective learning.
5. Retrieving Object Instances: Finally, the scene is populated with actual 3D objects. These objects are selected based on their class and dimensions to match the predicted oriented bounding boxes. They are then scaled, translated, and rotated according to the predicted poses, completing the full 3D scene.
Also Read:
- Efficient 3D Model Generation: A New Framework for High Quality and Low Storage
- Generative AI Helps Robots Navigate Unseen Spaces with Enhanced Prior Knowledge
Impressive Results and Human Perception
FactoredScenes has demonstrated significant improvements over previous state-of-the-art methods in generating realistic layouts, as measured by metrics like Fréchet Inception Distance (FID) and Kernel Inception Distance (KID). For instance, it showed a 38.3% FID improvement in bedrooms and a 40.1% FID improvement in living rooms compared to top prior works.
Perhaps the most compelling validation comes from a human study. Participants were shown pairs of rendered rooms – one from FactoredScenes and one from ScanNet – and asked to identify which was more realistic. The results indicated that FactoredScenes’ generated scenes were difficult for humans to distinguish from real ScanNet scenes, with a mean accuracy of 0.67 in choosing the real ScanNet scene. This suggests a high level of realism achieved by the framework.
The modular design of FactoredScenes is a key strength, allowing different components to be improved independently and to leverage various data sources. While the framework occasionally faces challenges with LLM consistency in program generation and the accuracy of ground-truth pose labels from ScanNet, its interpretable nature and strong performance mark a significant step towards synthesizing highly realistic, real-world 3D environments.
