TLDR: TrueCity is a novel benchmark dataset for 3D urban semantic segmentation that provides synchronized real-world and simulated point clouds of the same city. It addresses the challenge of limited real-world annotated data and the domain gap between synthetic and real data. The dataset features cm-accurate real point clouds, semantic 3D city models, and simulated data with standardized classes. Experiments show that synthetic data can significantly improve model performance, especially for transformer-based architectures, and that a balanced mix of synthetic and real data often yields optimal results, particularly for geometrically regular classes. However, complex or fine-scale objects like windows and doors still pose a significant challenge for synthetic data.
The field of 3D computer vision, particularly in understanding complex urban environments, faces a significant hurdle: the scarcity of high-quality, annotated real-world 3D data. While synthetic data offers a scalable and cost-effective alternative with perfect labels, it often struggles to capture the intricate complexity and sensor noise of the real world, leading to a notable ‘synthetic-to-real domain gap’. This gap makes it challenging to develop models that perform well in real-world applications after being trained on simulated data.
Addressing this critical issue, a new benchmark dataset called TrueCity has been introduced. TrueCity is a pioneering effort, offering the first urban semantic segmentation benchmark that provides centimeter-accurate annotated real-world point clouds, detailed semantic 3D city models, and annotated simulated point clouds, all representing the exact same urban location. This unique synchronization allows for a coherent and in-depth analysis of the domain shift between synthetic and real data, which was previously difficult due to the lack of synchronized datasets.
One of TrueCity’s key innovations is its proposal of novel urban semantic segmentation classes. These classes are carefully aligned with international 3D city modeling standards, such as CityGML 2.0 and OpenDRIVE 1.4. This standardization is crucial because it enables consistent evaluation of the synthetic-to-real gap and facilitates seamless integration with existing urban planning and management workflows. By harmonizing class definitions, TrueCity helps overcome the heterogeneity often found in urban class definitions across different datasets, which can hinder the development of robust transfer learning approaches.
The creation of the TrueCity dataset involved a meticulous process. For the real-world data, the inner city of Ingolstadt, Germany, a challenging urban setting, was selected. High-precision laser scans were conducted using a Mobile-Road-Mapping-System (MoSES), capturing over 113 million points with high density and accuracy. The semantic annotation of these real-world point clouds followed a three-step process: dividing points into connected components, separating ground surfaces, and then manually assigning class IDs from the standardized list. This process resulted in a dataset reflecting the realistic long-tail distribution typical of real-world data.
To generate the synthetic data, a semantic 3D city model was first meticulously created by manually modeling buildings, facades, and road networks based on the acquired real point clouds and converting them to the CityGML standard. This detailed model served as the foundation for simulating laser scans. The CARLA driving simulator, equipped with a configurable LiDAR sensor model, was used to replicate the real-world scanning process, including the incorporation of a Gaussian-distributed range error to enhance realism. It’s important to note that the simulation primarily accounts for static environment objects, meaning dynamic elements like vehicles and pedestrians were excluded to maintain clarity in domain shift analysis for static scene understanding.
Extensive experiments were conducted on TrueCity using various semantic segmentation baselines, including point-based, kernel-based, and transformer-based models. These experiments evaluated performance under different synthetic-real (S–R) mixtures, ranging from purely synthetic (100S–0R) to purely real (0S–100R) training data, while keeping the validation and test data strictly real-only. The findings revealed that synthetic data can indeed be beneficial, but its utility often depends on the model’s architecture and the specific object classes.
For instance, transformer-based models with global attention, like Point Transformer v3, showed significant improvements with balanced mixes (e.g., 50S–50R), sometimes even outperforming models trained solely on real data. Classes with geometric regularity and simple structural contexts, such as WallSurface, RoadSurface, GroundSurface, and SolitaryVegetationObject, exhibited an ‘insignificant domain gap’. For these, synthetic data provided a strong baseline, and even small amounts of real data were sufficient for refinement.
However, other classes presented a ‘significant domain gap’. Fine-scale geometric objects or those with complex contextual cues, like Door, RoofSurface, and BuildingInstallation, remained challenging, with performance rarely exceeding 8% IoU. These categories often depend on subtle details and contextual information that are typically absent or oversimplified in simulations. The ‘Noise’ class, representing diverse real-world variations, also showed substantial improvement with real data. Interestingly, ‘CityFurniture’ often displayed a U-shaped performance trend, suggesting that intermediate synthetic-real ratios could trigger domain conflicts, while fully real data resolved these inconsistencies. ‘Window’ segmentation also proved difficult due to small size, thin geometry, reflections, and occlusions, which synthetic data struggles to capture.
While TrueCity offers unprecedented accuracy and quality, the authors acknowledge certain limitations. The high acquisition cost means it cannot match the scalability of image-based datasets. Additionally, the current focus is on static objects, as incorporating dynamic objects with accurate, synchronized simulation would introduce significant complexities for domain shift analysis. Nevertheless, the dataset provides the foundational data for future research into dynamic object simulation.
Also Read:
- Advancing Autonomous Driving with the Pandar128 Lane Line Dataset
- Advancing 3D Point Cloud Generation: New Metrics and a Transformer Model
In conclusion, TrueCity stands as a crucial benchmark for the 3D computer vision community. It enables a deeper, more coherent analysis of the domain gap between real and simulated urban scenes, removing uncertainties from fictitious environments. The research demonstrates that simulated point clouds can partially substitute real ones without compromising performance for many tasks, and in some cases, even enhance it. It also clearly highlights areas where current simulators fall short, particularly for material-complex objects like glass or intricate non-manmade structures. This dataset is poised to foster further advancements in sim-to-real gap quantification and the development of generalizable data-driven models. More information about TrueCity can be found at the research paper.
