ModelMine

Description

Large scale 3D model housing database for Architecture design

Idea

The Model Mine dataset addresses the historic problem of insufficient 3D annotated datasets in architectural computer vision. It offers expert-built 3D house models with numerous attributes, enabling style-based generation and detection.

We also present Model Mine Parts, a derivative dataset that uses Grounding Dino to detect and catalog parts in 3D from 2D renderings. This provides a comprehensive library of architectural objects in various styles.

Using these datasets, we introduce a 2-stage pipeline for generating 3D houses based on input text and reference images. A diffusion model creates view-consistent images, which are then used with camera poses to reconstruct a 3D model using neural radiance fields.

This tool enhances architectural design by combining our datasets and generative model, offering new levels of creativity and precision.

Workflow

Figure 1a displays examples of 3D models from the Model Mine dataset, each annotated with semantic attributes like Region, Roof and Door styles, etc. Figure 1b presents the pipeline used for generating the Model Mine Parts dataset, where we first detect architectural assets such as doors, windows, and roofs on 2D renderings of the model and then perform 3D lifting of these assets to obtain the final parts dataset. Figure 1c shows our generative pipeline. We first utilize a fine-tuned 2D diffusion model conditioned on text and image prompts to generate view-consistent images. Here, the text represents the style of the house and A1e, A2e …… Ane are the positional image embeddings of the assets input by user. Following this, a transformer-based sparse-view 3D reconstructor is applied to these images to generate the final NeRF representation

Achievements

MM selected for poster presentation at CVPR 2024!

Model Mine received UMOR-TCAUP RCI Award!

Key Contributors

Matias del Campo

Sandra Manninger

Janpreet Singh

Yuxin Lin