The estimation and reconstruction of 3D architectural structures is of great interest in computer vision, as well as cultural heritage. This dissertation proposes a novel approach to solve the difficult problem of estimating architectural structures from sparse images and efficiently generating 3D models from estimation results for cultural heritage. This approach takes as input one plan drawing image and a few façade images, and provides as output the volumetric 3D models which represent the structures in the sparse images. Support of this research goal has motivated new investigations in underlying structure estimation problems including detecting structural feature points in 2D images, decomposing plan drawings into semantically meaningful shapes for medieval castles, estimating rectangular and Gothic façades using shape priors, and estimating complete 3D models for architectural structures using a novel volumetric shape grammar. Major outstanding challenges in each of these topic areas are addressed resulting in contributions to current state-of-the-art as it applied to these difficult problems.