High-Fidelity Image-Based Modeling and Rendering – HFIBMR

There is an increasing need for three-dimensional (3D) ``content'' in entertainment, engineering, and scientific applications. We propose core computer vision research that will enable the development of advanced image-based modeling and analysis software, transform consumer-grade cameras into flexible, affordable and accurate 3D sensors, and allow the widespread acquisition and use of 3D content in engineering and science. Concretely, we will focus on high-fidelity image-based modeling and rendering, and we will demonstrate applications of the technology developed in this project to film post production and special effects, and cultural heritage conservation, both pursued via collaborations with external partners. We will focus on three core computer vision research problems. * Core image-based modeling: we propose a novel approach to calibrated multi-view, wide-baseline stereopsis, that we recently proposed, as a core technology for image-based modeling. Our preliminary experiments demonstrate the recovery of sub-millimeter relief detail for objects about 20cm in diameter (a relative accuracy better than 1 part in 200, comparable to mid-level laser range scanners, for a fraction of the cost). Our goal is to go much further, and advance the state of the art of image-based modeling on three parallel fronts: (1) geometric and photometric fidelity, with a target of 10- to 100-fold resolution improvement compared to current computer vision technology; (2) acquisition flexibility, with the objective of eliminating the need for the cumbersome calibration charts and markers currently used in photogrammetry; and (3) an optimization approach to image-based modeling, with the objective of giving a clear, formal statement of the multiview stereopsis as a combinatorial problem, amenable to optimization. * Physical and statistical shape models: we propose to construct realistic physical and statistical shape models and to integrate them as shape priors into the image-based modeling and rendering process. We propose research in three major areas: (1) constructing physical shape and reflectance/illumination models, with the objective of better modeling non-Lambertian surfaces ; (2) learning shape priors from image data, with the goal of designing practical 3D models capturing essential modes of shape evolution and/or appearance change; (3) integrating physical and learned priors in the image-based modeling process, with the objective of improving the robustness and accuracy of the image-based modeling process. * Advanced image-based rendering: after the acquisition of our image-based models, we will focus on rendering them, as we want to integrate our image-based models inside classical geometrical scenes. Our work will focus on efficient and realistic rendering of image-based models, including lighting, shadowing and re-lighting effects. We will focus our research in three major directions: (1) efficient rendering of image-based models, with a rendering cost proportional to the visual complexity of the model, not to the image resolution; (2) self-shadowing and masking effects: just like parts of the models are hiding each other in the acquisition process, they are also hiding each other in the rendering process. This has an influence both on the picture of the model (masking effects) and on the illumination of the model (self-shadowing effects); (3) re-lighting of the models: the image-based model was acquired under specific lighting conditions. For a realistic rendering under different lighting conditions, we need to model the influence of lighting on the image-based model, including both direct and indirect lighting. We will apply this core technology in interdisciplinary efforts with high scientific, societal, and/or economic impact. In particular, we will target film post production and special effects, and cultural heritage preservation. These efforts will involve collaborations with Industrial Light and Magic (ILM) and the Getty Conservation Institute (GCI). The majority of the requested funds will be used to support one PhD student in the WILLOW project, one postdoctoral fellow for 18 months in the ARTIS project, and one PhD student for 18 months in the COMSEE project. It is expected that they will visit ILM and GCI.