Light Transport Acquisition and 3D Reconstruction in the Presence of Light Refraction

Abstract

3D reconstruction is an important topic in both computer vision and computer graphics. Many techniques have been proposed for objects with Lambertian reflectance. It assumes that the reflected light from the object surface is uniformly distributed in all directions. However, light interacts with real-world objects in complex manners, e.g. refraction, scattering and specular reflection. By ignoring these effects, traditional methods, when applied directly, produce large errors. For example, due to light refraction, a transparent surface appears differently when observed from different viewpoints. Thus the traditional color/texture correspondence-based methods cannot be used. This dissertation presents novel hardware setups and software designs for 3D reconstruction in the presence of light refraction.I start with capturing the light transport characteristics, i.e. the environment matte, of objects that are either refractive or reflective, or both. The proposed approach can locate the contributing light sources at the pixel level and render photo-realistic images of the object under novel illumination background.Then I propose to exploit the light transport for reconstructing 3D shape of transparent and refractive objects. In particular, a novel imaging setup is built to capture the light rays before and after refraction. By introducing a novel normal consistency constraint that encodes the light refraction effect, I design an optimization procedure, which jointly reconstructs the 3D positions and normals of the object, as well as the refractive index.I also present a new method to recovering 3D dynamic fluid surfaces by leveraging light refraction. Two cameras are used to capture the distortion of a random pattern through the wavy fluid surface. After estimating the correspondence between the captured image and the original pattern, I develop a refraction-based optimization framework for recovering the 3D shape and the refractive index of the fluid surface.Finally, I consider the imaging scenario of viewing an underwater scene through a water surface. By explicitly accounting for light refraction at the water surface, I present a novel approach for simultaneously recovering the 3D shape of both wavy water surface and the moving underwater scene.