Realtime Free Viewpoint Video System Based on a new Panorama Stitching Framework

Abstract

Free Viewpoint Video and TV is regarded as the future of digital entertainment allowing users to navigate through multiple video streams of an event to select novel viewpoints. This new capability will be able to give to the users the illusion that they are present at the event. In this thesis, we propose a novel framework for real-time video panorama generation. We first present our work for acquiring multiple high-definition video streams using hardware synchronized cameras with GPU accelerated Bayer demosaicing. The multiple streams are then color corrected using camera gain adjustment and Macbeth color checker chart to remove color variations between cameras. Next, we propose an automatic real-time video stitching algorithm using feature matching based approach and automatic camera calibration that is capable of producing spherical panoramas. The algorithm works by splitting the automatic image stitching steps into one-time and recurring computations. We utilize GPUs for accelerating the recurring computations and achieve real-time performance. By effectively caching the repeated transform maps database, we improve the stitching performance by two orders of magnitude compare to CPU implementations. In order to remove seams between the stitching images, we use Vornoi diagram based optimal seam selection algorithm. The stitched video streams are blended together using the first real-time implementation of Laplacian pyramid blending algorithm using multi-GPUs. Finally, we analyze the problems associated with the scalable transmission and rendering of the virtual views using commodity computers and network resources. The multiple stitching steps are pipelined to distribute the panorama stitching computations between multiple processing nodes to produce visually appealing panoramas at 30Hz without any noticeable blurring or stitching artifacts.