Sparse Input Novel View Synthesis of Dynamic Scenes

Abstract

Novel view synthesis involves generating unseen perspectives of a scene based on videos captured from limited viewpoints. It is an interesting problem in computer graphics and computer vision, with many applications such as virtual and augmented reality (AR), film production, autonomous driving, and sports streaming. Methods to model static radiance fields, such as neural radiance fields and 3D Gaussian splatting, have achieved remarkable results in synthesizing photo-realistic rendering of novel views. However, learning scene representations of a dynamic scene introduces several challenges in modeling the motion in the scene. Further, existing models require dense viewpoints to generate good-quality rendering. The performance of these models goes down significantly as we reduce the number of viewpoints. This thesis focuses on the problem of dynamic view synthesis for sparse input views. In the first part of this thesis, we focus on studying reliable and dense flow priors, to constrain the motion in dynamic radiance fields. We propose an efficient selection of dense flow priors, as naively obtaining dense flow leads to unreliable priors. In the second part of this thesis, we study the challenges introduced by volumetric motion modeling. Specifically, we address the limitations of unidirectional motion models, leading to many-to-one mapping of points. We enforce cyclic motion consistency with the help of bidirectional motion fields to achieve superior reconstruction of novel views of dynamic scenes. Further, the design of the bi-directional motion field allows us to track object motion in synthesized views.