Neural reconstruction and super-resolution for foveated real-time rendering

Rendering high-resolution photorealistic images in real time is challenging for video games and emerging virtual reality headsets. Thus, fovea sampled image reconstruction and super-resolution technologies become more and more crucial. However, most current methods process foveated reconstruction and super-resolution separately, which is slow. To address this issue, we propose a novel multi-scale spatiotemporal kernel prediction network for real-time foveated rendering that can perform sparse peripheral region reconstruction and supersampling simultaneously, resulting in a substantial reduction in rendering computation without visually noticeable quality degradation. Thanks to the multi-scale kernel prediction architecture, different levels of details can be effectively preserved. Furthermore, we introduce an effective motion vector mask to explicitly identify occluded regions, which can help to use historical information more effectively. Our network runs in real time and achieves superior image quality and better inter-frame stability than existing methods.