Surabhi Gupta, Ashwath Shetty, Avinash Sharma
This work has also been presented as an extended abstract, 'Facial De-occlusion Network for Virtual Telepresence Systems' at "Sixth Workshop on Computer Vision for AR/VR" in CVPR Workshop on Computer Vision for Augmented and Virtual Reality, New Orleans, LA, 2022. Check it out here.
Traditionally, video conferencing is a widely adopted solution for remote communication, but a lack of immersiveness comes inherently due to the 2D nature of facial representation. The integration of Virtual Reality (VR) in a communication/telepresence system through Head Mounted Displays (HMDs) promises to provide users a much better immersive experience. However, HMDs cause hindrance by blocking the facial appearance and expressions of the user. We propose a novel attention-enabled encoder-decoder architecture for HMD de-occlusion to overcome these issues. We also propose to train our person-specific model using short videos of the user, captured in varying appearances, and demonstrated generalization to unseen poses and appearances of the user. We report superior qualitative and quantitative results over state-of-the-art methods. We also present applications of this approach to hybrid video teleconferencing using existing animation and 3D face reconstruction pipelines.
- Paper
- Supplementary
- Code coming soon !
@INPROCEEDINGS{9866956,
author={Gupta, Surabhi and Shetty, Ashwath and Sharma, Avinash},
booktitle={2022 19th Conference on Robots and Vision (CRV)},
title={Attention based Occlusion Removal for Hybrid Telepresence Systems},
year={2022},
volume={},
number={},
pages={167-174},
doi={10.1109/CRV55824.2022.00029}}