Remember meForgot password?
    Log in with Twitter

article imageGoogle's headset removal tech lets you see faces in mixed reality

By James Walker     Feb 24, 2017 in Technology
Google has developed a system that lets you see the faces of people participating in mixed reality environments. The headsets used in current mixed reality setups completely mask the face of the wearer, hiding the visual cues which indicate emotion.
In mixed reality, you can collaborate with other people to create a shared experience. It's part of the core appeal of the technology but is currently fundamentally lacking. Because the headset obscures the wearer's face, you can't see what they’re thinking or gauge their reactions.
Google's Research labs teamed up with its Daydream VR team to attempt to solve this problem. Using a combination of 3D computer vision and machine learning, the company managed to create a solution that overlays the headset wearer's face onto the mixed reality output in real-time.
First, the headset wearer sits in front of a computer which generates a 3D model of their face. It can monitor and record differences in the face as the person blinks, looks around or moves their head. This data can be used to accurately replicate facial expressions as the person moves around in the mixed reality world.
With the system setup, the headset wearer can begin to participate in mixed reality as normal. A specially-modified HTC Vive is used. It features eye-tracking technology that continually monitors the wearer's eye-gaze from inside the headset. This information is matched against the previously captured facial data to work out the wearer's current expression in real time.
Finally, the 3D face model is overlaid on the camera's video stream, letting you see the other person's face when wearing your own headset. A complex compositing system is used to work out where to position the rendered face and how it should move relative to the rest of the scene. The Vive tracking system and external camera must be precisely calibrated to achieve this.
Google s headset removal technology
Google's headset removal technology
Google Research
The result lets you watch people's faces as they interact with the world, rather than gazing into the blank front of a VR headset. Google overlays the face with a translucent version of the headset, making it obvious the other person is in VR.
The "scuba mask" also helps avoid the "uncanny valley" effect. Because people can detect even the slightest abnormality in a human face, rendering errors could be deeply uncomfortable if the image isn't filtered. By overlaying a translucent headset, the viewer is continually reminded that everything is virtual. Minor flaws in the tracking system are more likely to be overlooked.
Google has already used its headset removal tech in its Tilt Brush mixed reality painting app. It has wider visions for the system though, seeing it as something that could make VR communications not only natural but also normal. A VR video calling app today would leave you gazing at identical VR headsets. By rendering the wearer's faces, the concept becomes more usable.
"Our initial work focused on mixed reality is just one potential application of this technology. Seeing beyond VR headsets could help enhance communication and social interaction in VR," said Google. "Imagine being able to VR video conference and see the expressions and nonverbal cues of the people you are talking to, or seeing your friend’s reactions as you play your favorite game together."
Google is continuing to develop the technology as part of a collaboration between multiple teams. People on the YouTube team are also involved, seeking to bring the tech to the YouTube Spaces VR section. Google suggested it could bring headset removal to content creators in the future.
More about Google, Virtual reality, mixed reality, google daydream
Latest News
Top News