YouTube announces mobile real-time video segmentation

Google has announced a new video segmentation with real-time to the YouTube app that will be integrated to the latter’s stories feature designed specifically for YouTube creators on its beta version.

The new on-device mobile video segmentation is a lightweight video format, which will allow creators to replace and modify the background, effortlessly increasing videos’ production value without specialised equipment. The new technology has been developed using machine learning to solve a semantic segmentation task using convolution neural networks.

Video segmentation is a widely used technique that enables movie directors and video content creators to separate the foreground of a scene from the background, and treat them as two different visual layers. By modifying or replacing the background, creators can convey a particular mood, transport themselves to a fun location or enhance the impact of the message. However, this operation has traditionally been performed as a time-consuming manual process (e.g. an artist rotoscoping every frame) or requires a studio environment with a green screen for real-time background removal (a technique referred to as chroma keying). “In order to enable users to create this effect live in the viewfinder, we designed a new technique that is suitable for mobile phones,” reads its blog.

“Our immediate goal is to use the limited rollout in YouTube stories to test our technology on this first set of effects. As we improve and expand our segmentation technology to more labels, we plan to integrate it into Google’s broader Augmented Reality services,” the blog states.