Our recent Face AR SDK release includes improved real-time body segmentation with deep learning that runs on mobile and the web. With this, you can recognize people in video, remove backgrounds or use it to create AR body filters. Here we explain how our body recognition software works and its use cases.
What is Human Body Segmentation
Body segmentation lets you extract the human in video flow or an image to cut out the backgrounds or overlay an effect. The result is achieved by solving a semantic segmentation task. It means classifying every pixel in an image, whether it belongs to a human silhouette or the background.
Body Segmentation Technology Demo
One should not confuse it with body tracking, which builds on key points of a human body (18-20 average) and returns a moving skeleton model. Body segmentation doesn’t track points but creates a mask. In its idea, the technology is pretty similar to background subtraction for selfies but features different use cases.
Full Body Segmentation vs Selfie Segmentation
Both body and selfie segmentation build on neural networks and are mostly used for background replacement. The difference is as follows.
Selfie segmentation
As it’s prompted from the term ‘selfie’, the technology is trained on datasets of upper torso portraits. The major use cases are placing virtual backgrounds for video conferencing, removing backgrounds in video calls or blurring them, creating a bokeh effect with a camera.
It supports a shorter distance between a person and a camera. On a longer distance, the technology may return false coloring or fall apart.
Body segmentation
Extends video communication use case to entertainment Snapchat-like effects, photo and video editors, clip production and many more, which we’ll cover later.
It supports a longer distance allowing for the entire body placed into the camera view.
Technology
Our body recognition technology is based on the Fully Convolutional Neural Network of UNet type and fast backbone MobileNetV3. The network processes an RGB image and returns a semantic mask of "probabilities" that the pixel belongs to a person.
The tech stack comprises a learning pipeline developed utilising Tensorflow and Keras, open-source frameworks built with Python. The trained model is then converted to the TFLite to make it run smoothly on mobile devices. We trained body segmentation to achieve high precision and stable performance at different conditions and platforms.
Precision
The required precision and stable output were met by augmenting the training dataset with human body samples that addressed problematic cases. It includes different lighting conditions, body angles and a variety of backgrounds.
Distance
Body recognition can handle a wide range of distances from the segmented person to the mobile device camera. The height of the silhouette must be located within the camera view and must not be smaller than half of the input image's height.
Multi body
The network can detect and track multiple people in the image frame. You can automatically remove and change backgrounds for all of them.
Cross-platform
The person segmentation technology runs on the web, iOS and Android platforms, meaning you can integrate it into your website or mobile app.
- iOS: top iOS devices down to iPhone 7.
- Android
- Web: all browsers
Where you can use body segmentation?
Entertainment
Remember a Black Mirror episode where one could block a person who then appeared as a blurred silhouette? Body segmentation lets you reproduce this filter in entertainment apps. The most well-known example is Snapchat which provides effects with body segmentation. If you want a similar technology in your Snapchat-like app, our person segmentation lets you create AR body filters and effects.
Video communication
Let users remove, replace or animate background for a full body in video conferencing or live streaming apps.
Photo and video editing
Source: Snapchat
Apps that allow you to remove an unwanted person from images are already available in App Store and Play Market. In most of them, you need to manually point the area for erasement, but with body segmentation, this process becomes automated and way more fast. You can bring more features with AR video editing to your users, let them create short videos, collages, and even professional movie-like clips.
Photo booth
You can use body segmentation in AR advertising, interactive photo booths, virtual stands and exhibitions. Visitors can place any background, be it a static image or video. Full body subtraction lets you immerse users on a deeper level and give them more freedom to move in front of the AR mirror.
Interested to test our body segmentation tech in your app?
Get in touch!