Unity is the world's most popular cross-platform game engine.
However, developers still face a range of limitations when it comes to building face tracking games and apps in Unity.
One of them is the ability to separate a person from the background in real-time.
In this post, you'll learn how to create a background Unity subtraction featuring:
- Why is Unity background segmentation beneficial
- 4 use cases for background subtraction
- How to subtract background in Unity with Banuba.
[navigation]
TL;DR:
- Background subtraction is useful for games, video calling apps, and live streaming software;
- It can be integrated using Banuba's Face AR plugin for Unity;
- See the step-by-step installation instructions below.
Why Background Subtraction in Unity
Being the most popular cross-platform game engine, Unity still provides limitations.
One of them is real-time virtual background which is crucial for building face tracking games and apps in Unity.
Our background remover is designed specifically for a selfie-focused experience. It works accurately with all skin tones and hides whatever is behind the person in real-time.
Our Face AR SDK for Unity now supports this feature along with face detection and tracking, allowing users to replace backgrounds in real-time.
What's more, the backdrops are compatible with face filters and support multi-face tracking.
You can build cross-platform games with face tracking, video chats, and augmented reality software in Unity or even use this feature for mixed reality projects.
You can place the user in front of a green screen, and replace it with images, animated wallpapers, or videos.
Background segmentation in Unity by Banuba
The existing Unity chroma key tools select the color to be removed, e.g. green used in physical green screens. Developers need to avoid colors that match the skin tone so it won’t blend into the background.
Our Unity background segmentation is based on the neural network trained to recognize whether a pixel belongs to the class 'person' or 'background', enabling automatic real-time background separation.
Unity Background Segmentation: Technical Features
We tested how background removal works with other Face AR features in Unity both on early and latest devices (iOS, Android) and the results are pretty good.
Unity for iOS, FPS
| iPhone 5s | iPhone Xr | |
| Background | 21 | 30 |
| Background + Face Filter | 18-20 | 30 |
| Background 2 faces | 16-19 | 29-30 |
Unity for Android, FPS
| Galaxy A 50 | Galaxy S 10+ | |
| Background | 24-30 | 29-31 |
| Background + Face Filter | 17-19 | 29-31 |
| Background 2 faces | 25-29 | 29-30 |
4 Background Subtraction Use Cases
The demand for virtual backgrounds has grown fast in recent months, becoming a number one feature for any face tracking and video chatting app.
Here are 4 common use cases when Unity background subtraction becomes a must-have feature for your app.
Games with face tracking
You can build face-to-face games in Unity where players can see each other or themselves while playing. Players may have fun changing backgrounds, applying face filters, or even full-face avatars if they don’t want to show their faces.
Hyperspeed | Face Tracking Case Study
Video chat apps
Video chat apps are gaining more popularity. By adding Face AR features, you can make the user experience more immersive and fun, allowing users to change boring backgrounds into scenery per their choice.
Live streaming
Background replacement for streaming apps brings digital privacy and comfort, letting users not bother about their real surroundings. Users can easily create high-quality streams teleporting themselves into the center of the video featuring their favorite game, movie, or just abstract animation.
Unity Chroma Key (Green screen)
Our technology can be perfectly used as an alternative to Unity Chroma Key tools to create mixed reality and augmented reality videos and experiences.
Developers can record the user without the background and then overlay video animation to get high-quality video clips.
Integration
This is how you can integrate a background replacement SDK in your Unity project. It will also allow you to use other AR features, if you wish.
1. Obtain a trial token
This is simple. Just send us a message through a contact form.
Run
- Clone the repository
- Download the latest libs package BanubaSDK-quickstart.unitypackage and import it using Unity.
- Put your Client Token to the Assets/Resources/BanubaClientToken.txt
- Open the Assets/Scenes/SampleScene and click Run.
Import To Your Project
- Download the latest assets package BanubaSDK-import.unitypackage and import it using Unity.
- Put your Client Token to the Assets/Resources/BanubaClientToken.txt
- Open the Assets/BanubaFaceAR/Demo and click Run.
Note:
if you use BanubaSDK-import.unitypackage you need to add BanubaFaceAR/BaseAssets/Scenes/LoaderScene.unity for Windows and Android Build and set Scene for loading like on the image below:
Banuba Face AR SDK Unity integration
-
The official technical documentation for Banuba Face AR SDK is available online and includes requirements, setup instructions, and configuration details:
- Banuba Face AR SDK iOS documentation
- Banuba Face AR SDK Android documentation
- Banuba Face AR SDK Flutter documentation
- Banuba Face AR SDK React Native documentation
- Banuba Face AR SDK Web documentation
- Banuba Face AR SDK Unity documentation
- Banuba Face AR SDK Desktop documentation
This Banuba Face AR SDK documentation should be your primary reference during integration.
-
You can access official integration code samples for the Banuba Face AR SDK on GitHub. Banuba provides platform-specific examples to help you integrate faster:
- Banuba Face AR SDK iOS integration code sample
- Banuba Face AR SDK Android integration code sample
- Banuba Face AR SDK Flutter integration code sample
- Banuba Face AR SDK React Native integration code sample
- Banuba Face AR SDK Desktop integration code sample
- Banuba Face AR SDK Web integration code sample
- Banuba Face AR SDK Unity integration code sample
These Banuba Face AR SDK repositories include setup instructions and minimal working examples for quick deployment.
-
Yes. The Banuba Face AR SDK includes an LLM-friendly documentation file designed for AI-assisted development and internal knowledge systems:
This structured Banuba Face AR SDK text file can be used with AI tools to streamline onboarding and developer workflows.
-
The Banuba Face AR SDK provides official cross-platform plugins to simplify integration into hybrid applications:
These Banuba Face AR SDK plugins are maintained by Banuba and support faster implementation for cross-platform apps.
-
The purpose of background subtraction is to detect and separate moving objects in video or image processing. The most common use cases are AR-powered face tracking apps, video chat apps, and live streaming.
-
The background subtraction algorithm in computer vision means detecting moving objects in video content. It separates objects from the background utilizing so-called classes which identify the background and input images.
