How To Make a Video Chat with Filters

Video chat with filters: the basics

“AR filters” is an umbrella term for a host of visual effects. These may include:

• LUTs (Instagram filters). A basic color scheme change for an entire picture/frame. For example, sepia or black-and-white.
• Background replacement. Detecting a person in a picture and changing the rest. The new background can be a blur, a static image, a GIF, a video, or even a 3D environment.
• Face masks and stickers. The effects that TikTok and Snapchat helped make popular. Masks are complex 3D objects, while stickers are simpler and two-dimensional. Read the article on this topic for more information.
• Beautification. Subtle changes that improve the person’s appearance: smoothing the skin, whitening teeth, removing acne and eye bags, etc. 
• Virtual makeup. Applying mascara, lipstick, eyeliner, and other cosmetic products as AR effects. Today they can be realistic enough to be used for virtual try-on.
• Avatars. 3D characters that replace the user on screen but mimic the person’s movements and facial expressions. 
• Overlays. Images, text, and GIFs placed over the video feed. 
• Triggers. Effects that are launched by making a gesture or a certain facial expression.

Most of these effects are based on neural networks that track certain facial features. There are more things to take into account if you are a developer and intend to make your own video chat app with filters:

• Real-time work. Some AR technologies only can transform your face on a static picture, as they aren’t optimized for real-time video feed. 
• Anti-jitter. In many situations, the camera in the person’t hands will be shaking slightly. This will affect the image quality, so an anti-jitter algorithm is needed to stabilize the feed.
• Multi-face tracking. What if there are several people on one screen (e.g. in a group chat) and you need to apply filters and effects to all of them? Face detection algorithms are scalable but there are still nuances to their implementation.
• Occlusion and lighting. Not all cameras are created equal. And people will not always have perfect conditions for seamless AR experiences. This means you need to account for situations when the lighting is bad or a person is obscured. 
• Video settings. Adjusting the parameters of a video feed automatically, or letting users do this. Some filters and effects can also be tuned, e.g. the degree of smoothing the skin or the size of a virtual hat.

Beautification at work

AR chat success stories

Rather than being just a standalone type of apps, video chats are often a part of a larger whole.

Social media 

TikTok has been a top dog for a while now and its competitors try to win the user base back. Filters provide opportunities to create engaging and viral content, which motivates people.

However, there are many other similar examples. Chingari is an Indian video-based social network. When New Delhi banned several Chinese apps, including TikTok, the company sprang into action. They launched an app with a distinctive Indian style, partially achieved by using custom face filters designed specifically for their country. As a result, they have over 50 million installs and the number keeps growing.

Virtual meetings

Professional video calls are popular among business users, politicians, lawyers, and many other people. So many online meeting providers use augmented reality to improve the group call experience.

Some, like Google Meet, use inbuilt functionality to provide background replacement - the core feature that helps protect users from embarrassment. Others go a different route. For example, Agora provides an SDK for conference calls. They have also launched an extensions marketplace where companies can buy complementary modules: sound suppression, automated moderation, and, of course, AR filters and effects. 

Casual video calls

Sometimes people just want to chat with one another while using fun effects. So the messengers adjusted accordingly.

Snapchat video calls have been popular for a while now. The app now has 319 million daily active users and the number is growing - 3 months ago it was 306 million. The company actively pushes its filters, for example by letting users make their own “lenses” or selling branded effects.

Live streaming

This can include anything from gaming marathons to live shopping. In all cases, the filters have their use.

For example, Restream lets users show live feeds on several social networks at once. They currently deliver 8+ millions broadcasts per month. And they have AR filter capabilities, although they use integration with Snapchat to deliver them.

Building a video chat with filters: concept

Now let’s get to the actual app.

The first step is defining the type of the app, its target audience? and its positioning. 

You probably already know what kind of software you want to create (e.g. live streaming service). But it is important to see other options in case the business environment changes or your expectations turn out to be different from reality. YouTube used to be a dating service before it became the video hosting powerhouse it is today.

Specifying a target audience is a way to narrow down the niche you want to operate in. This will help you make informed decisions about design, marketing, feature set, etc. For example, Discord and Slack are both messaging platforms with similar functionality. But they cater to different audiences (gamers and professionals) and so have different approaches.

Positioning is a way to distinguish yourself and show users your unique offering. For example, TikTok could have been another social network but it showed something that no one else did before: fully automated video feed. This is something that can make or break your app.

Building a video chat with filters: monetization

In the vast majority of cases, the app you are working on will be a for-profit venture. As such, you need to think about how it will earn you money. These are the most common options:

• Advertising. If you want to keep the product free for users you can sell ad space. People watch the ads - you earn. Everyone is happy. Note that filters and effects can be a form of advertising too. For example Snapchat app sells branded filters.
• Freemium. Keeping advanced features behind a paywall. Certain high-level filters can be one of the paid options.
• License. Selling temporary or lifetime licenses is very common in both B2B and B2C software. This is a way to ensure a steady income stream.
• In-app purchases. Filters can be turned into a commodity and sold to the users. 
• Cut of donations. One of the monetization options used by TikTok is taking a cut of the donations to the content creators that use their platform. 
• Free. If your app is not intended to directly earn money, you can keep it free entirely. 

Integrating face filters

In the vast majority of cases, building such AR technology from scratch is unviable. Firstly, it is very expensive. You need to keep a team of skilled developers on hand for at least six months to even build an MVP. If you are a startup, this is a huge drain on your budget, and if your main focus isn’t technology, this is plain unviable. Secondly, you will need time to actually get everything working. And it would slow down your eventual entry into the market. 

The more reasonable choice for such situations is using an AR SDK - a ready-made module that already performs everything you need. It can be integrated with your app within a couple of days. 

We will show you how to integrate face filters with Banuba Face AR SDK. It is a complementary one, as it doesn't provide video chat features but enhances them. 

This is how the video conferencing API integration is handled.

The first step is the same for all platforms. To get the SDK and the token for a 14-day free trial, just send us a message through a contact form.

Android

This is what you will get from us: 

• Banuba Effect Player (compiled Android library project with .aar extension),
• Android project folder (src) which contains demo apps, located in src/app and src/offscreen respectively. 

You will also get a trial token that should be placed here: src/app/src/main/java/com/banuba/sdk/demo/BanubaClientToken.java. 

Requirements:

• Latest stable Android Studio
• Latest Gradle plugin
• Latest NDK

Then use the guide to build and launch the demo app or the integration instruction to connect the SDK to a project.

iOS

This is what you’ll get in the archive:

• Banuba Effect Player XCFramework (BanubaEffectPlayer.xcframework),
• BanubaSdk Xcode project,
• Effect examples in the effects folder. 

Token should be placed here: src/BanubaSdk/BanubaSdkApp/BanubaSdkApp/BanubaClientToken.swift. 

Requirements:

• iPhone devices 6+
• iOS 11.0+

You can find what you need in the demo app instruction and integration guide.

Web

Requirements:

• The latest release of Banuba SDK Web AR
• Banuba client token
• git client
• Nodejs installed
• Browser with support of WebGL 2.0

The instructions can be found in this document. You can do it in just a few lines of code.

Desktop

Requirements:

• Latest stable xCode on MacOS
• Latest stable Microsoft Visual Studio on Windows

The SDK archive will contain:

• bnb_sdk folder with dynamic libs and include files,
• third folder with third-party libraries,
• resources folder

Then follow the steps in this guide to integrate the SDK/video conferencing API.

Conclusion

Building a video chat with filters isn’t easy. Both from the technical and business point of view. However, it can be made much easier by using an AR SDK. With this guide, you will be able to get started with an AR chat for any platform. And if you need assistance, feel free to send us a message!