Platform and Tools To Create Face-based AR Experiences

Face-based AR experiences: where to use

Face-based augmented reality is enabled with face recognition and tracking technology that detects a face and overlays virtual content on it. Most commonly, the technology is called 'Face AR' meaning the user augmentation happens in real-time and can be experienced via a mobile or web camera.  It can be used to:

• Animate day-to-day communication and expression with personalized avatars and emojis, AR masks and face filters.
• Let consumers virtually try on makeup, glasses, jewelry at home to see how products look
• Develop creative advertising and marketing campaigns enabling consumers to create and share branded content.
• Edit, smarten up and express users’ identity in selfies and stories.
• Build cross-platform AR mobile apps and Unity games with Snapchat-like face filters.
• Empower social networking apps with AR helping users to generate content easily.

Face AR turns a camera into a lifestyle companion bringing warmth to the digital landscape and making the impersonal personal – through the human face.

How Face Filter SDK empowers developers

Face Filter SDK provides developers with three core components for building face-based AR experiences:

• Augmented Reality SDK
• Effect Constructor & Viewer
• Face Filter Catalogue

Augmented reality SDK includes technologies and libraries enabling a certain type of face augmentation experience. Each of them is a cross-platform SDK with face recognition, rendering engine and neural networks at its core to allow adding Face AR features to any app (for Android or iOS) and website or build new apps from scratch.

Effect Constructor and Viewer tools allow developers and designers to configure and preview filters before uploading to production.

Catalogue facilitates and speeds up the process of creating AR content. Developers can buy ready filters or customize them for their audience. 

Face-based AR experiences immerse users more deeply into developers’ apps. They increase the engagement, freedom and enjoyment of the camera experience allowing users to create interesting, fun content in a breeze.

How Developers Can Create AR Filters For Web and Mobile

With the release of Facebook Spark AR and Snapchat lens Studio, AR face filter creation has become accessible to the masses, and it has even fueled the emergence of a new profession - AR creators. Still, developers can’t use their masks outside Snapchat or Facebook and have little control over their content.

These tools are a perfect choice for individual developers, designers, or social media influencers. Face AR platform is designed for development teams and brands who want to launch their own face tracking solution.

Unlike Snapchat and SparkAR, with Face AR platform you can integrate AR face recognition and face filters in your apps, games or websites. To get started creating your face-based AR experiences you need to:

1. Install Face AR components

Get in touch with us to get links to downloading SDK, Effect Constructor and Viewer components. Before purchasing the license you have a free trial period to validate our SDK performance. 

2. Explore effect samples

Our Face AR SDK is delivered with examples featuring SDK applications for each platform and a variety of use cases giving developers a comprehensive overview of how to run and use it. View the full list of SDK features and try the functionality in demo apps for iOS and Android. 

3. Define your filter concept

Think of your USP (unique selling points) and app concept. Why will people go to your face filter app over the others? Think about how your filters will differ and bring value to users. If you nail down your killer filter concept, your app will go viral, and you’ll succeed.

4. Prepare materials and textures

You may use any 3D graphical editor like Maya to design the visual concept of your effect and import them in .fbx format.

5. Upload source files into Effect Constructor

Our Effect Constructor converts files into the supported formats so that you further be able to configure and run your effect.

6. Preview your effect with a Viewer

The Viewer component allows running the constructed effects in real-time on desktop before uploading them to the app. To preview effects, you just need to drag'n'drop the constructed effect folder into the Viewer window.

What Face-based AR Experiences You Can Create

As the purpose and the audience of your app vary, so our Face AR platform allows for creating different types of face-based AR experiences. These possibilities make our Face AR a great fit for a broad range of apps and use cases, from live chats and video streaming services, social networking and dating apps to professional makeup and skincare brands.

3D Animation

AR face filters with 3D animation is one of the most popular types of face-based AR experience in entertainment apps. You can animate 3D geometry making objects move on a given trajectory. The animation can play continuously in a loop, according to a specific scenario or enabled with triggers.

Background change

You can separate a user on the foreground from the background in a video to remove the background, replace it with another graphical image or add a video file. Above entertainment, the background subtraction technology that stands behind these filters find its usage in video chats and video conferencing solutions. It increases privacy and allows for safe and comfortable communication.

Face Touch Up

You can create beautification or face touch up effects with skin smoothing, eyes and teeth whitening, "expressive eyes" effect, color correction and skin tone adjustment (darker or lighter). You can use it as a standalone filter in live streaming, social networking, dating, video chats, or other face filter apps to enhance the user camera experience or as part of face filters adding animation, virtual accessories or entertainment masks.

Virtual Makeover

Makeup try on filters allow users to test cosmetics products virtually via a mobile camera. Unlike face beauty filters, virtual makeup try ons feature realistic representation of AR content (lipstick, eye shadows, hair color, etc.) without modifying the use face for beauty purposes but rather helping people find best-fit makeup products. 

Realistic Try On: Glasses, Jewelry, Hats

You can design AR models that behave like the real objects in the flow of real-world light, e.g. support gravity or mirror the light with the camera rotates and user head tilts. Physics property allows for realistic 3D model behaviour which is essential for virtual try on apps. 

Multiple Faces

With multiple face detection, you can apply face filters to several users where one effect is applied to each face recognized in the camera, or different effects are applied to several faces simultaneously. AR filters for multiple faces are extremely popular in magic-mirror photobooth solutions and kids apps to add fun and engagement of enjoying the AR experience together with friends and family.

Triggers

You can enable AR scenarios in face filters with user facial expressions. The user can interact with effects or call them opening mouth, smiling, raising eyebrows or frowning.

Game Face Filters

AR filters with triggers are the best fit for quick Face AR games or interactive face filters to increase engagement. Users can control AR objects with facial expressions, head movements or touches.

Morphing

Morphing allows for modifying the size of the face and its facial features by moving the vertices of the 3D model of the face. You can slim down or enlarge the face, change the shape of the nose, eyes, lips, chin, or produce any other deformations.

Instagram-like LUT Effects

You can create custom AR filters enabling real-time or offline color correction of recorded images.

Live Emojis and Avatars

You can animate 3D emojis, avatars or full-face AR masks that support and copy user facial expressions in real-time.

What It Takes To Create Quality Face-Based AR Experience

Good-looking and nice performing face-based AR experiences are the result of multiple efforts from the team and technology implementation. So, what does it take to create professional face filters that keep the audience engaged and inspire social shares?

1.  Robust technology. Fast and stable AR face recognition optimized for multiple platforms and broad device support is crucial. No matter how creative your AR filter, the users won’t be able to experience it to the fullest unless it functions well on their devices.
2.  Powerful effect renderer. Gone are the days of primitive 2D face stickers. Now, the AR industry requires creative Face AR experiences similar or even bigger than Snapchat, and our proprietary Graphical Renderer engine allows implementing AR face filters just like that.
3. 3D art team. Unique and interesting visual concepts and their graphical realization are the result of experienced Art Team who have been creating face AR experiences for 4 years and are constantly testing and discovering what different audiences need.
4.  Development. Complex effects may require coding part, therefore, knowledge of JS (or any other programming language at the Jr-Mid level) to write a script to control the effect are nice to have.
5.  Optimization and testing. A high-quality AR face filter needs to look visually appealing, run smoothly on a wide range of devices and take as little memory space as possible. AR mask optimization and testing steps allow achieving better performance and higher user satisfaction from your app.
6.  Regular updates. Dynamic and regularly updated AR content in your app provides better user retention. You always need to experiment with bringing new AR experiences to your users.

Summing up

Businesses and developers can capitalize on Face AR experiences to engage with their audience with AR face filters, advertise products, build brand awareness and drive consumers and prospects organically. Creating quality face-based AR experiences can be a science but we're always ready to share our knowledge and experience helping brands to realize their creative thinking and implement it with our Face AR platform. Get in touch if you want to discuss your idea!