How To Optimize 3D Face Filters

NOTE: The guidelines below refer to working with Banuba Effect Constructor which is our proprietary AR effect construction engine. It allows designers to make custom AR face filters and test them in real-time before uploading to  mobile apps or websites built with our Face Filter SDK. 

What impacts 3D face filter performance

Before we look at how to optimize your 3D face filter, let’s see which components impact its performance.

Textures

Textures are assigned to a 3D object to add information about its color, brightness, and roughness. The most common types of PBR textures include:

• Base color to define the overall color of the material.

• Metallic to define how “metal-like” the surface is.

• Roughness to define the sharpness of the reflection.

• Ambient occlusion to define the shadowing properties.

• Normal to add bumps and dents and create detailed shadows and highlights of the surface.

• Emissive to define areas of the textured material that will illuminate.

When designing your 3D face filter, you don’t need to use all of the textures. To display PBR materials, you need to use a set of textures which transfer the surface of the model in the rendering engine. The most optimal set includes 4 textures:

These textures are generally used to create most types of AR face filters and ensure a realistic representation. However, the final number of textures in your effect may differ depending on its type and task. For example, for non-PBR materials, you can use just one BaseColor texture, while in other AR filters you may need an additional Emissive texture.

Polygons

A polygon mesh is a collection of vertices and polygons that defines the shape or surface of 3D objects. The fewer number of polygons provide the "lighter" effect.

However, polygons have less impact on the effect performance compared to texture resolution. You can also increase polygons and reduce the size of textures to achieve greater effect productivity.

Making 3D face filter: Tech Guidelines

When you make your 3D face filter, you need to consider its performance on different types of devices — both powerful and low-end ones. The "heavy" effects may run perfectly fine on iPhone X, but lag on low-end Androids.

To avoid this, sometimes, you can even make two versions of one AR effect with different sizes of textures and properties for different devices.

Generally, 30 FPS performance on iPhone 5s level devices provides a quality user experience. To achieve it, we recommend adhering to the following technical guidelines:

• Keep the total resolution for textures in one scene not more than 1024x1024px for each type of texture. For example, you can have four sets of textures for PBR where each texture is 512x512px in size.

• Use the video resolution not more than 512x512px. If other filter parameters are well-optimized and the performance allows it, you can go with more. 

• For beauty filter or beautification layer, we recommend using textures of the following sizes: softlight_layer (256x256px), eye_highlite (768x768px), normal_layer (512x512px)

• The total number of triangles in one scene should not be over 50,000 and your 3d object should be divided into several files. You may have 100,000 triangles and AR filter will work fine if textures are small. The number of vertices in one FBX file cannot exceed 65536, however, you can bypass this restriction by loading a scene with several FBX files.

How to optimize 3D face filters

1. Reduce and optimize filter textures

To reduce the memory load, you need to use the texture resolution which provides the minimum acceptable quality level. It needs to be as small as possible yet still, allow for a good-looking effect.

Additionally, textures within one 3d model can have different resolutions. For example, Roughness or Metallic textures have less impact on the final rendering, so you can make their resolution smaller.

To get your effect load faster, particularly with large textures, it is important to have textures in KTX format. This is the native format for the GPU, and therefore it loads into memory quicker without conversion.

To get textures in KTX format, place them in the SRC folder when assembling your AR filter with the Effect Constructor. It automatically converts texture files into KTX format so your final effect will load faster.

Your source files must be in PNG format and with a specific suffix.

2. Optimize beautification 

The Beautification layer may come as a standalone filter or as part of your AR try-on or entertainment effect. It significantly improves the overall face filter experience, retouches user face imperfection and ensures a professional and harmonic AR face filter. 

Read more: How To Create Face Beauty Filter

To reduce the memory load, you can also optimize your beauty filter. If your softlight_layer or normal_layer beauty textures are symmetrical, you can cut off one part.

After that, in the effect folder find and edit the retouch.frag file

1- Find and uncomment the following line in your effect, in our example, it's *145 (in your effect, еру line number may differ)

   vec2 uvh = vec2(abs(2.0 * (var_uv_bg_uv.x - 0.5)),var_uv_bg_uv.y);

2 - If you use the right part of softlight_layer texture

-find the SOFT_LIGHT_LAYER block, line *148 (in your effect, the line number may differ) 

-replace the var_uv_bg_uv.xy variable with the uvh variable uncommented above

-the new line will look like this:

    res.xyz = blendSoftLight( res.xyz, texture( tex_softLight, uvh ).xyz );

If you use the right part of normal_layer texture

- find the NORMAL_LAYER block, line *152 (in your version of the effect, the line number may differ)

- replace the var_uv_bg_uv.xy variable with the uvh variable uncommented above

- the new line will look like this:

    vec4 makeup2 = texture( tex_normalMakeup, var_uv_bg_uv.xy );

3. Optimize and organize polygons

The fewer polygons you use, the more productive your effect is. However, polygons have less impact on the effect performance than texture resolution. You can also achieve greater productivity by increasing the number of polygons and reducing the size of textures.

The number of vertices in one FBX file cannot exceed 65536, however, you can bypass this restriction by loading a scene with several FBX files.

Also, you can optimize the geometry so that the front-facing surfaces located closer to the camera had a denser topology, and you can completely omit the invisible polygons.
This way, you save on the number of polygons and optimize resources without compromising on the filter experience since the user always sees the front part.

Polygons with a transparent shader affect the AR filter performance, especially if they are arranged in layers following one by another, for example, as in hairstyles or virtual wigs.

The real-time engine draws polygons based on the position of the shader in the draw order starting from top to bottom.s with a transparent shader occupied the maximum screen area. You'll see the productivity decreases as the size of the polygon area increases. That's why it's important to understand the principle of polygon rendering and draw order when working with transparency.

Ihe drawing starts with a polygon which has the lowest sequence number and goes to a polygon with the higher number.

A polygon with a higher serial number or lower position in DrawOrder overlaps one with a lower serial number or a higher position in DrawOrder, even if it's placed farther from the camera. The transparency of this higher polygon will be drawn on top and erase the preceding polygons located closer to the camera.

You can organize polygons as follows:

• In any 3D editor, copy an object to its polygons with the maximum polygon numbers assigned. You can copy objects in any order and then delete the old objects to get the required polygon order. 

• In any 3D editor, when you combine polygons into one object, the numbering will correspond to the order of the attached objects. 

• In Houdini modelling software, you can use Sort SOP node for sorting primitives. 

• In Blender software, you can sort primitives by Z-axis using Miscellaneous Editing Tools

4. Optimize skin files

The real-time engine supports a maximum of 4 influences per one point.

The number of Bones in one FBX file cannot be more than 256. You can bypass this restriction by loading the scene with several FBX files.

5. Optimize video files

For animated textures, you can use an MP4 video file instead of a sequence of images. It allows getting a significantly higher frame rate with a relatively small size of the file. Our filters achieve 30 FPS for 3d animation and video textures. You can read more about Animated texture and billboard and converting to this format.

The rendering engine limits the number of video files allowed for simultaneous use in one effect. You can use only one video file for textures, another one for the foreground and one more for the background. If your AR effect needs more video textures above that, you can bypass this restriction by composing all video textures in one file.

How to test your 3D face filter

Here’s a short checklist for AR face filter testing, however, each filter requires a unique testing approach with more or fewer actions needed to estimate the quality of its performance.

Performance iOS, Android

• Real-time AR face filter — at least 30 FPS on iPhone 5s, 6, 6plus | Samsung s8/Huawei P10/Mi6 and higher
• AR mask and video recording — at least 30 FPS on iPhone 5s, 6, 6plus | Samsung s8/Huawei P10/Mi6 and higher

User experience

• You can take a photo or/and video with the effect
• The effect works stably on the front and rear camera
• Effects with a distinctive feature which involves user action e.g. trigger, tap, swipe or voice changer have a hint about it (first couple of seconds).
• The effect works as required with face in the camera, no face, and disappearing face.
• Effects with analytics perform correctly (the name of the effect matches, the functional properties match)
• Effects are stable and perform correctly at different angles (90 degrees rotation, 180 degrees rotation (upside down))
• Check how the masks stretch on the face (wide open mouth, eyes, grimacing) to make sure that no holes in the mask appear and no element overlap one another

Device compatibility

• Check effects on Xsmax for crashes (some effects fall on Xsmax)
• Check effects on Xsmax for incomplete filling of the screen (sometimes the effect does not fill the entire screen with high resolution)