Facial Features Segmentation

To make it easier for you to mix-and-match our different examples, this template has been converted to an importable asset in the Asset Library with the name Facial Features Segmentation.

The Facial Features Segmentation Template demonstrates how to use the Face Occlusion Custom Component as well as the Face Segmentation Custom Component. Both of these Custom Components allow the user to occlude AR face effects when a user places an object in front of their face.

In the Template, the Scene Hierarchy panel has multiple examples that demonstrate how to use both the Face Segmentation and Face Occlusion Custom Components. Additionally, there is a debug view that shows what the output of each Custom Components.

Understanding the Difference Between Face Occlusion and Face Segmentation Custom Components

At first, the Face Occlusion and Face Segmentation Custom Components may appear to do the same thing, but there are some key nuances that differentiate the two Custom Components.

The primary difference between the two Custom Components is that the Face Occlusion Custom Component only has one output, while the Face Segmentation Custom Component has seven distinct outputs. The final output of the Face Occlusion Custom Component is a black and white image, where the white represents the area of the camera that contains a user's face, and the black is everything else. Use the Face Occlusion Custom Component when you are making a face effect that applies to the user's entire face.

Conversely, the Face Segmentation Custom Component outputs seven different Image Textures, each corresponding to a different element of a user's face:

• Beard
• Eyebrows
• Ears
• Eyes
• Face
• Lips
• Background

The Face Segmentation Custom Component has checkboxes for each of these face elements that you can turn on or off depending on the kind of effect you are looking to achieve. Use the Face Segmentation Custom Component when you are making a face effect that revolves around a specific part of the user's face.

How to Set Up the Face Occlusion and Face Segmentation Custom Components

This section explains how the Face Segmentation template is set up to work with either the Face Occlusion and Face Segmentation Custom Components. Each example uses two separate Cameras, one Perspective Camera for rendering face effects, and an Orthographic Camera used to render the output of the Face Occlusion or Face Segmentation Custom Components.

Some adjustments to the default Cameras are necessary in order to have these two Cameras communicate with each other. Make sure the "Camera Type" with the Face Occlusion/Segmentation Custom Component is set to Orthographic. Additionally, make sure that the child of the Orthographic Camera has a Screen Transform and a Screen Region component attached to it. Finally, make sure that the only child of the Screen Region Scene Object is the Face Occlusion or Face Parsing Custom Component, and that it also has a Screen Transform component attached to it. Lastly, this camera needs a custom Render Target with a few extra modifications. Select the custom Render Target in the Asset Browser panel and set the Clear Color parameter to Color, and the Color parameter is set to an RGB value of 0,0,0,

For the Camera that will render the face effects, all that is necessary is to fill in the "Mask Texture" field with the custom Render Target that the Orthographic Camera is rendering to.

Creating a Custom Face Effect with the Face Occlusion Custom Component

This section guides you through creating an Animal Nose Lens that works with the Face Occlusion Custom Component.

To begin, download the example assets used in this example:

1. Click on the Asset Library button at the top of Lens Studio. The Asset Library popup should appear. In the top left corner of the Asset Library is a search box.
2. Click on the search box and type in Face Attachments. You should see an asset titled Face Attachments Variety Pack.
3. Hover your mouse over the asset and click Import.

4. Return to the search bar in the Asset Library and enter Unit Plane.

5. Import the asset titled Unit Plane.

Ensure that the Examples Scene Object is disabled in the Scene Hierarchy panel. If needed, disable them by clicking on the filled-in checkboxes on the right-hand side of the Scene Hierarchy panel.

Next, set up your project structure:

1. Click on the + button at the top-left of the Scene Hierarchy panel and select Scene Object. A new SceneObject should be added to your scene.
2. Click and drag this new SceneObject so that it is a child of the Camera With Face Mask object.
3. Rename this object to My Animal Nose.
4. Click on your My Animal Nose and in the Inspector Panel click on the Add Component button.
5. Type in and select Head Binding.
6. In the Scene Hierarchy panel, right-click on My Animal Nose and select Create Scene Object. There should now be a new SceneObject that is a child of My Animal Nose.
7. Rename this new SceneObject to Mesh.

Your Lens Studio project should now look like this:

8. With the Mesh SceneObject selected, go to the Asset Browser panel type Unit Plane into the search bar.
9. Click and drag the mesh object titled Unit Plane into the Mesh dialog box in the Inspector Panel. You should see a small mesh appear in the Preview Panel.

As you can see, the Unit Plane is too small. Feel free to adjust the Transform of the Mesh SceneObject and use these values:

POSITION: 0.0, 6.0, 10.0;
ROTATION: 0, 0, 0;
SCALE: 5.0, 5.0, 5.0;

Finally, change the Mesh’s material to add an animal nose.

1. In the Asset Browser panel, click the + button in the top left and add a new Unlit Material.

2. Select the new Unlit Material in the Asset Browser panel.
3. Set Blend Mode to Normal.
4. Set the Texture to any asset within the Nose folder in the Face Attachment Resources asset downloaded earlier.

For this guide, we will use the nose_pig image asset.

Finally, apply this Unlit material to our Mesh:

5. Click on Mesh in the Scene Hierarchy panel and click and drag the Unlit Material into the Material 1 dialog box within the Inspector Panel.

You should now see the animal nose appear in your Lens.

Feel free to experiment and try replacing the Face Occlusion Custom Component with the Face Segmentation Custom Component.