Skip to main content
Version: 4.55.1

Fullscreen Segmentation

Segmentation Textures are masks that are updated in real time based on what is seen in the device's camera. Segmentation textures can be applied to a camera's Mask Texture input to show or hide certain areas of the scene. For example, the Portrait Background Segmentation texture creates a masked image of the portrait of the user. This mask texture can then be used as an input to a camera to only show elements behind the user. It's like a green screen without the green.

Add a Segmentation Texture​

To add a new Segmentation Texture, select + -> Segmentation Textures from the Resources panel. Then select from the provided Segmentation Texture types.

Available Segmentation Textures​

  • Portrait Background: Masks out the portrait of the user showing just the background.
  • Portrait Hair: Masks out everything but the user's hair. Allows you to color or apply a texture to the hair.
  • Portrait Shoulder: Masks our everything but the user's shoulders.
  • Portrait Face: Masks out everything but the user's face. This does not include the user's hair.
  • Portrait Head: Masks out everything but the user's head. This does include the user's hair.
  • Portrait Skin: Masks out everything but the user's Skin. This does include the user's hair.
  • Sky: Masks out everything in the scene but the sky. Great for rear camera experiences where you want to put content in the sky.
  • Body: Masks out everything in the scene but the full body of a person. Great for experiences where you want to put something behind the user.
  • Upper Garment: Masks out everything in the scene but garments starting from the top of the user. The mask may extend beyond the upper part of the body. Great for demonstrating clothes try-on.

Custom Segmentation Textures​

While Segmentation Textures can utilizes the built-in Segmentation Texture to mask content shown above, Lens Studio also supports Custom Segmentation where it's segmentation mask is provided through a machine learning model with ML Component.

To learn more about Custom Segmentation with SnapML, please visit the Custom Segmentation Template to learn how to get started.

Customizing the Segmentation Texture​

Segmentation textures can be tuned by selecting them in the Resources panel and tuning its settings in the Inspector panel.

Any segmentation texture can be easily inverted by clicking the Invert Mask checkbox.

Any segmentation texture can have their edges feathered. Change the Feathering slider to tune the feathering sharpness.

Any segmentation texture can have their edges refined by clicking the Refine Edge checkbox. When this is enabled, post-processing algorithms are applied to the segmentation resulting in a higher quality mask. We recommend utilizing Refine Edge for most use cases.

Using the Segmentation Texture​

Now that you have a new Segmentation Texture added to your Resources panel, in most cases, you'll want to use this texture as a mask for a camera. To do this, select your camera in the Objects panel. With the camera selected, in the Inspector panel, set the Mask Texture field to your newly created Segmentation Texture.

Your camera's output will now be masked by the selected Segmentation Texture.

Try adding an object to your scene now, and you can see that the part of the object outside the portrait is masked out.

You can tick the Invert Mask checkbox as mentioned above to get the object to look like it's in front of the user.

Similarly, you can apply a Mask Texture on the Orthographic camera to have your Screen Images show behind the user's portrait.

Finally, you can also add an Opacity Texture to your materials to mask out specific materials. To do this, make sure a Blend Mode which allows transparency is selected (e.g. Normal), and then enable the Opacity Texture checkbox. You can then put your opacity texture in the Texture field under the Opacity Texture section

Segmentation Templates​

There are a couple templates to help you get started on various ways to use the different segmentation textures.

Segmentation Template​

When using segmentation, we recommend checking out the Segmentation Template. The template has a controller that allows you to easily add a background color, image, tiled image, post effect and more segmented by your selected Segmentation Texture.

Body Segmentation​

When using the Body Segmentation texture, we recommend checking out the Body Segmentation template. The template provides various examples of how to use it to create different effects. This template is great for making  a dancing or music video type Lens.

Upper Garment Segmentation Template​

When using Upper Garment Segmentation, we recommend checking out the Upper Garment Segmentation Template. The template demonstrates how you can demonstrate virtual try-on by using Body Tracking to attach your content to the Snapchatter's shirt. In addition, it provides additional materials to add shadows to the garment so that it will look more realistic.

Portrait Skin Segmentation Template​

When using Portrait Skin Segmentation, we recommend checking out the Portrait Skin Segmentation Template. The template demonstrates how you can demonstrate placing any content, such as images, 3d meshes and more to the skin and attach your content to a human.

Was this page helpful?
Yes
No