Paint To Erase
With the ML Eraser Custom Component, Lens Developers can build unique AR experiences powered SnapML to alter and remove real world content in real-time. The Paint To Erase Template allows users to paint freely over their devices screen, paint within a mask, or to capture a mask in order to erase different regions from the screen.
ML eraser offers a real-time GAN-based inpainting which improves the realism of the results, in comparison to classic approaches (such as the current shader in LS). This GAN-based approach can learn the texture and complete different regions in a convincing manner, while traditional approaches are smooth and blurry.
ML Eraser Component
The ML Eraser Component contains various inputs that can be modified manually or via a script. It can be introduced to a project by dragging it from the Asset Browser panel onto a new Scene Object.
- Input Mask: The mask texture to be inpainted.
- Eraser Mix: Controls the opacity of the inpainting while blending with the Input Mask.
- Output Material: The material created from the Custom Component.
- Use fallback: Allows for a fallback option that uses a shader instead of the ML inpainting neural network. In this case we use a naive solution which is more blurry.
Template Walkthrough
The project on the top level has an eraser camera containing the ML Eraser Custom Component, complete with a swappable mask responsible for inpainting areas of the scene.
There are two options you can choose to erase certain things on the camera: The paint tool, and the dissolve effect, that will swap out the ML Eraser mask to achieve their own effects. Finally, a hint camera displays a hint when the project is opened that gradually fades out.
Paint Tool
The paint tool is enabled by default, allowing you to erase parts of the screen in different ways. Within this effect, there are three options that can be toggled between by pressing the button in the UI.
Paint to Erase
The first option, enabled by default, enables you to tap and drag on the screen to erase the spot you drew on. This works by disabling the clear color option on the paint brush camera’s render target. This means every frame will render on top of the previous frame.
Paint with Body Segmentation Mask
This option blends the drawing with a Body Segmentation mask, so that you can only paint within the mask.
Within the Brush Controller script, there is logic to show a hashed outline masked by the Body Segmentation that will be displayed in the Segmentation Blend Background Screen Image. This is located within the UI Camera while you draw to highlight the masked painting region.
Tap to Erase Body
This option copies a Body Segmentation texture from the Segmentation Snap render target to use as a mask for the ML Eraser effect when you tap on the screen.
The Snap Segmentation Effect Post Effect is enabled while a user is holding the finger down, highlighting the masked area to be erased at Touch Ended.
Each of these effects can be manually toggled in the top level Brush Component along with the brush size.
Try clicking Show Paint Debug to see what the mask looks like when you draw and see how changing the brush size changes the blending effect.
Dissolve Effect
The dissolve effect uses Body Segmentation to erase the user. Since it's using Body Segmentation, the effect will persist and can control the alpha blending through the ML Eraser component Eraser Mix.
This effect combines a custom material that uses noise to dissolve the segmented image of the person, and a Custom VFX that spawns on the segmented image of the person, which then floats away, completing the effect. To activate this effect, disable the Paint Brush example and enable the Dissolve.
This effect uses Body Segmentation mask as the ML Eraser’s mask. The Dissolve Controller script swaps the mask automatically when this effect is enabled.
The top level controller script has various options that can be toggled.
Try to change and alter the shape, size and softness to see how it changes the effect.
This is all activated with a Behavior script inside the example. To change the type of event that triggers the effect, or the delay, you can edit the Behavior script trigger.
The restart button will reset the tween animation responsible for making the person in the scene invisible, as well and hides the particle VFX in the Dissolve Controller script.
Previewing Your Lens
You’re now ready to preview your Lens! To preview your Lens in Snapchat, follow the Pairing to Snapchat guide.
What's Next?
Now that you have learned about how to create different types of disappearing effects with ML Eraser, take a look at some of the other Templates that use ML Eraser.