Hand Segmentation
The Hand Segmentation texture allows you to easily segment your hand and replace it with an image or a custom material.
This guide outlines several examples of using hand segmentation, such as occluding an object as it moves behind the hand, placing an image on the hand, and adding images with a parallax effect.
The Hand Segmentation 2D and Hand Segmentation 3D examples outlined in this guide are available in the Lens Studio Asset Library.
Scene Setup
To add Hand Segmentation texture to your project:
- Click on the
+button in theAsset Browserpanel and selectHand Segmentation Texture.
- In the
Scene Hierarchypanel add newScreen Image. Set the texture of the Image component to the Hand Segmentation texture.
- With the same scene object selected, click on the
Add Componentbutton in theInspectorpanel and selectRectangle Setter. Set the Crop Texture input of Rectangle Setter component with the Hand Segmentation texture.
Examples
Hand Segmentation 2D and Hand Segmentation 3D assets are available in Lens Studio Asset Library. Import asset and then drag included prefab to the Scene Hierarchy panel.
In both examples Hand Segmentation Mask Camera is used to generate a full screen segmentation we used to work with. This camera also contains the Hand Tracking object which we use to get the position and rotation of the hand in the screen space so we can convert that into 3D space for use in the world space Camera.
This means that the camera uses a render target to render the hand segmentation to it. You can find the Render Target in the Asset Browser panel, under the Textures folder, called Mask Camera Render Target.
Make sure the camera that renders the mask texture is set to render before the Orthographic camera. To do so select the Scene asset in Asset Browser and tweak the camera order in the Inspector panel:
To learn more about this set up, check out the Camera and Layers guide to learn more about render targets and camera.
Both examples share same helper scripts.
Attach To Hand Helper
The Attach To Object Tracking script allows you to attach any objects, regardless of whether it’s 2D, 3D, Text and more to your hand.
To make sure objects appear on the hand, just place any content as a child of the Attach To Hand In 3D in the Scene Hierarchy panel.
To customize, select the Attach To Hand In 3D object in the Scene Hierarchy panel and adjust its setting in the Inspector panel.
On Tracking Lost: allows you to either keep or hide all the child objects as soon as hand tracking is lost.
Apply Rotation: if enabled applies rotation data from the hand tracking object to all the child objects.
Behavior Trigger: if enabled - allows to send custom triggers to the behavior script as soon as hand tracking starts tracking or lost tracking.
To use this trigger and make a custom action without coding, you need to import the behavior script into your project. To learn more about how to import and use behavior script please refer to the Behavior guide.
Advanced: enables additional settings.
Perspective Camera: the perspective camera in scene (Make sure it's NOT an orthographic camera).
Object Tracking: reference to the Object Tracking component. This example uses Hand Tracking 2D but this script can be used for other 2D tracking.
Simple Hand Texture
This example uses custom shader to apply a texture to hand segmentation. To see how it works, unpack the imported asset package and select the segmentation_simple shader. Double click on the asset to open Graph editor panel.
Parallax Hand Texture
Parallax Hand Texture example uses the position data from the Hand Object Tracking to create a parallax effect look applied to the hand. This example uses a helper script called ScreenTransformToMaterialProperty to get a position data from Hand Object Tracking and then convert that data into a screen point.
After converting the data to screen point, this will pass the data to a material pass which is defined by the user. In this case it passes the data to the center property.
To see how the custom shader works on this example unpack the imported package for editing and double click on the segmentation_parallax_material shader to open Graph editor panel.
3D Object Example
The 3D Object example included in Hand Segmentation 3D asset shows how you can use an occluder in 3D space so you can have an object to move around a hand and as soon as it goes behind the hand it will get occluded.
Hand Occluder has a material which contains Render Target from the mask camera. You can use this object to occlude any objects that go behind the object. You can use the gizmo provided in this example to see where the object gets occluded in the Scene Hierarchy panel.
The gizmo is just an image of the hand and can be used as a reference in the Scene Hierarchy panel. It is not actually displayed in the scene.
The Airplane object uses a Tween script to rotate around the hand.
Effects (Helper Scripts)
This example comes with some helper scripts to help create a more immersive experience through interactivity.
You can find these effects in the Scene Hierarchy panel, under the Attach To Hand In 3D [OBJECTS_HERE] -> 3D Object Example -> Airplane -> Effects Controllers.
Let’s take a look at all the effects as well as helper scripts listed below.
Hand Gesture Controller
Hand Gesture Controller uses hand gestures to create unique effects. To learn more about how the Hand Gesture Controller is working please refer to Hand Gestures guide.
World Particle Controller
Although this approach to particles is pretty old it still can be used in Lenses. Please refer to VFX guides to more particle capabilities added since this guide was released.
With the help of the world particles we can create a trail of smoke in the scene.
The way world particles work is that we used the same particle system which comes with the Lens Studio. With the WorldParticleController helper script you can assign your particles effect and then this script creates copies of the same particle and places it in the scene.
This helper script exposes some APIs so you can update the particles in run time. You can use Behavior to call these APIs with no coding at all.
Take a look at the following examples below to see how they work:
Play or Stop Particles
To start or stop the particle you can call script API using behavior script and call the function to play or stop. Select the Play or Stop Particles scene object and check out Behavior script setup.
Update Particles Color on Gesture
To change the color of the particles you can use a behavior script and call the updateParticlePasses function. When you call this function make sure to pass a pass name as well as a value related for the pass you want to change. In our example, we modify the colorRampTexture of the particle material.
Select the Update Particles Color on Gesture scene object to see the Behavior script setup:
Gesture Speed Control Effects
Another helper allows to use the rotation of the hand and apply an effect based on the rotation value. In this case we use rotation angle to change the speed of the plane, as well as changing the hand color as value increases.
This effect starts working as soon as the user does the One Finger gesture and then can change the speed.
For example, in the Objects Rotation to Material Parameter example, we get data from the Hand Tracking Center Object and then we convert that into RGBA data. With use of this script we can change the color on the user's hand to turn to white on the lowest value and turn red on the max value.
There are a couple more examples using Object Rotation which you can find under the Gesture Speed Control Effects object in the Scene Hierarchy panel. Feel free to play around with these as you did above.
Previewing Your Lens
You’re now ready to preview your Lens! To preview your Lens in Snapchat, follow the Pairing to Snapchat guide.