3D Body and Hand Tracking
You are able to track joints on the body, as well as the hands in 3D space with the Object Tracking 3D component. However, depending on the parts you want to track, you will use different Tracking Assets.
You can add the different tracking by selecting either the 3D Body or 3D Hand tracking Component under the Object Tracking 3D category in Objects panel > +. Lens Studio will then add the object containing the unique Tracking Asset.
3D Body Tracking
The 3D Body Tracking component allows you to attach 2D or 3D contents to a set of 3D skeletons found in the camera. It currently supports the tracking of the joints of one or more bodies.
Previewing Your Model for Use
You can preview rigged skeleton of any 3d model or any tracking component under the Object Tracking 3D category.
If this rigging visualization is not enabled by default, please go to the top right corner of the Scene view in Lens Studio and tap on the tools icon, then select Joint Visualization.
Note that different types of rigs will have different colors. By default, user uploaded custom rigging will have the turquoise color and bones provided by 3D tracking component will have a grey color as shown in the image above.
You may tap on an individual joint and drag it around to adjust shape of your rigged model.
Removing Existing Animation Component
If your 3D model already has animation component, please make sure to disable the animation component, otherwise your 3D model might not move with tracking component because its existing 3D animation is overwriting its movement.
Adding 3D Body Tracking
3D Body Tracking allows you to track the joints of one or more bodies in 3D. Unlike 2D Body tracking, it will track the rotation of the joints even as the user turns around.
To add 3D Body Tracking to your project, in the Objects panel, select + -> Object Tracking 3D Full Body Tracking
A Body Tracking Asset will be generated in the Resources panel.
If you're looking to create 3D Body Tracking Lenses, we recommend taking a look at the 3D Body Tracking template.
Choosing Tracking Mode
There are several ways your content can be attached to the tracker.
- Proportions and Pose Applies rotation correction from the root of the model to the leaf bones. If bones further down the hierarchy are missing, we still apply corrections up to that point. Additionally, it will scale the your object such that it matches the proportion of the Snapchatter.
- Pose Only - The same as
Proportions and Pose, but does not apply scaling. This is useful when you don't want your 3D model to be distorted based on the Snapchatter's body. - Attachment - Moves the attached scene object to the world position of the joint. An object with 0 rotation will be aligned to the axis of the attached joint. This mode is useful when you are attaching separate 3D models to the joints.
If you started your project prior to 4.34, and your project did not use a full character rig, you will likely need to switch your tracking mode to Attachment.
Adding 3D Models to the Tracking Component
There is a list of Attachment Points on each Object Tracking 3D component. Each of these points represent a specific joint on the body part. Any object linked in this slot will be attached to the specific part of the body.
Using Match Hierarchy
No visual element will show up in the scene unless we apply a custom 3D model to the Object Tracking 3D component. The most convenient way to do this is by using the Match Hierarchy option.
Note that Match Hierarchy would work only if the 3D Model you provided has the same rigged bone structure as the default models we provided. Feel free to check out the 3D Body Tracking template or the Stick Figure 3D Model on the Asset Library as a reference.
Hand Tracking in Body Tracking
The 3d Body Tracking asset contains an option to enable hand tracking. You can enable this option by ticking the Track Hand option in the Inspector panel, with the asset selected. This option is useful when you want to track both the body and the hand, but should be disabled if you object doesn't have hand dedicated joints.
Take a look a the 3D Body Template guide to see the hierarchy used in 3D Body Tracking, as well as examples of it in action.
3D Hand Tracking (Standalone)
In addition to hand tracking provided by the body tracking, you can also add a dedicated 3D Hand Tracking asset which would work more precisely.
The Hand Tracking hierarchy in the Body Tracking asset uses a more standard hierarchy used in 3D animation. In contrast, the standalone 3D Hand Tracking contains 5 more extra bones: wrist-to-thumb, wrist-to-index, wrist-to-mid, wrist-to-ring, wrist-to-pinky. In addition, they use a 0-based naming convention for the remaining 4 bones of each finger, e.g., thumb-0, thumb-1, thumb-2, thumb-3.
3D Hand Tracking allows you to track the joints of one or more hands in 3D. As before, to add 3D Hand Tracking to your project, in the Objects panel, select + -> Object Tracking 3D -> Hand Tracking
A Hand Tracking Asset will be generated in the Resources panel. You can select the Hand Type to specify which hand this asset will be tracking.
By default, Two hands tracking is enabled. If you only need to track one hand a time, disable Two hands tracking for device optimization.
Additional joints are provided between the wrist and the different fingers to allow scaling and rotation of "knuckles" joints independently, resulting in a more flexible 3D model that are more accurate to the tracked hand. Without these extra joints all 5 knuckles joints would share the same parent joint (wrist) and thus would share the same parent coordinate system; In other words, they could only be scaled and rotated all at once using single shared scaling and rotation parameters.
Adding 3D Models to the Tracking Component
Similar to the 3D Body Tracking component, there is a list of Attachment Points on each 3D Hand Tracking component. Each of these points represent a specific joint on the body part. Any object linked in this slot will be attached to that specific part of the hand.
Only one hand can be tracked in scene at any given time. If there are multiple hands in camera, the project will usually default to the first hand being observed.
Using Match Hierarchy
Similar to the 3D Body Tracking component, Match Hierarchy on the 3D Hand Tracking component would work if the 3D Model you provided has the same rigged bone structure as the default models we provided. Please find the structure of the reference for 3D Hand model in the 3D Hand Reference asset in the Asset Library or in the 3D Hand Tracking template!
Make Sure Models Are Not Using Animations
Sometimes our models have built-in animations when we import them to Lens Studio. If so, the built-in animation might overwrite the movement and cause our 3D models to not follow tracking position, even if they are assigned in Attachment Points.
To prevent native animation from taking over our 3D model position. First we have to disable the Animation Mixer component on the object.
Then, we have to also disable all Skeleton components on all the joints.
After we've done these 2 steps, we can make sure that there is no existing animation overwriting our model, we'll see our model being controlled by the tracking component instead.