ML Component Overview

Although neural networks have been already widely presented in Lens Studio - in 3D face tracking, segmentation, object tracking, SnapML allows users to create their own Lens features based on neural networks (“models“) they, or third parties, have trained. The neural networks, or ML models, are used in Lens Studio via the ML Component.

ML Model Import

Lens Studio supports importing several common ML model formats:

• Open Neural Network Exchange (.onnx).
• TensorFlow Lite model (.tflite).

These files can be imported into Lens Studio project by:

• Clicking the + Button in the Objects Panel, selecting ML Component, and selecting the model file in the pop-up window.
• Or by dragging and dropping model files into the Resources Panel.

You can tune your import settings in the Model Import Dialog pop up.

Inputs are presented on the left. For each input of the model you can:

• See information about it’s name and shape.
• Configure input transform by selecting a combination of settings (See Transformer settings in the Inputs).
• Set channel scale and bias.
  • If the number of channels is less than or equal to 4, scale and bias can specified for each of them separately.
  • Otherwise, scale and bias can be modified for all channels at once.

You can use this Python code to calculate scale and bias from standard deviation and mean:

std = [0.229, 0.224, 0.225]
mean = [0.485, 0.456, 0.406]
# n = (i - mean)/std
scale = [1.0 / 255 /s for s in std]
print(scale)
bias = [-m / s for m, s in zip(mean, std)]
print(bias)

Outputs are presented on the right. For each output of the model you can:

• See it’s name and shape
• Specify an input that will provide it’s transform source (Output transform is reverse to the input transform).
• Set channel scale and bias.
  • If the number of channels is less than or equal to 4, scale and bias can specified for each of them separately.
  • Otherwise, scale and bias can be modified for all channels at once.

Once imported, Lens Studio will refer to the ML model as a ML Asset. Specified settings will be the default settings for this ML Asset for ML Component initialization.

You can specify different input and output transform settings (but not scale and bias), as well as their shapes, later in the ML Component UI.

Since model training is an iterative process and you might want to update the asset. To do so, you can Right-click on the model asset in Resources Panel and select update from source or relink to new source.

Though you should make sure your model is as small as possible to improve the Lens experience, you can have up to 10MB of SnapML Assets.

Compatibility Table

SnapML will try to optimize running your model across many different platforms. However, different devices may have different hardware acceleration available for ML. The Compatibility Table allows you to check whether every layer your model uses is supported.

You can check out the compatibility of layers used in your neural network model by pressing on the Compatibility Table button in the import dialog or later in the Inspector panel when selecting an MLAsset in the Resources panel.

The button is highlighted red and displays a warning message if there are any unsupported layers.

If it is green - means all layers are supported.

The table will show you a list of layers used in the ML Asset and provide information about their compatibility on CPU, iOS GPU, Android GPU and iOS Accelerator (see MachineLearning.InferenceMode)

You can find out more information about supported layers in the Compatibility section.

Compression

You can select the Compress Model Checkbox at the bottom of the Import Dialog. It will reduce model size up to half using K-Means quantization with minimum degradation of accuracy. Compressing might take some time depending on the model size.

ML Component

Once you’ve imported your model, you can use it in your Lens by using the ML Component.

This section covers how to set up a Ml Component in a new project. You can also follow these steps to add ML Component to an existing project.

Creating the ML Component

You can create ML Component by:

• Clicking on the + in the Objects Panel, selecting ML Component, then selecting ML model file on your computer and click Open.
• Or with object selected you can press Add Component -> ML Component on the Inspector panel

You can create and build ML Component completely in a script.

ML Component Settings

Let’s take a look at the ML Component UI:

• Render Order: the model can be run at a specific point during frame rendering if certain run options are selected. More about this in Scripting section.
• Model: is intended for your MLAsset resource.
• Auto build: if selected, the model will be built with specified input and output settings.
• Auto run: if selected, after the model is built it will run every frame, synchronously. It will start processing on frame update, and output results on frame render.

Auto run is the equivalent of running: mlComponent.runScheduled(true, MachineLearning.FrameTiming.OnRender, MachineLearning.FrameTiming.OnRender);

More information about run modes and FrameTiming can be found in Scripting ML Component guide .

Inputs

This section of Ml Component UI provides information on all input placeholders of the ML model and allows users to configure their settings.

Each Input is described with a name specified in the ML model asset (name can't be changed).

Next Options are available for each input:

• Shape: is represented with Width, Height and Channels. Width and Height can be changed if your network supports it. Number of channels can’t be changed.

Shape always consists of these three parameters. They will be set to 1 if the dimension of input and output is lower than 3.

• Texture: If the channel amount of the input is <= 4 MLComponent will suggest that this input could possibly (but not necessarily) be a texture and enable input texture field. Click on the Texture field and select input texture. In other cases you will need to modify input data using JavaScript. See scripting guide for details

Textures are fed into the ML Component are in the range \[0, 255\]

If the input texture has a different number of channels than model expects  - color space will be converted automatically for you and you don’t need to apply any additional operations.

• Input Transformer settings: Sometimes texture needs some preprocessing before feeding it to the neural network. To avoid complicated setup (e.g. Extra cameras) Lens studio allows you to perform a set of simple transformations on the input texture:
  • Stretch: if enabled - texture is stretched to the size (width and height) of model input.
  • Horizontal alignment - sets the horizontal alignment of the texture if stretch is off, adds padding if needed.
  • Vertical alignment - sets the vertical alignment of the texture if stretch is off, adds padding if needed.
  • FlipX: flips image on x axis.
  • FlipY: flips image on y axis.
  • Rotation: sets rotation to the one of 4 options.
  • Fill Color: if stretch is off defines a color added to the sides of the image to fit the input aspect ratio.

Outputs

Output section of the UI shows information about output placeholders. Every output is specified with a name.

• Shape: can’t be edited, but can change depending on the input shape.
• Transform: select an Input to get transform from. Output Transform is inverse to the Input Transform.
• Output Texture: Similarly as input, if the number of channels of the output is <= 4 the Create Output Texture button will be available. When you click the button, the new Texture Asset will be created in Resources. You can use it in components and materials as any other texture.

``This texture is not initialized until the model has finished its first run. And you may use it reliably only after this event.

Continue reading:  ML Component Scripting and Lifecycle