Multi-Object Detection

The Multi-Object Detection Template allows you create a machine learning model that detects certain objects on the screen, bring it to Lens Studio and run different effects based on the ML model output.

Template Walkthrough

The Multi-Object Detection Template comes up with a dedicated Berry Detection machine learning model that allows to classify and detect four different kinds of berries on the user's screen; Blueberry, Strawberry, Blackberry and Blueberry

You can learn how to import your own model here.

This template is controlled by of two main scripts: Ml Controller and ObjectDetectionController that are attached to scene objects with corresponding names.

ML Controller

This script will configure and run an ML Model, along with processing the Model’s outputs. This script generates a list of object detections on the device’s screen for each frame.

Each detection provides such information as

• Bounding box of the object on the screen
• Probability of the detection
• Class index
• Class label (if provided)

ML Controller Inputs

Listed below are the ML Controller script inputs:

• ML Model: An ML model asset exported from the notebook.

• Input Texture: Texture that is being processed with ML model. This value is set to Device Camera Texture by default.

• Class Settings: An array of pairs of the class name, that can be used by other scripts for display, and a boolean that defines whether detections of this class are being tracked.

• NMS Settings: A boolean that allows to tweak some advanced settings such as:
  • Score Threshold: A probability threshold class has to pass in order to be considered as detected. The lower the value is, the more false positive results you may receive, while a higher value may be more accurate but return less results.
  • IOU Threshold: Set Intersection Over Union threshold for detection matching. If IOU of two detected object’s bounding boxes is higher than this threshold value, they will be considered the same object (This is called NMS Algorithm).
  • Print Debug: Enable to print certain messages to logger or specified Text Component.

How ML Controller detects objects

Now that you have an idea on what the ML Controller can use as inputs, you will now learn how the script obtains information about detections.

MlController script has an onDetectionsUpdated event, that any script can add callback too. It passes an array of detections as its parameter:

/** @type {EventWrapper} Event that is called when detections are updated */
var onDetectionsUpdated = new EventWrapper(); //

Take a look at the EventModule script module, also located in the Asset Library, to learn more about EventWrapper class.

From any other script, you can add callback to this event and obtain list of detections as parameter:

// @input Component.ScriptComponent mlController
/** @type {ScriptComponent} */
var mlController = script.mlController;
mlController.onDetectionsUpdated.add(onUpdate);
/**
 * @param {DetectionHelpers.Detection} detections
 */
function onUpdate(detections) {
  /// do something with detections
}

Where Detection class is defined as:

/**
 * represents single object detection
 * @class
 * @param {[number, number, number, number]} bbox - bounding box coordinates in screen space
 * @param {number} score - detection score
 * @param {number} index - class index
 * @param {string} [label] - class label
 */

Object Detection Controller

ObjectDetectionController script creates a certain amount of SceneObject copies and configures them accordingly to object detections provided by MLController script.

• Object To Copy: A SceneObject to copy and place on each detection.
• Max Count: Maximum amount of copied objects.

There are several examples of objects to copy provided in the scene. To see how different frames will look, you can drag and drop the provided objects into the Object To Copy input field to see different effects applied:

• Match Detections: A boolean that allows you to match detected objects on screen with the detection from the previous frame. This allows the same scene object to follow the same detection and allows smoothing effects.
  • If the Match Detections checkbox is enabled, you can also set a Smooth Coefficient value to make objects move smoother on the screen.
• Match Threshold: An IOU coefficient threshold to consider if two detections are the same one.
• Lost Frames: Will keep detection for several frames, even if it was lost.
• Set Object Position: A boolean that enables control over screen or world object position of copied scene objects.

In the example image below, you may see the effect with no matching, then with matching and smoothing enabled:

In addition, the ObjectDetectionController script will look up a script component on the Object To Copy instances and will call certain APIs, if available. You can see a small function example below;

script.updateDetection(detection);
script.resetDetection(detection);

See Example [EDIT ME] script for code example which allows you to apply custom logic to each detection.

Another example to see how the script can be used, the next function allows you to change the material color based on the detection class index.

/**
 * @param {DetectionHelpers.Detection} detection
 */
function updateDetection(detection) {
  // set material color based on script index
  material.mainPass.baseColor = colors[detection.index];

  // set text background color
  labelText.backgroundSettings.fill.color = colors[detection.index];

  // set label
  labelText.text = detection.label + ' ' + detection.score.toFixed(2);
}
script.updateDetection = updateDetection;

Object Counter Example

The Object Counter example provides a simple setup that allows you to display a total of the amount of objects of a certain class on the screen. Select the Object Counter scene object to take a look at the available ObjectCounter script inputs:

• MLController: A reference to the MLController script.
• Counter Text: An array of Text components used to display a number of the objects.

Locate Scripts/ExampleScripts/ObjectCounter.js in the Resources panel to see how it is implemented. This script adds a callback to the onDetectionsUpdated event of the MLController script and iterates over the array of Detections to calculate the amount of detections per class.

You can see an example of how the script is set up here:

// add callback
//@input Component.ScriptComponent mlController
/** @type {ScriptComponent} */
var mlController = script.mlController;
//@input Component.Text[] counterText{"hint" : "index in array should match class index"}
/** @type {Text[]} */
var counterText = script.counterText;

mlController.onDetectionsUpdated.add(onDetectionsUpdated);

function onDetectionsUpdated(detections) {
  // reset counters
  for (var i = 0; i < classCount; i++) {
    countPerClass[i] = 0;
  }
  // update from detections
  for (i = 0; i < detections.length; i++) {
    countPerClass[detections[i].index] += 1;
  }
  // update text components if set
  for (i = 0; i < classCount; i++) {
    if (!isNull(script.counterText[i])) {
      counterText[i].text = countPerClass[i].toString();
    }
  }
}

Switch Class On And Off

The MLController script allows you to turn detection of certain classes on and off in script inputs. You may also do so in runtime by tapping on certain mesh visual by calling setClassEnabled api function of MLController.

mlController.setClassEnabled(classIndex, isTracking);

To set this up you will need the following:

• Attach ToggleClassTracking.js script to the mesh visual of your choice, like an Image for example, to tap on.
• Provide a reference to the MLController script.
• Select a class index in script input.

Training ML Model

While the template comes with a berry detection model for the MLComponent, you can make any kind of object detection by importing your own machine learning model. You will see an example of what this might look like below.

To learn more about Machine Learning and Lens Studio, please visit ML Overview page.

There are many different ways you can train your model. For this example, we will use Google Colaboratory.

Download the notebook from snapml-templates repository.

Then head over to Google Colaboratory, select the Upload tab, and drag the downloaded python notebook into the upload area.

The provided example is using a custom dataset annotated with Roboflow. Running the notebook will install all the necessary libraries and datasets.

With our files added, you can run the code by selecting Runtime > Run All in the menu bar.

This process may take a while to run, as creating a model is computationally intensive.

Download Model

You can scroll to the Train Loop section of the notebook to see how your machine learning model is coming along. Once you are happy with the result, you can download your .onnx model using the code we provided towards the end of the notebook.

Importing Model

Open the Multi-Object Detection template from the Lens Studio home screen or create a new Lens Studio project.

When bringing your model into the Lens Studio please consider the input range model accepts. Exported model takes input in the range [0, 1] while the Texture input of the ML Component in Lens Studio takes values from 0 to 255.

Drag and drop the exported .onnx file into the Resources panel and configure the input channel scale to be 0.0039215 (1/255).

If you need to change scale and bias after the model was imported, right-click on the model asset and select Update From Source. Then modify settings to your liking.

You will need to configure MLController to work with your model.

• Set Ml Model input with your model
• Configure Class Settings input to reflect your models class labels.

At this point, you should see your model up and running. You can tune experience to your liking as described in the section above.

Publishing Your Lens

Consider using the Scan Trigger feature when publishing your Lens. This is a great combination with SnapML as it enhance user experience by allowing them to immediately start detecting objects in the camera right after they activated the Scanned those objects.

Related guides

• SnapML Overview
• Object Detection
• Scan