Skip to main content
Version: 4.55.1

ASL Fingerspelling

ASL Fingerspelling template allows to detect and translate American Sign Language alphabet letters or numbers. This template is built on top of 2D hand tracking technology in combination with SnapML.

The template comes up several script examples that show how to use Fingerspelling Custom Component for different experiences.

Fingerspell Detector Custom Component can be found in the Custom Components section of the Asset Library

The model used in the Fingerspelling Custom Component is provided by SignAll. This template is a great example of how developers can use SnapML to deploy their machine learning models to add new unique capabilities for distribution across Snapchat.

Guide​

The ASL Fingerspelling template has two main objects: Fingerspell Detector and Fingerspell Hint

Fingerspell Detector​

Fingerspell Detector Custom Component detects user gestures by processing 2D Hand Tracking joint positions with an ML model

  • Mode: switch between Letters or Numbers mode.
  • Print Log: prints messages to the Logger if enabled.
  • Print To Text: set Text component to duplicate logger messages to.
  • Threshold: minimum probability required in order to detect gesture .
  • Min Occurrence: how many frames a gesture has to occur for in order to be detected.
  • Repeat After: number of frames required for character to be detected again.

Let's take a look at the Fingerspell Detector Custom Component API:

//@input Component fingerspellDetector

//On Character found event
//Usage: adding callback function
script.fingerspellDetector.onNewChar.add(function (c) {
  print(c);
});

//On Hand Found Event
//Usage: adding callback function
script.fingerspellDetector.onHandFound.add(function () {
  print('Hand Found');
});

//On Hand Lost Event
//Usage: adding callback function
script.fingerspellDetector.onHandLost.add(function () {
  print('Hand Lost');
});

//Enable or disable log printing
script.fingerspellDetector.printLog = true;

//Enable or disable numeric mode
script.fingerspellDetector.numericMode = false;

// Set minimum probability required in order to detect gesture
script.fingerspellDetector.threshold = 0.5;

// Set how many frames a gesture has to occur for in order to be detected
script.fingerspellDetector.minOccurrence = 5;

// Set number of frames required for character to be detected again
script.fingerspellDetector.repeatAfter = 80;

Fingerspell Hint​

Fingerspell Hint allows users to visualize letters of the English Alphabet by playing the gesture animation and displaying text on the screen.

The template comes with a specific 3D model of hand with animation for each letter of the alphabet

  • Text Component: Text component to display current character.
  • Animation Mixer: Animation Mixer of a 3D model.
  • Animation Speed: multiplier of the animation speed.
  • Hand Tracking: Hand Tracking component.

Examples​

The template comes with four examples to demonstrate how you can use the Fingerspell Detector to create an interactive experience. Take a closer look at the scripts in the Scripts/Examples folder for more information.

Display Sign​

Allows you to detect a sign gesture and display it on the screen.

Spell a Word​

Prompts user to repeat gestures of the characters from a specified word or combination one-by-one. The script displays the next character once the previous was detected. Finally, a custom behavior trigger is called once every character has been detected.

  • Text to Spell: a Text component to get text from.
  • On Complete Behavior: a custom Behavior trigger name that will be called once text is spelled.

To set a word or word combination to spell, select a Text to Spell [EDIT_ME] scene object and edit text input of the Text Component in the Inspector panel:

Reference to this Behavior is used by Particles Scene Object

Take a look at the Behavior guide to learn more.

Sign To Type​

Allows the user to display the letter the user signed with their hand and adds it to the Text component.

  • Text Display: Text component to add text to.
  • Default text: hint text that is displayed before the first character is added.
  • Delay Before Next: delay in seconds before the next character can be added.

Keyboard​

Allows the user to tap on screen to edit text using keyboard and then use the on screen hint to sign the specified sentence.

  • Text: a Text component to take text from.
  • On Complete Behavior: a custom Behavior trigger name that will be called once text is spelled.
Was this page helpful?
Yes
No