Style Transfer

Style Transfer is available in Lens Studio Asset Library. Import the asset to your project, create new Orthographic camera and place the prefab under it.

The Style Transfer template allows you to use a Style Transfer model to visually change how the camera feed looks based on another image. The template allows you to choose between always running it in real time, running it on demand, or automatically choosing between the two for the best user experience.

Tutorial

Guide

  If you have your own style transfer model, you can skip down to the Importing Your Model section below.

Creating a Model

The Style Transfer template relies on a machine learning model to define how the camera feed should be visually changed. The template comes with an example model, but you’ll want to create your own to make your unique style transfer Lens.

To learn more about Machine Learning and Lens Studio, take a look at the ML Overview page.

Prerequisites

To create a model, you’ll need a

• Style reference image: any image that has the style you want
• Machine learning training code: code that describes how the model is trained (this is sometimes referred to as a notebook). Please find our example notebook here.
• Data set: collection of data that our code will use to learn from (in this case we will use the COCO data set)

Try using your own reference image! You can use the same training code and dataset.

When using a data set to train your model, make sure that you adhere to the usage license of that dataset.

Training Your Model

There are many different ways you can train your model. For our example, we will use Google Colaboratory. To see other ways of training, take a look at the ML Frameworks page of the guide section.

Head over to Google Colaboratory, select the Upload tab, and drag the notebook to the upload area.

The example notebook is well documented with information about what each section of the code is doing. Take a look at the notebook to learn more about the training process itself!

Once your notebook has been opened, you can add your style reference image. You can also add an image to test your results on:

Don't forget to upload a style reference image style_image.png and the image you want to test on test_image.png``

With our files added, you can run the code by choosing  Runtime > Run All in the menu bar. This process may take a while to run, as creating a model is computationally intensive.

Downloading your Model

You can scroll to the Train Loop section of the notebook to see how your machine learning model is coming along.

Once you are happy with the result, you can download your .ONNX model!

Importing your Model

Now that we have our model, we’ll import it into Lens Studio.

You can drag and drop your .ONNX file into the Asset Browser panel to bring it into Lens Studio.

Next, we’ll tell the template to use this model. In the Scene Hierarchy panel, select ML Component. Then, in the Inspector panel, click on the field next to Model, and then in the pop up window, choose your newly imported model.

Connecting your Model

With our model hooked in, what we need to do now is to tell the Style Transfer Controller to connect to our Style Transfer model.

First, take a note of the input and output parameter in the ML Component. This should be the same as what the notebook has specified. Then, select the Style Transfer Controller object in the Scene Hierarchy panel, and in the Inspector panel, type in Input name and Output name as it is shown in the ML Component.

You should now see the Preview panel showing a result similar to what you saw earlier!

Customizing your Lens Experience

With your style transfer working, you can preview and publish your Lens! However, you can customize your Lens even more!

Changing Run mode

By default the Lens will adapt to running your model in real time or on demand depending on the device it is running on. This is the best option as it will try to optimize the Lens for your users.

To change this, select the Style Transfer Controller object in the Scene Hierarchy panel, and choose one of the options:

• Adapt to Device Performance: Depending on the device, the style transfer will run in Realtime or On Demand
• Realtime: Style transfer will run every frame on every device (may result in lower frame rate for some devices)
• Run on Demand: Style transfer will run when the user presses a button.

Changing Icons

You’ll notice that when you are in Run on Demand mode, you have a set of icons that you can press to run the style transfer. To use your own icon, in the Scene Hierarchy panel, press the drop down next to Live Camera, On Demand Buttons. Underneath this object, you will see two objects: Photo Button and Reset Button. Select the button you want to modify, and in the Inspector panel, replace your texture. Take a look at the Image guide for more information

Changing Output Image

By default, the Style Transfer result is displayed as a full screen image. However, you may want to change how it is displayed. To change this, in the Scene Hierarchy panel, press the down arrow next to the Orthographic Camera, then Style Transfer Output, and finally select the Output Image. Then, in the Scene panel, you will be able to modify your Output image.

To learn more about laying out your image on the screen check out the Screen Transform guide.

 This image is referenced in the Style Transfer Controller in the Output Image field. You can set your own image in this field as well.

Previewing Your Lens

You’re now ready to preview your Lens! To preview your Lens in Snapchat, follow the Pairing to Snapchat guide.