Web Considerations

Performance Considerations & Limitations

While Camera Kit Web supports a variety of Lens features, performance may vary depending on the device, platform, and complexity of the Lens. Keep the following considerations in mind:

Feature Support & Limitations

• 6DoF (World Mesh) is not fully supported on Web and will default to surface tracking (3.5DoF), which may lead to inconsistent performance depending on the device and environment.
• Resource-intensive features, such as 3D Body and Hand Tracking, and SnapML-powered Lenses, may cause lower framerates, especially on Windows devices and mobile platforms.

Best Practices for Performance Optimization

• Test Lenses in a Web environment early and often.
  The preview window in Lens Studio does not accurately reflect how a Lens will perform in a Web browser. Performance varies based on factors like browser optimizations, GPU capabilities, and device constraints.

  • Use the Push-to-Web extension to send Lenses directly from Lens Studio to your Camera Kit Web app during development.
  • Alternatively, test Lenses in our sample web app, which supports Push-to-Web for quick validation.
  • Test across different browsers and devices to catch potential performance variations.

• Minimize resource-heavy tasks and rendering complexity.
  Lenses that perform well on mobile apps may struggle in a Web environment due to differences in rendering and processing power.

  • Reduce the complexity of 3D models, animations, and other assets to improve rendering speed.
  • Be mindful of SnapML models, 3D tracking, and high-resolution textures, as they can significantly impact performance.
  • Consider optimizing effects dynamically based on device capability to maintain smooth performance.

By keeping these factors in mind, you can ensure that your Lenses run efficiently across a wide range of Web environments.

Media Stream Best Practices

Requesting Ideal Video Dimensions

When requesting a MediaStream using navigator.mediaDevices.getUserMedia, always specify ideal dimensions in your MediaStreamConstraints. Without explicitly defining these constraints, cameras often default to low-resolution video with a 4:3 aspect ratio, even if they natively support 16:9. Defining ideal dimensions helps ensure higher quality and correct aspect ratio for your application.

const mediaStream = await navigator.mediaDevices.getUserMedia({
  video: {
    width: { ideal: 1280 },
    height: { ideal: 720 },
  },
});

Confirm Actual Video Dimensions

Requested dimensions in MediaStreamConstraints are not guaranteed. Always check the actual dimensions of the MediaStreamTrack using track.getSettings() and use these as the source of truth in your application.

const track = mediaStream.getVideoTracks()[0];
const { width, height } = track.getSettings();

MediaTrackSettings Inaccuracy on Mobile

On mobile devices in portrait mode, MediaTrackSettings may report incorrect values immediately after creating a MediaStream. This can lead to inaccurate resolution handling.

Camera Kit automatically compensates for this, but if you call source.setRenderSize, this behavior is disabled. To ensure accuracy, retrieve MediaTrackSettings after a short timeout.

setTimeout(() => {
  const track = mediaStream.getVideoTracks()[0];
  const { width, height } = track.getSettings();

  source.setRenderSize(width, height);
}, 100);

Avoid Requesting Unconventional Dimensions

Cameras can only return resolutions that the hardware natively supports. Requesting arbitrary dimensions—such as the full browser window size—to achieve a "full-screen" experience can cause the device to return low-resolution, 4:3 fallback video.

Instead, request a standard resolution (720p, 1080p, etc.) and use CSS to crop the video to the desired display size. In most cases, achieving a "full-screen" video experience requires either cropping the raw video or adding pillarboxing to maintain the correct aspect ratio.

canvas {
  width: 500px;
  height: 500px;
  object-fit: cover;
}

Alternatively, you can use source.setRenderSize to adjust the actual size of the output canvas. This approach may be more suitable depending on the Lenses being rendered.

const mediaStream = await navigator.mediaDevices.getUserMedia({
  video: {
    width: { ideal: 1280 },
    height: { ideal: 720 },
  },
});
const source = createMediaStreamSource(mediaStream);

source.setRenderSize(500, 500);

Use Standard Landscape Dimensions for Portrait Video on Mobile

Instinctively, developers often request 720x1280 (9:16) video or use window.innerWidth and window.innerHeight to match the screen size for full-screen portrait video. However, this frequently leads to unpredictable behavior, such as zoomed-in or distorted video.

Mobile devices do not support arbitrary aspect ratios and will return portrait or landscape video based on their physical orientation. To reliably capture portrait video, request a standard landscape resolution (e.g., 1280x720). The device will automatically adjust based on its orientation.

Requesting unconventional dimensions can force the camera to return low-resolution fallback video with an incorrect aspect ratio.

const mediaStream = await navigator.mediaDevices.getUserMedia({
  video: {
    width: { ideal: 1280 },
    height: { ideal: 720 },
  },
});

Stop Existing MediaStreamTracks Before Switching Cameras

When switching cameras, always stop the current MediaStreamTrack before requesting a new one. Failing to do so can cause unpredictable behavior, especially on mobile Safari, and may even lead to crashes.

let mediaStream: MediaStream;

async function getMediaStreamForDevice(deviceId: string): Promise<MediaStream> {
  if (mediaStream) {
    const track = mediaStream.getVideoTracks()[0];
    track.stop();
  }

  mediaStream = await navigator.mediaDevices.getUserMedia({
    video: {
      deviceId,
      width: { ideal: 1280 },
      height: { ideal: 720 },
    },
  });

  return mediaStream;
}

Request a MediaStream Before Enumerating Devices

Calling navigator.mediaDevices.enumerateDevices() before requesting a MediaStream will return incomplete device information. To ensure full device details are available, first call navigator.mediaDevices.getUserMedia().

const mediaStream = await navigator.mediaDevices.getUserMedia({
  video: {
    width: { ideal: 1280 },
    height: { ideal: 720 },
  },
});

const mediaDevices = await navigator.mediaDevices.enumerateDevices();

Remember the User’s Last Selected Device

For a better user experience, consider caching and reusing the most recently used deviceId when requesting a MediaStream. After obtaining the stream, always verify the deviceId from track.getSettings() to ensure the correct device is selected and update the UI accordingly.

let currentDeviceId = window.localStorage.getItem('currentDeviceId') ?? undefined;

const mediaStream = await navigator.mediaDevices.getUserMedia({
  deviceId: currentDeviceId
  video: {
    width: { ideal: 1280 },
    height: { ideal: 720 },
  },
});

const track = mediaStream.getVideoTracks()[0];
const settings = track.getSettings();
const deviceId = settings.deviceId ?? '';

if (deviceId !== currentDeviceId) {
  currentDeviceId = deviceId;
  window.localStorage.setItem('currentDeviceId', currentDeviceId)
}

Camera Kit Integration

Set Camera Type for Back-Facing Cameras

When using a Lens with the back-facing camera, ensure Camera Kit is aware of this by setting the correct camera type. This enables necessary features for proper rendering and tracking.

let mediaStream: MediaStream;

async function updateCameraKitSource(
  deviceId: string,
  facingMode: 'user' | 'environment' = 'user',
  session: CameraKitSession
): Promise<MediaStream> {
  if (mediaStream) {
    const track = mediaStream.getVideoTracks()[0];
    track.stop();
  }

  mediaStream = await navigator.mediaDevices.getUserMedia({
    video: {
      deviceId,
      facingMode,
      width: { ideal: 1280 },
      height: { ideal: 720 },
    },
  });

  cost source = createMediaStreamSource(mediaStream, {
    cameraType: facingMode
  })

  session.setSource(source);
}

Account for Pixel Density When Setting Render Size

When defining the render size of your output canvas, consider the display’s pixel density using window.devicePixelRatio. This ensures that Lens assets render at their intended resolution, preventing unwanted upscaling and maintaining visual clarity.

const width = 1920;
const height = 1080;

const mediaStream = await navigator.mediaDevices.getUserMedia({
  video: {
    width,
    height,
  },
});
const source = createMediaStreamSource(mediaStream);

await source.setRenderSize(
  width * window.devicePixelRatio,
  height * window.devicePixelRatio
);

Audio Handling

Explicitly Specify Audio in MediaStream Requests

Browser behavior varies when the audio property is omitted from MediaStreamConstraints. To avoid unexpected results, always explicitly define whether your MediaStream should include an audio track.

const mediaStream = await navigator.mediaDevices.getUserMedia({
  video: {
    width: { ideal: 1280 },
    height: { ideal: 720 },
  },
  audio: false,
});

Camera Kit Does Not Output Audio

Camera Kit renders video to a <canvas> but does not output audio. Some Lenses can react to microphone input, but the rendered video itself contains no sound.

If you need to capture both video and microphone audio together, you must capture the canvas stream and manually combine it with the original audio track.

const liveRenderTarget = <HTMLCanvasElement>document.getElementById('canvas');
const cameraKit = await bootstrapCameraKit({ apiToken: '<API_TOKEN>' });
const session = await cameraKit.createSession({ liveRenderTarget });
const mediaStream = await navigator.mediaDevices.getUserMedia({
  video: {
    width: { ideal: 1280 },
    height: { ideal: 720 },
  },
  audio: true,
});
const source = createMediaStreamSource(mediaStream);

await session.setSource(source);

session.play();

const canvasMediaStream = liveRenderTarget.captureStream();
const videoTrack = canvasMediaStream.getVideoTracks()[0];
const audioTrack = mediaStream.getAudioTracks()[0];

return new MediaStream([videoTrack, audioTrack]);