Skip to main content

Camera Module

SPECS offers APIs to retrieve the camera frame (what the user is currently seeing) to better understand and build experiences around the user’s real-world environment.

Requesting + Receiving Frames​

Deprecated: requestImage is deprecated and doesn't work on SPECS. Use createCaptureSession from the session-based camera API for high-resolution stills. requestCamera stays supported: a session Frame provides pixels (texture and imageData), a timestamp and exposure metadata, but not yet the camera's intrinsic and extrinsic parameters. If your Lens projects 3D points with DeviceCamera, keep using requestCamera.

The CameraModule enables developers to request the camera frame. Camera frame requests deliver a steady stream of images, albeit at lower quality: this approach is recommended for applications that need video-like consistency in frame delivery. For high-resolution stills, use a CaptureSession.

Camera Frame Requests​

To begin receiving camera frames, construct a CameraRequest that names the camera and the resolution you want. Use Default_Color for the RGB camera or Left_Grayscale for the grayscale camera. SPECS doesn't support Left_Color, and Right_Color returns the same RGB camera as Default_Color. Set the resolution with imageSmallerDimension, the size in pixels of the smallest dimension to return.

For example:

@component
export class CameraTextureExample extends BaseScriptComponent {
  private cameraModule: CameraModule = require('LensStudio:CameraModule');
  private cameraTexture: Texture;

  onAwake() {
    this.createEvent('OnStartEvent').bind(() => {
      const cameraRequest = CameraModule.createCameraRequest();
      cameraRequest.cameraId = CameraModule.CameraId.Default_Color;
      this.cameraTexture = this.cameraModule.requestCamera(cameraRequest);
    });
  }
}

createCameraRequest may not be invoked inside onAwake event.

The requestCamera method takes a CameraRequest and returns a Texture. The texture contains the opaque handle that refers to the underlying camera data, and can be used as an input into an MLComponent or uploaded to a remote service via RemoteMediaModule.

To receive a notification each time the frame is updated, use the CameraTextureProvider which can be accessed via cameraTexture.control. The CameraTextureProvider has an onNewFrame event that can be utilized as shown below:

@component
export class CameraFrameExample extends BaseScriptComponent {
  private cameraModule: CameraModule = require('LensStudio:CameraModule');

  onAwake() {
    this.createEvent('OnStartEvent').bind(() => {
      const cameraRequest = CameraModule.createCameraRequest();
      cameraRequest.cameraId = CameraModule.CameraId.Default_Color;
      const cameraTexture = this.cameraModule.requestCamera(cameraRequest);

      const onNewFrame = (cameraTexture.control as CameraTextureProvider)
        .onNewFrame;
      const registration = onNewFrame.add((frame: CameraFrame) => {
        // Process the frame
      });
      this.createEvent('OnDestroyEvent').bind(() => {
        onNewFrame.remove(registration);
      });
    });
  }
}

Getting Camera Information​

The Camera API also provides information about the cameras used on SPECS in order to understand how the camera transforms 3D space into 2D space. This information is provided by the DeviceCamera which is accessible via the DeviceInfoSystem.

@component
export class CameraInfoExample extends BaseScriptComponent {
  onAwake() {
    this.createEvent('OnStartEvent').bind(() => {
      // Select the camera
      const camera = global.deviceInfoSystem.getTrackingCameraForId(
        CameraModule.CameraId.Default_Color
      );

      // Retrieve camera properties
      const focalLength = camera.focalLength;
      const principalPoint = camera.principalPoint;
      const resolution = camera.resolution;
      const pose = camera.pose;
    });
  }
}

Additionally, DeviceCamera includes project and unproject methods that make it easier to convert 3D points into 2D points and vice versa.

Code Example​

In Lens Studio click "+" in the Asset Browser window, select "General" and create a new TypeScript or JavaScript file. Copy and paste the following code into the file and attach the script to an object in the scene. Add an image component to the empty reference in the inspector.

@component
export class ContinuousCameraFrameExample extends BaseScriptComponent {
  private cameraModule: CameraModule = require('LensStudio:CameraModule');
  private cameraRequest: CameraModule.CameraRequest;
  private cameraTexture: Texture;
  private cameraTextureProvider: CameraTextureProvider;

  @input
  @hint('The image in the scene that will be showing the captured frame.')
  uiImage: Image | undefined;

  onAwake() {
    this.createEvent('OnStartEvent').bind(() => {
      this.cameraRequest = CameraModule.createCameraRequest();
      this.cameraRequest.cameraId = CameraModule.CameraId.Default_Color;

      this.cameraTexture = this.cameraModule.requestCamera(this.cameraRequest);
      this.cameraTextureProvider = this.cameraTexture
        .control as CameraTextureProvider;

      this.cameraTextureProvider.onNewFrame.add((cameraFrame) => {
        if (this.uiImage) {
          this.uiImage.mainPass.baseTex = this.cameraTexture;
        }
      });
    });
  }
}

Camera Stream and Capture Sessions​

The session-based camera API gives Lenses direct access to the SPECS cameras: camera and stream discovery, continuous streaming with an auto-exposure region of interest, and high-resolution still capture with per-shot crop. It extends CameraModule.

The session API runs on SPECS only. In Lens Studio Preview, getCameras() returns an empty array, so guard your code to run on device. On device, getCameras(), getStreams() and createStreamingSession() / createCaptureSession() can throw an Error when the camera client cannot be created or discovery fails; wrap them in try/catch.

Camera frames are sensitive user data. A Lens that sends frames to a backend (for example, Base64-encoding imageData for an ML or LLM service) combines camera access with internet access, which requires Transparent Permission to be publishable.

Discovering Cameras and Streams​

getCameras() returns the physical cameras. Each CameraInfo reports a type (Rgb or Grayscale), a location (Left or Right), a facing (World or User) and a human-readable name. camera.getStreams() returns that camera's stream capabilities, sorted largest-first by pixel area. Each StreamInfo has a fixed width and height, an imageFormat (Jpeg, Yuv420 or Y8), a quality tier (Performance, Balanced or Enhanced), a maxFrameRate and an operation.

Each capability works with one kind of session, reported by StreamInfo.operation:

  • CameraModule.Operation.Streaming: use with a StreamingSession.
  • CameraModule.Operation.Capture: use with a CaptureSession.

Choose by operation first, then filter by format, size or quality. Do not infer the operation from format or resolution: the same size and format can be offered once for each operation. description is a human-readable label for logs. Its format can change and it is not unique, so do not parse it or use it as an identifier.

const cameraModule: CameraModule = require('LensStudio:CameraModule');
let cameras: CameraModule.CameraInfo[] = [];
try {
  cameras = cameraModule.getCameras();
} catch (e) {
  print('Camera unavailable: ' + e);
}
const rgb = cameras.find((c) => c.type === CameraModule.CameraType.Rgb);
const streams = rgb ? rgb.getStreams() : [];
const preview = streams.find(
  (s) => s.operation === CameraModule.Operation.Streaming
); // largest streaming capability
const photo = streams.find(
  (s) =>
    s.operation === CameraModule.Operation.Capture &&
    s.imageFormat === CameraModule.ImageFormat.Jpeg
); // largest JPEG still

Current SPECS hardware reports two world-facing cameras:

CameraSizeFormatQualityOperation
rgb (Right)4032×3024Yuv420EnhancedCapture
rgb (Right)4032×3024JpegEnhancedCapture
rgb (Right)2016×1512Yuv420EnhancedStreaming
rgb (Right)2016×1512JpegBalancedCapture
rgb (Right)1008×756Yuv420PerformanceStreaming
rgb (Right)1008×756JpegPerformanceCapture
mono (Left)1124×1364Y8PerformanceStreaming

Every capability reports a maxFrameRate of 30. Always select capabilities at runtime; the list can differ on other devices and software versions.

Lenses built for SPECS S8​

Lenses built for SPECS S8 find capabilities with getSupportedStreams() and StreamFilter. Those Lenses keep working unchanged. Both APIs are deprecated and left out of the Lens Studio type definitions: in new Lenses, use getCameras() and camera.getStreams(), and read stream.camera.type, stream.camera.location and stream.maxFrameRate instead of the StreamInfo fields type, location and frameRate.

Streaming​

A StreamingSession delivers frames through onNewFrame until you stop it. Sessions are restartable: stop() releases the camera and start() can be called again.

Set StreamConfig.minFrameRate to the camera rate your Lens needs (0, the default, requests the capability's maxFrameRate). Assigning a negative value throws, and start() throws when the rate, rounded up to a whole number, exceeds maxFrameRate. The camera can run faster when another client needs more. Delivery to your Lens can be lower than the camera rate when its per-frame work is slow. See Performance for measured rates.

@component
export class CameraStreaming extends BaseScriptComponent {
  @input
  image: Image;

  private cameraModule: CameraModule = require('LensStudio:CameraModule');
  private session: CameraModule.StreamingSession;

  onAwake() {
    const rgb = this.cameraModule
      .getCameras()
      .find((c) => c.type === CameraModule.CameraType.Rgb);
    // Smallest streaming capability: full frame rate at the lowest cost.
    const stream = rgb
      ? rgb
          .getStreams()
          .filter((s) => s.operation === CameraModule.Operation.Streaming)
          .pop()
      : undefined;
    if (!stream) {
      print('No streaming camera available');
      return;
    }

    this.session = this.cameraModule.createStreamingSession();
    this.session.onNewFrame.add((frame) => {
      if (frame.texture) {
        this.image.mainPass.baseTex = frame.texture;
      }
    });
    this.session.onStatusChanged.add((status) => {
      if (status === CameraModule.StreamStatus.Disconnected) {
        print('Camera disconnected; create a new session to continue');
      }
    });

    const config = new CameraModule.StreamConfig();
    config.stream = stream;
    config.minFrameRate = 15;
    const result = this.session.start(config);
    if (result !== CameraModule.Result.Success) {
      print('Streaming failed to start: ' + result);
    }

    this.createEvent('OnDestroyEvent').bind(() => this.session.stop());
  }
}

Tuning keys can be set before start() with config.setConfig(key, values) and changed while streaming with session.setConfig(key, values):

ConfigKeyValuesNotes
AutoExposureRegion[left, right, top, bottom] (+ weight)Normalized 0.0–1.0 image coordinates, origin top-left, with left < right and top < bottom; optional fifth value is a weight from 0 to 1 (default 1). Steers auto-exposure toward the region; does not crop.
SnapshotMfnrEnable[0] or [1]Capture sessions only (set before start()): multi-frame noise reduction. Rejected on streaming sessions with Result.InvalidConfig.

AutoExposureRegion takes one region per session, and the rest of the frame can then be brighter or darker. The region can stay in effect after the session stops: set [0, 1, 0, 1] before stop() to meter the whole frame again. It works on the 1008×756 RGB and grayscale streams, and has no effect on the 2016×1512 RGB stream or on capture sessions. Lenses built for SPECS S8 that set ConfigKey.AeRoi keep its [left, top, right, bottom] order.

Manual exposure isn't supported: cameras always run automatic exposure. Read the values in use from each frame with frame.getMetadata(CameraModule.MetadataKey.Iso) and MetadataKey.ExposureTimeNs.

Still Capture​

A CaptureSession prepares the still pipeline in start() and returns one Frame per capture(). Start the session when your Lens loads rather than at shutter press: the first capture also pays for warm-up. One capture may be in flight at a time.

To save power, the camera service ends a capture session that goes 60 seconds without a capture. onStatusChanged reports Disconnected, and later capture() calls reject. Each capture() restarts the 60-second timer. After Disconnected, create a new session to capture again.

@component
export class CameraCapture extends BaseScriptComponent {
  @input
  image: Image;

  private cameraModule: CameraModule = require('LensStudio:CameraModule');
  private capture: CameraModule.CaptureSession;
  private ready = false;
  private busy = false;

  onAwake() {
    const rgb = this.cameraModule
      .getCameras()
      .find((c) => c.type === CameraModule.CameraType.Rgb);
    const stream = rgb
      ? rgb
          .getStreams()
          .find(
            (s) =>
              s.operation === CameraModule.Operation.Capture &&
              s.imageFormat === CameraModule.ImageFormat.Jpeg
          )
      : undefined;
    if (!stream) {
      print('No still capture available');
      return;
    }

    this.capture = this.cameraModule.createCaptureSession();
    this.capture.onStatusChanged.add((status) => {
      if (status === CameraModule.StreamStatus.Disconnected) {
        this.ready = false; // create a new session to continue
      }
    });

    const config = new CameraModule.StreamConfig();
    config.stream = stream;
    this.ready = this.capture.start(config) === CameraModule.Result.Success;

    // Capture two seconds after start; call takePhoto() from your own trigger instead.
    const delay = this.createEvent('DelayedCallbackEvent');
    delay.bind(() => this.takePhoto());
    delay.reset(2);

    this.createEvent('OnDestroyEvent').bind(() => this.capture.stop());
  }

  async takePhoto() {
    if (!this.ready || this.busy) {
      return;
    }
    this.busy = true;
    try {
      const overrides = new CameraModule.StreamConfigOverrides();
      overrides.crop = Rect.create(-0.5, 0.5, -0.5, 0.5); // centered 50% crop for this shot
      const frame = await this.capture.capture(overrides);
      if (frame.texture) {
        this.image.mainPass.baseTex = frame.texture;
      }
    } catch (e) {
      print('Capture failed: ' + e);
    } finally {
      this.busy = false;
    }
  }
}

capture() rejects when the session is not started, another capture is in flight, the crop is not finite, the session stops or disconnects before the frame arrives, the camera service rejects the request, or no frame arrives within 30 seconds. A frame whose image could not be read still resolves, with null texture and imageData, so check before use.

Still size​

A capture session returns stills at its capability's size unless you set StreamConfig.outputWidth and outputHeight. Any positive, even size up to the capability's width and height works, for example a 756×1008 portrait still from the 4032×3024 capability. A size with another aspect ratio than the capability is a centered crop of the camera's view, so the image is never stretched.

config.stream = photo; // a Capture capability, for example 4032x3024 Jpeg
config.outputWidth = 756;
config.outputHeight = 1008;

Assigning zero, a negative or an odd value throws. start() throws when only one of the two is set, when the size is larger than the capability, or when either is set on a StreamingSession: streams always have their capability's size. Set both to null to use the capability's size again.

Crop​

StreamConfig.crop sets a default crop for every capture of the session; StreamConfigOverrides.crop overrides it for one shot. Crops are Rect values in normalized image space: [-1, 1] on each axis, with +1 at the top and right of the image, created with Rect.create(left, right, bottom, top). The cropped region is scaled to the still's size: outputWidth×outputHeight when set, otherwise the capability's size. Give the crop the same aspect ratio as the still. Streaming sessions ignore crops.

RegionRect
Full imageRect.create(-1, 1, -1, 1)
Centered 50%Rect.create(-0.5, 0.5, -0.5, 0.5)
Top-left quadrantRect.create(-1, 0, 0, 1)
Bottom-right quadrantRect.create(0, 1, -1, 0)

Crop and AutoExposureRegion use different coordinate spaces: crop is [-1, 1] normalized image space with y up, AutoExposureRegion is 0..1 image coordinates with y down.

Working with Frames​

  • frame.imageData is the raw image, tightly packed:

    • Jpeg: the JPEG file.
    • Yuv420: the Y plane (width * height bytes), then half-resolution chroma interleaved V then U (NV21, width * height / 2 bytes).
    • Y8: one byte per pixel.

    The array shares the frame's buffer; treat it as read-only.

  • frame.texture is an RGBA texture of the image; grayscale frames have R = G = B. Each session reuses one Texture: reading texture on a newer frame, including the next capture, replaces the image in the Texture you got from an earlier frame, and a later session of the same size can reuse it after stop(). To keep an image, keep its imageData, or copy the texture (see below). After a session stops, reading texture on a frame you kept decodes the image again into a new texture, so read it while the session runs.

  • Streaming frames are short-lived. A session keeps the pixels of its two most recent frames only. On older frames texture returns null, and so does imageData unless your Lens still holds the array it returned earlier. Read what you need inside onNewFrame. An imageData array you have already read stays valid and keeps that frame's bytes until it is garbage collected, so read imageData only on the frames you process. Captured frames keep their pixels for as long as you reference them.

  • frame.getMetadata(key) returns TimestampNs (CLOCK_MONOTONIC, always present) and Iso, ExposureTimeNs and FrameRate when the camera reports them. It returns null when a value is unavailable.

Reading texture pixels​

Assigning a frame texture to a material (image.mainPass.baseTex = frame.texture) works inside onNewFrame, and Base64.encodeTextureAsync works on a captured frame right after await capture(). To read pixels on the CPU, read frame.texture inside onNewFrame, keep that Texture, and copy it with ProceduralTextureProvider.createFromTexture in an UpdateEvent: inside onNewFrame or right after await capture(), createFromTexture throws an error. Hand the texture to your update handler instead:

@component
export class CameraPixels extends BaseScriptComponent {
  private cameraModule: CameraModule = require('LensStudio:CameraModule');
  private session: CameraModule.StreamingSession;
  private pending: Texture | null = null;
  private pixel = new Uint8Array(4);

  onAwake() {
    const rgb = this.cameraModule
      .getCameras()
      .find((c) => c.type === CameraModule.CameraType.Rgb);
    const stream = rgb
      ? rgb
          .getStreams()
          .filter((s) => s.operation === CameraModule.Operation.Streaming)
          .pop()
      : undefined;
    if (!stream) {
      print('No streaming camera available');
      return;
    }

    this.session = this.cameraModule.createStreamingSession();
    this.session.onNewFrame.add((frame) => {
      this.pending = frame.texture; // keep a reference only
    });
    const config = new CameraModule.StreamConfig();
    config.stream = stream;
    const result = this.session.start(config);
    if (result !== CameraModule.Result.Success) {
      print('Streaming failed to start: ' + result);
      return;
    }

    this.createEvent('UpdateEvent').bind(() => {
      if (this.pending) {
        const copy = ProceduralTextureProvider.createFromTexture(this.pending);
        this.pending = null;
        const provider = copy.control as ProceduralTextureProvider;
        provider.getPixels(0, 0, 1, 1, this.pixel); // RGBA of the pixel at (0, 0)
      }
    });
    this.createEvent('OnDestroyEvent').bind(() => this.session.stop());
  }
}

Results and Status​

start(), setConfig() and stop() return a CameraModule.Result. Invalid values of minFrameRate, outputWidth and outputHeight throw instead, where you assign them or from start() (see Streaming and Still size).

ResultMeaning
SuccessThe call succeeded.
NotInitializedThe session's camera client is not available (rare).
InvalidArgumentMissing stream, an invalid setConfig() value, or a non-finite crop.
InvalidConfigThe capability's operation does not match the session or it cannot be opened, or a key is not allowed here.
AlreadyStartedstart() on a running session.
NotStartedstop() or setConfig() before start().
SessionStoppedThe session was disconnected; create a new one.
ErrorGenericThe camera service could not be reached, could not open the camera or could not apply a setConfig() change, or a session limit was reached.

onStatusChanged reports:

  • Interrupted: frames pause, for example while the camera switches modes because another session started, or while the camera recovers from an error. The session stays started. The camera can drop a capture in flight: its Promise rejects about 30 seconds after capture(). Capture again after Resumed.
  • Resumed: frames flow again.
  • Disconnected: the system ended the session: camera access was revoked, the camera service stopped the stream (for example, another stream switched the camera to a mode without this stream's size), the camera service restarted, or a capture session went 60 seconds without a capture. The session is finished; create a new one.

Performance​

Measured on SPECS with October 2026 software, three runs each. Your results vary with scene, temperature and other camera clients. Frame rates delivered to onNewFrame depend on what the Lens reads each frame:

StreamFrames onlyimageData every frametexture every frameReading texture (mean)
mono 1124×1364 Y830.1 fps29.9–30.0 fps30.0 fps8.6–8.7 ms
rgb 1008×75630.0–30.1 fps30.0 fps29.9–30.0 fps3.9–4.0 ms
rgb 2016×151227.7–30.1 fps29.9–30.1 fps29.8 fps14.8–15.1 ms

Reading texture converts that frame to RGBA and uploads it to the GPU, which costs the most at 2016×1512. Read it only on the frames you display or process. Reading it on occasional frames costs more per read, because the conversion then runs when you read it: at 2016×1512, one read every ten frames took 29–92 ms, with a mean of 41–61 ms per run.

First frame after start(): mono 195–281 ms, rgb 1008×756 184–236 ms, rgb 2016×1512 418–465 ms.

Capturestart()First capture()Later captures (mean)
1008×756 Jpeg176–247 ms173–209 ms98–116 ms
2016×1512 Jpeg365–397 ms125–149 ms125–133 ms
4032×3024 Jpeg285–310 ms253–263 ms200–223 ms
4032×3024 Yuv420283–449 ms252–878 ms220–233 ms
4032×3024 Jpeg, MFNR enabled549–560 ms

Limits and Best Practices​

  • IR cameras are not exposed: getCameras() lists only the RGB and grayscale cameras.
  • Still capture from the grayscale camera is not supported; use the RGB camera.
  • A Lens can run up to three streaming sessions across all its CameraModules, at most one at 2016×1512, or one streaming session and one capture session. start() past a limit returns Result.ErrorGeneric; stop a session to free its slot.
  • Starting a capture session while an RGB stream runs briefly interrupts the stream (Interrupted, then Resumed).
  • Full-resolution images are large: a 4032×3024 frame's texture holds 48.8 MB of RGBA, and its Yuv420 imageData 18.3 MB. A captured frame keeps its pixels while you reference it, so drop captures you are done with and keep few full-resolution images at once. The script engine frees a dropped capture only when it next collects garbage. Until then, many 4032×3024 Yuv420 captures can add up to hundreds of MB. For repeated full-size stills, use the much smaller Jpeg. Under system memory pressure, SPECS can close a Lens that uses a lot of memory.
  • minFrameRate is rounded up to a whole number of frames per second.
  • StreamingSession and CaptureSession stop automatically when the Lens closes.
Was this page helpful?
Yes
No