Gaussian Splatting
Gaussian splatting, introduced in the 2023 paper “3D Gaussian Splatting for Real-Time Radiance Field Rendering”, is a modern approach for modeling and rendering 3D scenes. It stands out from Radiance Field methods like NeRFs for its quick setup (training) and faster rendering times, while maintaining or improving visual quality, and being easier to understand and modify.
The data is a cloud of “Splats”, generated automatically from a set of photos from different angles.
Importing Gaussian Splatting
In the Asset Browser
panel, click on +
, then Import Asset
. Next select, the gaussian splatting .ply
file you want to import.
You can drag a .ply
file directly into the scene hierarchy to import the asset and create a new visual.
You can see the number of splats, and size that the compressed Gaussian Splatting asset file takes up inside the Lens, in the Inspector
panel when your asset is selected.
PlayCanvas provides a powerful 3D Gaussian Splat editor called SuperSplat. Additionaly, they provide a guide to authoring PLY files.
Gaussian Splatting Component
This component allows you to display the Gaussian splatting itself. Add it to any scene object to see your splats in your scene.
This is similar to how you would use the Render Mesh Visual
component to display 3D meshes.
Gaussian Splatting Material
Optionally, you can override the default Gaussian Splatting material with a custom one, to create special effects.
- Create a new material + shader graph and in the
Asset Browser
panel. - In the shader graph, set the
Shader
node toGaussian Splatting
mode. - Use the material in the
Gaussian Splatting
component.
In Gaussian Splatting mode, there are two new inputs that can be overridden. They are applied in the Vertex shader stage.
Splat Center
: change the location of the splatGSplat Point Size
: specify the size of the splat, in pixels.
Examples for custom Gaussian Splatting materials
Colorize
Discard by world position
Point Cloud
Wavy Fruit
GSAF - Gaussian Splatting Animation Frames
In the next section we will address a feature of Lens studio that will allow the user to import a series of .ply files from a designated folder.
These files will be imported into Lens studio in such a manner that it will compress and provide a single GS file, that can be animated using a simple animation script.
Preparing the .ply folder for import
- The .ply frame sequence files will all need to be under the same folder.
- This folder MUST only contain ply files in order for Lens studio to initiate the GSAF Asset settings.
- PLY files naming convention should be in order sequentially to avoid frame jumps in the GSAF final output. For example:
Frame_001.ply
,Frame_002.ply
, etc.
Importing the GS files into Lens studio
-
Once the folder with the .ply files are sorted, in the
Asset Browser
panel, click on+
thenImport Asset
. -
Next select the first .ply file within the folder you want to import the .ply frame files.
-
Once Lens studio recognizes that the .ply is within a folder with only .ply files a new message-box will pop-up. Enable the checkbox that indicates to convert all .ply files from the same folder. (Checkbox should be enabled by default).
-
Below the checkbox there are 4 attributes for Gaussian Spaltt import and compression capabilities and you can see it will display each of these attributes with a range from 4 to 16 bit. Higher settings for these attributes will yield better visuals of the Gaussian splatt at a cost of the GSAF file size and import time.
You can also drag the .ply file directly into the asset browser to initiate the GSAF process of Lens Studio.
-
The import process should take a few minutes depending on the .ply files’ size, quantity and the attributes settings. And should be added to the asset browser once the process is complete.
-
Once importing and compression is done, you can inspect the newly added GSAF, the number of splatts and size that the compressed Gaussian Splatting asset file takes up inside the Lens. You will notice a new field which will indicate the total number of frames for this GSAF file as well:
Placing the GSAF file into the scene
- Placing the newly created GSAF is done by using the same Gaussian splatting component for a single ply file.
- Once added to the scene, an additional field to indicate the current active frame number for this Gaussian splatting animation file as seen on the image below.
Animating the newly imported GSAF file
Once the GSAF asset is placed, the way the animate it can be achieved using a dedicated script.
In your Asset Browser
panel, press the +
> JavaScript
to add a new Javascript file to your project. Double-click the file to open it, and paste the following code in the Script Editor
panel.
Next, select the object with your Gaussian Splat, press Add Component
in the Inspector
panel, and add your newly added script.
//@input Component.Text textComp
//@input float frameDuration {"widget":"slider", "min": 0.01, "max": 2, "default":0.333}
//@input int maxFrames {"min": -1}
//@input bool pingPong
var gsVisual = script.sceneObject.getComponent(
'Component.GaussianSplattingVisual'
);
function getFrameCount() {
var numFrames = gsVisual.asset.getNumberOfFrames();
if (script.maxFrames > 0) numFrames = Math.min(numFrames, script.maxFrames);
return numFrames;
}
var forward = true;
function nextFrame() {
var numFrames = getFrameCount();
var cur = gsVisual.activeFrame;
var next;
if (forward) {
if (cur < numFrames - 1) {
next = cur + 1;
} else if (script.pingPong) {
next = numFrames - 2;
forward = false;
} else {
next = 0;
}
} else {
if (cur > 0) {
next = cur - 1;
} else {
next = 1;
forward = true;
}
}
gsVisual.activeFrame = next;
next += 1; // UI should display 1-based index not 0-based
//print("frame := " + next);
if (script.textComp) {
var gsfps = Math.round((1.0 / script.frameDuration) * 100) / 100.0;
script.textComp.text =
'Frame #' + next + ' / ' + numFrames + '\n' + gsfps + ' fps';
}
}
var interval = script.frameDuration; // in seconds
var timeNextSwap = getTime() + interval;
script.createEvent('UpdateEvent').bind(function (eventData) {
var delta = getTime() - timeNextSwap;
if (delta >= 0) {
nextFrame();
timeNextSwap = getTime() + Math.max(0, interval - delta);
}
});
Once the script is added to the component, The component will look something, like this:
Take a look at the Scripting section to learn how to add scripts to your project.
Script parameters:
- Text Component: Shows the Current frame of the GSAF rendered (For debug purposes)
- Frame duration: Set the time in which every GSAF frame will be rendered. Lower duration means the entire GSAF will be rendered faster.
- Max Frames: Sets the number of frames to be rendered (Default: -1 which will rendered all of the GSAF’s frames)