Splat characters

A character is a splat avatar produced by the AvatarOS pipeline. It is not a mesh with a texture and it is not a point cloud: it is a few hundred thousand gaussians whose centres, covariances and colours are recomputed on the GPU every time the face or the camera moves.

Status

Implemented in milestone M3. @aosengine/character owns the rig, the decoders and the lift; @aosengine/splat owns the buffers and the sort.

The pipeline

Five stages, once per branch, all on the GPU after the first one:

expression weights            setExpression / setRig
        |
        v
  [ rig backend ]             ORL (wasm) or GNM (WGSL) -> posed vertices
        |                     packages/character/src/rig/
        v
  [ vert transform ]          vertices -> the decoders' training frame
        |                     src/wgsl/vert_transform.wgsl
        v
  [ decoders ]                ONNX: rig -> trunk -> geom_uv, (code, view) -> color_uv
        |                     src/decoder/, onnxruntime-web on the shared device
        v
  [ lift, two passes ]        UV texels -> gaussians
        |                     pass 1: jacobian, triangle bounds, scale, cull
        |                     pass 2: eigen-decompose, rotate, Sigma = M*M^T, pack
        v
  SplatSink.buffers           written at this branch's slot offset

The "branches" are the parts the exporter split the avatar into — head, eyes, and on a clothed bundle top, bottom, hair and so on. Every branch has its own decoders and its own slot range, but they all write into one AnimatedSplat, because gaussians only sort correctly against gaussians in the same object. A character that rendered its hair in a second splat object would show the hair in front of or behind the whole head, never through it.

What the lift actually computes

The geometry decoder emits a UV map whose texels are positions on the avatar's surface. Pass 1 differentiates that map to get a local frame (the jacobian), bounds each texel against the triangle it came from, applies the scale clamps, and folds the validity test into the triim value so the pass stays inside eight storage buffers. Pass 2 eigen-decomposes the frame, builds M = R(q) · diag(s · worldScale), and writes Σ = M · Mᵀ as its upper triangle. Colour is rgb → u8; opacity is a logit, so it is written as u8(255 · sigmoid(x)). A culled texel is written with alpha 0, which is invisible — the slot keeps its place rather than being compacted away.

That last point is the one non-obvious design rule: slots are fixed, never compacted. The sort builds an index → splat mapping and reuses it across frames; if a cull removed a gaussian and shifted its neighbours down, every subsequent frame would draw a different avatar than the one that was sorted. Culling changes alpha, not addresses.

The device order is load-bearing

There is exactly one GPUDevice and everything borrows it. The order these calls happen in is not a style preference:

initWebGPUPatches(); // @aosengine/core, before the renderer exists
await renderer.init(); // -> renderer.backend.device
const device = prepareLiftDevice(renderer);
attachOrtDevice(device); // ort.env.webgpu.device = device
// ...only now may the first ONNX session be created...

onnxruntime-web caches its device on the first WebGPU session create. Setting env.webgpu.device after that is silently ignored, ORT builds a device of its own, and every copy between a decoder output and a lift buffer throws Buffer is associated with [Device]. attachOrtDevice therefore reports whether it was in time (OrtDeviceAttachment.shared) instead of assuming; when it was not, the pipeline falls back to DownloadOutputs — correct, but it round-trips every decoder output through the CPU.

createCharacter does all of this for you. The exported functions exist for a host that wants to attach the device before it loads any character.

WebGPU only

Characters need compute shaders and a shared device. On the WebGL fallback backend there is neither, and there is no CPU path worth demoting to, so createCharacter rejects with CharacterUnsupportedError. EngineCaps.characters tracks this exactly; check it before calling, and show the user something else rather than catching the error.

prepareLiftDevice also asserts maxStorageBuffersPerShaderStage >= 8. Eight is WebGPU's default limit, which every adapter grants, and the lift's heaviest pass binds exactly eight.

The bundle

A bundle is a directory of files the exporter produced. loadCharacterBundle reads it:

const bundle = await loadCharacterBundle('/assets/characters/myra');

It takes a URL by default but knows nothing about URLs: pass a resolver and the same loader runs against an asset registry, an in-memory map, or the Asset Manager adapter.

Two on-disk layouts are supported and produce the same object:

Layout	Marker	Branches
multi_region	scene.json, mode field	as many as order lists
schema_version: 1	export_manifest.json, no scene	one, fused, adapted to look alike

A multi_region scene.json names, per branch, its decoders (geom, appr, optional trunk), its mesh (mesh_bin + mesh_json, or mesh_glb), its UV resolution and its lift tuning. The eyes branch additionally carries its own static geometry and is gaze_conditioned.

The two blocks the engine adds

The exporter's schema is left byte-for-byte alone — a shipped bundle must keep parsing. The engine adds two optional blocks to scene.json:

{
  "rig": {
    "backend": "gnm",
    "pack": "myra.aosrig",
    "control_names": ["..."],
    "vertex_count": 17821
  },
  "expression_space": {
    "kind": "gnm",
    "dim": 387,
    "segments": [{ "name": "left_eye", "start": 0, "count": 100 }]
  }
}

rig states which rig posed this face, because that is the first question asked of a wrong-looking one and the answer must not depend on a URL flag. expression_space states what the animation layer is supposed to send, because two spaces that happen to share a width are otherwise indistinguishable.

Every bundle shipped so far declares neither, so both have defaults: rig.backend: "orl" with the bundle's own rig_names.json, and an arkit52 expression space mapped through the rig gather. Those defaults are what the prototype actually ran.

Rig backends

The rig stage is pluggable behind RigBackend — init, setControls, encode(encoder), vertsBuffer, dispose. Two implementations ship:

orl — OpenRigLogic. The character's own DNA, evaluated by a vendored wasm module, then int8 blendshape deltas and linear-blend skinning on the GPU. It is the exact rig the avatar was authored with, so it is correct by construction, and it is per-character: the pack ships inside the bundle. Its neutral is in centimetres. A small branch (under 4096 vertices — hair, a garment) uses the cheaper shell mode instead of full skinning.

gnm — a baked parametric head. A linear expression basis plus LBS, packed offline into one .aosrig file and run by gnm_blend.wgsl. There is no ONNX export and there will not be one: the model is a matrix multiply and an LBS, and pushing that through an inference runtime would be slower than the shader. 17,821 vertices, metres, 383 expression coefficients plus four gaze angles. gnmReference.ts is a CPU implementation of the same maths, and the GPU path is tested against a fixture generated by the python original.

Both share lbs_common.wgsl, so "the skinning differs between backends" is not a bug that can exist.

Expression spaces

kind	dim	What you send
arkit52	52	the ARKit blendshape weights, in ARKIT_NAMES order
gnm	387	head_ext: 383 expression coefficients + [pitchL, yawL, pitchR, yawR] in radians
gnm68	68	the reduced ML view: 64 truncated coefficients + the same four gaze angles

setExpression takes whichever the bundle declares. setRig bypasses it and drives the bundle's own control vector directly — use it when you are authoring against one specific character, not when you are playing back animation.

Sending an ARKit vector to a GNM bundle that carries no trained arkit_map is refused, loudly. The two spaces are unrelated; a silent reinterpretation is a face that moves wrongly with nothing on screen to explain it.

Rig preview / debug lift

The rig stage is ready long before a character is: a .aosrig pack or an ORL DNA poses vertices on its own, while the geometry and appearance decoders are trained per character and per topology. For GNM they do not exist yet at all — they are being retrained on GNM topology by another team — so between those two dates the rig has nothing to render into and a rig bug has nowhere to show itself.

DebugVertexLift closes that gap. It is a second consumer of RigBackend.vertsBuffer that writes one small isotropic gaussian per rig vertex into a SplatSink slot range:

import { createRigPreview, GnmRigBackend, jointTint } from '@aosengine/character';

const preview = await createRigPreview({
  renderer,
  sink: splat,
  backend: new GnmRigBackend({ packFile: 'myra_head.aosrig' }),
  getBytes: (name) => files.get(name),
  sigma: 0.002, // metres
});

preview.setControls(headExt);
preview.render(); // rig pass + debug lift, one encoder, one submission

It binds the same four sink buffers in the same order as lift_pass2_cov.wgsl, writes the covariance in the same writeCovariance split, and addresses slots the same way (slot_offset + i, fixed for life) — so a preview exercises the real buffer contract rather than a parallel one that could agree with nothing. What it does not do is appearance: every gaussian is a sphere of fixed radius and flat colour, because anything that looked like a face would hide the thing being debugged.

Colour comes from the CPU, one packed RGBA per vertex, uploaded once:

Tint	Reads	What it shows
jointTint	the skinning	a vertex weighted to the wrong joint; eyes are called out in cyan
uvTint	uv	a seam in the wrong place, a flipped island, a missing uv blob
heightTint	the vertices	nothing but geometry — works for a rig this package cannot parse
'normal'	the vertices	a radial pseudo-normal, for reading curvature without a palette

The bounds must describe vertsBuffer, not the model. GNM keeps its neutral head-local so the expression basis and the gaze rotation stay in the frame they are defined in, but the shader emits vertices in the body's bind space — bindTransform is folded into the skin matrices. On the shipped head that is a metre of difference, and a bounding sphere placed by the model's own bounds puts the sort's depth bins a metre away from the gaussians, which renders exactly nothing.

examples/character-showcase is the whole path as a running app, including a ?selftest=1 that reads vertsBuffer back once and compares it against gnmPose on the CPU over the ten recorded reference frames.

Per frame

update(dt, camera) is cheap when nothing changed. It splits the work in two:

• geometry — rig, vert transform, trunk and geom decode, lift. Runs when the expression, the pose or the gaze changed.
• appearance — plücker rays and the appr decode, against the geometry pass's cached code. Runs when the camera moved, which is most frames.

A camera that moved less than 2 mm (plus a rotation and fov term) re-runs nothing at all, because a pass is a full decode plus a lift. Passes are coalesced latestOnly: while one is in flight, further calls collapse to the most recent state rather than queueing.

Call sink.markGaussiansChanged() once per frame after the compute pass — without it a moving avatar seen from a still camera renders in a stale depth order.

Host integration

Game logic never touches any of the above. It says character.spawn(entity, 'char.guide'), character.setExpression(entity, 'arkit52', weights) and character.lookAt(entity, point) — six commands in the WIT vocabulary — and the host decides what, if anything, they draw.

The thing that decides is the character bridge, createCharacterBridge from @aosengine/wasm-host/characters. It is a separate entry point rather than part of the package's index on purpose: @aosengine/character brings onnxruntime-web and @aosengine/splat brings the animated splat fork, and a game with no characters must not download either. engineAdapter.ts names only the CharacterBridge type, which disappears at compile time, so importing the module is the opt-in:

import { createEngineAdapter } from '@aosengine/wasm-host';
import { createCharacterBridge } from '@aosengine/wasm-host/characters';

const characters = createCharacterBridge({ engine, renderer: engine.renderer });
const adapter = createEngineAdapter(engine, { modules, characters });

What each command becomes:

Command	Becomes
spawn-character	a rig backend, an AnimatedSplat and an Animator, under the entity
set-character-state	animator.setState — locomotion blends from the velocity, not the name
set-clip-weights	CharacterState.clips; inert until the character has a clip library
set-expression	the animator's face layer, mapped into the bundle's space
look-at	CharacterState.lookAt — head aim into joints, the residual into gaze
say	a log line; speech and visemes are not implemented
despawn	dispose the preview, the backend, the sink and the animator

On spawn-character the bridge resolves the manifest entry and picks a path from it:

• rig.backend === 'gnm' or rig.pack is set — the rig path. Fetch the .aosrig, build a GnmRigBackend and a createRigPreview over a per-character AnimatedSplat, parented under the entity's own Object3D. This is the only path that runs today, because GNM decoders do not exist yet, and it renders the debug point cloud described above.
• a bundle with scene.json branches — loadCharacterBundle plus createCharacter, the real thing. Guarded behind decoders: true and reached through a dynamic import, because deciding costs a fetch against a directory that no shipped bundle has.
• neither — the placeholder capsule stays, with one warning naming what to add.

Each rendered frame the adapter's update(dt, alpha) interpolates transforms and then, per character: animator.update(dt); the animator's expression vector is copied into the backend's control space; animator.gaze is added into the four gaze channels at the tail, negating pitch because GNM spells it positive-down and the animator positive-up; animator.jointOverrides go to setJointOverrides in the rig's (w, x, y, z) order; then preview.render().

Two failure modes are handled rather than thrown. On the WebGL fallback (engine.ctx.caps.characters === false) the bridge warns once and every character keeps its placeholder. A pack that 404s, a bundle that will not parse or a device that refuses the lift's storage-buffer limit all warn with the URL and the cause, and the capsule stays — a missing head is never a black screen or an exception.

Give an NPC a face is the whole thing as a recipe.

Memory

memoryReport() returns resident bundle bytes, approximate GPU bytes and slot counts, broken down per branch. The two numbers that dominate are the decoders (tens of MB of ONNX, halved by the fp16 siblings when the bundle ships them) and the splat capacity itself, at 52 bytes per gaussian, fixed at allocation.

See also

• Gaussian splats
• Animation
• Load a character
• Preview a rig without decoders
• Give an NPC a face
• Load a character from the Asset Manager
• packages/character/README.md — @aosengine/character