Gameable Engine

ADR 0012: The GNM head reaches the browser as an aosRig BakedHead, baked to .aosrig ​

  • Status: Accepted — shipped in M5
  • Date: 2026-09-13
  • Plan decision: elaborates architecture decision 6 (ADR 0007)

Context ​

ADR 0007 put the rig stage behind RigBackend and said gnm would run as WGSL rather than ONNX. It did not say where the browser gets the model from, and that turned out to be the whole problem.

Google's GNM is a parametric head — 253 identity and 383 expression parameters, Apache-2.0 — and it exists as Python and .npz only. There is no ONNX export and there is not going to be one: after identity is chosen, the model is a linear expression basis plus linear-blend skinning, which is a matrix multiply and an LBS. Pushing that through an inference runtime would be slower than running it as a shader, and would drag a second execution provider into the frame.

F:\work\aos\aosRig already integrates GNM (a google/GNM submodule plus aosrig/head/{gnm_model,block,baked,attach,seam,identity}.py) and already exports a BakedHead npz: a numpy-only artefact carrying neutral (V,3), basis (E,V,3) in f16 or f32, faces, quads, uv, quad_uv, eye_positions (2,3), eye_weights (2,V), stitch_local, skin_index (V,4) and skin_weight (V,4). That is the handoff, and inventing a second one would mean re-deriving identity baking, the neck seam and the eye attachment in TypeScript.

Decision ​

BakedHead is the contract. packages/character/tools/gnm_pack.py converts one to a .aosrig container that src/rig/gnm/gnmPack.ts parses, and GnmRigBackend runs it as gnm_blend.wgsl sharing lbs_common.wgsl with ORL.

The script reads the npz keys directly rather than importing aosrig, so a pack is reproducible on a machine with numpy and nothing else. It does try from aosrig.head.baked import BakedHead for one purpose only: if it imports, the parameter layout and region names come from block.HeadLayout instead of the script's constants, so a change to the model's parameter block cannot silently bake a stale layout — and the header records which source was used.

Four conversions happen offline because the browser cannot do them cheaply at load:

  1. Transpose the basis to vertex-major (V,E,3). The bake is coefficient-major, which makes a GPU thread stride V*3 floats between its 383 reads.
  2. Quantise to fp16 with a per-coefficient scale, two halves to a u32 word: 78 MB → 39 MB. unpack2x16float is core WGSL, so no adapter feature is needed. The scale is max|B[c]| / 65504, clamped so a coefficient with tiny deltas keeps its exponent range.
  3. Remap the skinning to a compact joint list. The head touches 4–8 of the body rig's 54 joints; shipping 54 rows would waste most of a binding whose ceiling is what keeps the deform pipeline inside the default storage-buffer limit.
  4. Record the attachment as bindTransform. BakedHead.pose skins in the body's bind space, not head-local, so head-local has to travel head_bind_world @ local_to_head first.

The runtime input is head_ext: 387 floats — 383 expression coefficients in the order left_eye 100, right_eye 100, lower_face 150, tongue 32, pupils 1, then gaze [pitchL, yawL, pitchR, yawR] in radians. Output is 17,821 head-local vertices in metres, +Y up, +X subject-left, +Z out of the face. Joints are neck, head and the two eyes; there is no jaw, and neck and head rotation belong to the body's neck_01 / neck_02 / head.

Consequences ​

  • The browser never runs GNM; it runs a bake. Identity is therefore frozen per pack: a .aosrig is one character, and a new identity is a new bake.
  • Correctness is checkable against the source of truth. gnm_reference.py and gnmReference.ts are the same maths on CPU, and the GPU path is tested against a fixture generated by the Python original — vertices within 1e-3 cm of reference_frames.npz.
  • The expression space is gnm (387) or the reduced ML view gnm68 (68), and it is declared in the bundle rather than inferred. An ARKit-52 vector sent to a GNM bundle with no trained arkit_map is refused; the two spaces are unrelated, and a silent reinterpretation is a face that moves wrongly with nothing on screen to explain it.
  • GNM's decoders are a separate schedule. The rig stage is swappable now; a bundle whose geometry and appearance decoders were trained on ORL topology still uses orl, and nothing in a game changes when that flips.
  • The fp16 basis is a quality decision, not just a size one. 39 MB is resident per character; a future character that needs more fidelity than fp16 gives would need a second pack format, and the header carries a version for it.
  • One more Python dependency in the offline path, and a marked one: gnm_pack.py is a tool, not part of the browser build, and AGENTS.md rule 5 covers the pack it produces — .aosrig files are generated, and the generator is the thing to change.