@gameable/conversation

Interfaces

ActionTracks

A clip as the service sends it: parent-local rotations per bone, the hips' path.

Properties

bones

bones: Record<string, number[]>;

Per bone name, x y z w per frame.

fps

fps: number;

frames

frames: number;

prompt?

optional prompt?: string;

root?

optional root?: number[];

The hips' path, x y z per frame; absent for a clip that stands where it is.

space?

optional space?: string;

Whose joints the clip names, when the service or a carry says (soma: the studio's standard rig).

---

ConversationCharacter

Story configuration is host-owned; game code names only a character id.

Properties

prefabs?

optional prefabs?: Readonly<Record<string, readonly SpeechChunk[] | undefined>>;

This character's prepared performances, selected by the service.

storyId

storyId: string;

The story whose snapshots are this character's. Empty: the one the relay names in its hello.

url

url: string;

---

ConversationModule

Session control. Ending an interview retains its session and story revision.

Extends

• EngineModule

Properties

id

readonly id: string;

Unique id. Also the service id when init returns a service.

Inherited from

EngineModule.id

order?

readonly optional order?: number;

Sort key. Lower runs first; equal keys keep registration order.

Rough convention: input -100, gameplay -50, physics 0, rendering helpers 200. Gameplay runs before physics so that the commands a guest tick emits are simulated by the step that follows rather than the next one.

Inherited from

EngineModule.order

Methods

ask()

ask(text): boolean;

Parameters

Parameter	Type
text	string

Returns

boolean

beginFrame()?

optional beginFrame(): void;

Run before the frame's fixed steps. Input capture lives here.

Returns

void

Inherited from

EngineModule.beginFrame

dispose()

dispose(): void;

Release everything this module took.

Returns

void

Inherited from

EngineModule.dispose

end()

end(): void;

Returns

void

endFrame()?

optional endFrame(): void;

Run after rendering. Input's end-of-frame bookkeeping lives here.

Returns

void

Inherited from

EngineModule.endFrame

fixedUpdate()?

optional fixedUpdate(dt): void;

Run once per fixed step.

Parameters

Parameter	Type	Description
dt	number	Always ctx.config.fixedDt, whatever time.timeScale is; time scale changes how many steps a frame runs, not their length.

Returns

void

Inherited from

EngineModule.fixedUpdate

init()

init(ctx): void | object | Promise<void | object>;

Acquire resources.

Parameters

Parameter	Type	Description
ctx	HostContext	The host surface.

Returns

void | object | Promise<void | object>

Nothing, or the service to publish under id.

Inherited from

EngineModule.init

interrupt()

interrupt(): void;

Stop the character talking, here and at the service.

Returns

void

say()

say(text): boolean;

Have the character say this line word for word (the service's TextOut): a verdict the game decided, a scripted beat. It is spoken in the character's voice and kept in the conversation's history.

Parameters

Parameter	Type
text	string

Returns

boolean

False when there is no open session or the text is empty or too long.

start()

start(character, options?): void;

Open a conversation. A character met before resumes its session (its story where it was), unless fresh: then the service starts the story over from its first phase.

Parameters

Parameter	Type
character	string
options?	{ fresh?: boolean; }
options.fresh?	boolean

Returns

void

update()?

optional update(dt, alpha): void;

Run once per frame, after the fixed steps and before rendering.

Parameters

Parameter	Type	Description
dt	number	Clamped wall-clock seconds since the previous frame, scaled by time.timeScale.
alpha	number	Interpolation factor in [0, 1) for smoothing transforms.

Returns

void

Inherited from

EngineModule.update

---

ConversationOptions

Optional conversation dependencies and callbacks.

Properties

characters

characters: Readonly<Record<string, ConversationCharacter | undefined>>;

player

player: SpeechPlayer;

prefabs?

optional prefabs?: Readonly<Record<string, readonly SpeechChunk[] | undefined>>;

Prepared performances keyed by Convorcher prefab id; no story mutations.

Methods

connect()?

optional connect(url): ConversationSocket;

Parameters

Parameter	Type
url	string

Returns

ConversationSocket

onAction()?

optional onAction(clip): void;

A motion that was still being made when its sentence came (its action marked pending) has come, as a pre-made one, or will not (an ActionTracks message): the page plays it with its sentence as the studio does.

Parameters

Parameter	Type
clip	ServerClip

Returns

void

onError()

onError(message, detail?): void;

Something went wrong. retryable: the session is still open and the same question can be asked again (a reply that took too long, a service error); false: the session closed and start must be called again. A game must not count either as the player's attempt.

Parameters

Parameter	Type
message	string
detail?	{ retryable: boolean; }
detail.retryable?	boolean

Returns

void

onHello()?

optional onHello(data): void;

The session's hello (features.gesture_gen says whether the service sends its own action clips).

Parameters

Parameter	Type
data	Readonly<Record<string, unknown>>

Returns

void

onStatus()

onStatus(status): void;

Parameters

Parameter	Type
status	string

Returns

void

onStory()

onStory(character, snapshot): void;

Parameters

Parameter	Type
character	string
snapshot	StorySnapshot

Returns

void

---

ConversationSocket

Browser connection surface; injectable for service replay tests.

Properties

onclose

onclose: ((event) => void) | null;

onerror

onerror: ((event) => void) | null;

onmessage

onmessage: ((event) => void) | null;

onopen

onopen: ((event) => void) | null;

readyState

readyState: number;

Methods

close()

close(): void;

Returns

void

send()

send(text): void;

Parameters

Parameter	Type
text	string

Returns

void

---

FaceFrame

Face frames, with seconds relative to their audio chunk: 52 weights in ARKit-52 order, or, when the session asked the conversation service for the head model's own face (a2f=isaac-gnm), its 383 expression codes.

Properties

blendshapeWeights

blendshapeWeights: readonly number[];

timeCode

timeCode: number;

---

GnmVoice

The voice model's published information, which turns a native frame's codes into head coefficients: coefficient = scale x (code - offset), held within limit (see parseGnmVoice).

Properties

limit?

optional limit?: Float32Array<ArrayBufferLike>;

Per coefficient, GNM_CLAMP_STD x |std x scale| (std: the spread the voice model was trained with). Past it the head model's lips cross and the splats on them tear, so each coefficient is held inside. Absent when the file carries no std.

offset

offset: Float32Array;

scale

scale: Float32Array;

---

ScheduledSentence

A sentence as the player scheduled it: its loudness ahead of time, on the audio clock.

Properties

at

at: number;

When it starts, on clock.

clock

clock: () => number;

The audio clock the sentence is scheduled on.

Returns

number

levels

levels: Float32Array;

Its loudness, 0..1, one per step seconds (voiceLoudness).

sentenceNum

sentenceNum: number;

step

step: number;

Seconds per level (1/50).

text

text: string;

---

ServerAction

One action of a reply.

Properties

actionText

actionText: string;

The words the action came from.

id?

optional id?: string;

name

name: string;

pending?

optional pending?: boolean;

Its clip is still being made: it comes in an ActionTracks message with this id.

performance

performance: boolean;

A whole-body motion (a jump, a dance) rather than a small gesture under the talk.

tracks

tracks: ActionTracks | null;

The clip; null for a stock gesture, and for one still being made (pending).

---

ServerClip

What came of a motion that was still being made (an ActionTracks message).

Properties

action

action: ServerAction;

The action, with its clip when ready (tracks null otherwise).

id

id: string;

sentenceNum

sentenceNum: number;

status

status: "ready" | "preset" | "dropped";

---

SpeechChunk

One bounded speech performance, already decoded from the service wire format.

Properties

actions?

optional actions?: readonly ServerAction[];

The actions the service sent with this sentence (its own clips, or stock tags), as the studio reads them.

frames

frames: readonly FaceFrame[];

gesture?

optional gesture?: string;

A prepared performance's gesture, or the first action's name (the older road).

pcm

pcm: Uint8Array;

sentenceNum?

optional sentenceNum?: number;

The sentence's number in its reply (what an ActionTracks message names).

text

text: string;

---

SpeechPlayer

Audio-clock speech queue; update reuses a single facial vector.

Methods

clock()

clock(): number;

The audio clock the sentences are scheduled on, seconds.

Returns

number

dispose()

dispose(): void;

Returns

void

enqueue()

enqueue(chunk): void;

Parameters

Parameter	Type
chunk	SpeechChunk

Returns

void

level()

level(): number;

The voice's loudness right now, 0..1 (the studio's level meter).

Returns

number

levelAhead()

levelAhead(seconds): number;

The loudness seconds from now, 0..1: a sentence's whole loudness is scheduled before it plays.

Parameters

Parameter	Type
seconds	number

Returns

number

stop()

stop(): void;

Returns

void

update()

update(): void;

Returns

void

---

SpeechPlayerOptions

Options for createSpeechPlayer.

Properties

gnmVoice?

optional gnmVoice?: GnmVoice | (() => GnmVoice | null);

Take native frames: the parsed voice information (parseGnmVoice), or a getter while it loads (null then: the audio plays and the face rests). Without it a 383-wide chunk is refused, as it always was.

---

SpeechPresentation

Presentation callbacks run on the audio clock.

Methods

actions()?

optional actions(actions, sentenceNum): void;

The sentence's actions, the moment its voice starts (the studio plays a reply's clip as its sentence is spoken, never ahead of the words it goes with).

Parameters

Parameter	Type
actions	readonly ServerAction[]
sentenceNum	number

Returns

void

expression()

expression(weights, space): void;

The face now. space says what weights holds: 52 ARKit weights in [0, 1] (arkit52), or, from native frames, the head's 387 gnm coefficients (the character bridge takes both as they are: characters.setExpression(entity, space, weights)). A presentation that only drives ARKit may ignore space: a player made without gnmVoice never sends gnm.

Parameters

Parameter	Type
weights	Float32Array
space	SpeechFaceSpace

Returns

void

gesture()

gesture(id): void;

Parameters

Parameter	Type
id	string

Returns

void

reset()

reset(): void;

Returns

void

scheduled()?

optional scheduled(sentence): void;

A sentence was scheduled: its whole loudness and its words are known before it plays (the studio plans the talking gestures from the two at this moment).

Parameters

Parameter	Type
sentence	ScheduledSentence

Returns

void

speaking()

speaking(active): void;

Parameters

Parameter	Type
active	boolean

Returns

void

subtitle()

subtitle(text): void;

Parameters

Parameter	Type
text	string

Returns

void

---

StorySnapshot

Stable full-state message emitted by Convorcher after analysis and on join.

Properties

completedObjectives

completedObjectives: string[];

currentPhase

currentPhase: string;

playerState

playerState: Record<string, string>;

revision

revision: number;

schemaVersion

schemaVersion: 1;

sessionId

sessionId: string;

storyId

storyId: string;

transitionCause

transitionCause: string;

---

TalkCommand

A command from the game (conversation.command), as the engine adapter hands it on.

Properties

action

action: 
  | "start"
  | "end"
  | "ask"
  | "interrupt"
  | "microphone-on"
  | "microphone-off";

character

character: string;

entity

entity: number;

text

text: string;

---

TalkEvent

A conversation event for the game, as the host's event queue takes it.

Properties

tag

tag: "conversation-event";

val

val: object;

entity

entity: number;

kind

kind: "status" | "error" | "input" | "story" | "subtitle";

text

text: string;

---

TalkingCharacterBridge

The part of the page's character bridge a talking character uses (gameable/host/characters).

Methods

entryOf()?

optional entryOf(entity): 
  | {
  animator?:   | {
     gesture: {
        play: void;
        stop: void;
     };
   }
     | null;
  clips: readonly string[];
}
  | undefined;

Parameters

Parameter	Type
entity	number

Returns

| { animator?: | { gesture: { play: void; stop: void; }; } | null; clips: readonly string[]; } | undefined

setExpression()

setExpression(
   entity, 
   space, 
   weights
): void;

Parameters

Parameter	Type
entity	number
space	SpeechFaceSpace
weights	Float32Array

Returns

void

---

TalkingCharacters

The page's handle on its talking characters.

Methods

dispose()

dispose(): void;

Close the conversation, stop the microphone and remove the talk box.

Returns

void

handle()

handle(command): void;

Run one of the game's commands: conversation: (command) => talk.handle(command) in the engine adapter.

Parameters

Parameter	Type
command	TalkCommand

Returns

void

input()

input(text): void;

Give the game something the player said, as the talk box does (an input event).

Parameters

Parameter	Type
text	string

Returns

void

say()

say(text): boolean;

Have the open conversation's character say this line word for word.

Parameters

Parameter	Type
text	string

Returns

boolean

---

TalkingCharactersOptions

What talkingCharacters needs.

Properties

audio

audio: object;

The engine's audio module (engine.get('audio')): the voice plays on its voice bus.

buses

buses: object;

buses.voice

voice: AudioNode;

context

context: AudioContext;

characters?

optional characters?: TalkingCharacterBridge;

The page's character bridge, for the face and gestures. Without it the character only speaks.

connect?

optional connect?: (url) => ConversationSocket;

Open the socket another way (tests).

Parameters

Parameter	Type
url	string

Returns

ConversationSocket

names?

optional names?: Readonly<Record<string, string>>;

The name the talk box shows for each character id (elora: 'Elora'). An id not named here is shown as itself; the relay decides which ids exist (its src/<id>.story.yaml files).

panel?

optional panel?: boolean | HTMLElement;

The talk box: a subtitle line and a field to type (and a button to speak) into, shown while a conversation is open. true (the default) puts it at the bottom of the page; an element puts it in there; false leaves the page's own UI to call TalkingCharacters.input.

url?

optional url?: string | URL;

The relay, as a URL the page can reach. Default: talk/ under the page's own address.

Methods

emit()

emit(event): void;

Where events for the game go: usually (event) => adapter.events.push(event).

Parameters

Parameter	Type
event	TalkEvent

Returns

void

Type Aliases

SpeechFaceSpace

type SpeechFaceSpace = "arkit52" | "gnm";

What a face vector holds: ARKit-52 weights, or the head model's native gnm coefficients.

Variables

ARKIT_FRAME_WIDTH

const ARKIT_FRAME_WIDTH: 52 = 52;

ARKit weights in one frame.

---

GNM_CLAMP_STD

const GNM_CLAMP_STD: 3 = 3;

How many standard deviations of the voice model's training spread a coefficient may reach.

---

GNM_FRAME_WIDTH

const GNM_FRAME_WIDTH: 383 = 383;

The head model's expression codes in one native frame (a2f=isaac-gnm).

---

GNM_SPACE_DIM

const GNM_SPACE_DIM: 387 = 387;

The gnm expression space: the 383 coefficients, then a four-wide gaze tail the player leaves at zero.

---

LEVEL_FPS

const LEVEL_FPS: 50 = 50;

The loudness meter's rate: fifty levels a second (the studio's LEVEL_FPS).

---

LEVEL_STEP

const LEVEL_STEP: number;

Seconds per loudness level.

Functions

acceptStorySnapshot()

function acceptStorySnapshot(
   current, 
   next, 
   sessionId, 
   storyId
): StorySnapshot | null;

Idempotent reducer scoped to an expected session and story.

Parameters

Parameter	Type	Description
current	StorySnapshot | null	Last accepted snapshot.
next	StorySnapshot	Candidate snapshot.
sessionId	string	Expected session.
storyId	string	Expected story.

Returns

StorySnapshot | null

The accepted snapshot, or the previous state.

---

asActionTracks()

function asActionTracks(v): ActionTracks | null;

A clip as sent, or null when it is not one.

Parameters

Parameter	Type	Description
v	unknown	The tracks field.

Returns

ActionTracks | null

The clip, or null.

Example

import { asActionTracks } from 'gameable/conversation';
console.log(asActionTracks({ fps: 30, frames: 2, bones: { pelvis: [0, 0, 0, 1, 0, 0, 0, 1] } })?.frames); // 2

---

clipFitsRig()

function clipFitsRig(tracks, joints): boolean;

Does a clip address this rig's own joints, by their names? Half the clip's joints found is a fit; a clip that names none fits (nothing to judge by). The studio's clipFitsRig.

Parameters

Parameter	Type	Description
tracks	ActionTracks	The clip.
joints	readonly string[]	The rig's joint names.

Returns

boolean

Whether it fits.

Example

import { clipFitsRig } from 'gameable/conversation';
console.log(clipFitsRig({ fps: 30, frames: 2, bones: { pelvis: [] } }, ['Hips'])); // false

---

conversation()

function conversation(options): ConversationModule;

Build an optional session manager. Construction does not connect to any service.

Parameters

Parameter	Type	Description
options	ConversationOptions	Explicit character allow-list, playback and callbacks.

Returns

ConversationModule

An engine module. Call start only after an interview is requested.

---

createSpeechPlayer()

function createSpeechPlayer(
   context, 
   presentation, 
   destination?, 
   options?
): SpeechPlayer;

Build a bounded PCM player. The caller creates/resumes the context in a user gesture.

Frames 52 wide play as ARKit-52 weights. Frames 383 wide (the head model's own face, a2f=isaac-gnm) play as the head's gnm coefficients when options.gnmVoice is given: coefficient = scale x (code - offset), held within GNM_CLAMP_STD x |std x scale|, interpolated at the frames' own times and eased in and out at the ends of a run of speech; without it they are refused.

Parameters

Parameter	Type	Default value	Description
context	AudioContext	undefined	Shared or dedicated audio clock.
presentation	SpeechPresentation	undefined	Subtitles, gestures and facial rig.
destination	AudioNode	context.destination	Voice bus; defaults to the context destination.
options	SpeechPlayerOptions	{}	Native frames (gnmVoice).

Returns

SpeechPlayer

A player which owns only its sources, not the context.

---

faceSpaceOf()

function faceSpaceOf(width, native): SpeechFaceSpace | null;

The space a chunk's frames drive, from their width.

Parameters

Parameter	Type	Description
width	number	Numbers per frame.
native	boolean	Whether the player takes native frames (it was given gnmVoice).

Returns

SpeechFaceSpace | null

arkit52 for 52, gnm for 383 when native frames are taken; null for anything else (refused).

---

gnmCoefficients()

function gnmCoefficients(
   codes, 
   voice, 
   out?
): Float32Array;

One native frame's codes as head coefficients, each held within the voice's limit, with a zero gaze tail.

Parameters

Parameter	Type	Description
codes	ArrayLike<number>	The frame's 383 codes as the service sends them.
voice	GnmVoice	The voice information.
out	Float32Array	Output of length GNM_SPACE_DIM, reused.

Returns

Float32Array

out.

---

parseActions()

function parseActions(raw): ServerAction[];

data.animations of a TextOut, whatever came: never throws, drops what is not an action.

Parameters

Parameter	Type	Description
raw	unknown	The field.

Returns

ServerAction[]

The actions, in order.

Example

import { parseActions } from 'gameable/conversation';
console.log(parseActions([{ Name: 'Smile', ActionText: 'smiles' }])[0].name); // Smile

---

parseClip()

function parseClip(raw): ServerClip | null;

An ActionTracks message's data, whatever came: null when it is not one.

Parameters

Parameter	Type	Description
raw	unknown	The field.

Returns

ServerClip | null

The clip's arrival, or null.

Example

import { parseClip } from 'gameable/conversation';
console.log(parseClip({ id: 'm1', sentenceNum: 0, status: 'dropped' })?.status); // dropped

---

parseGnmVoice()

function parseGnmVoice(raw): GnmVoice | null;

Read a voice information file (gnm_voice.json: scale, offset and optionally std, 383 numbers each).

Parameters

Parameter	Type	Description
raw	unknown	The parsed JSON.

Returns

GnmVoice | null

The voice, with limit when the file carries a valid std; null when it cannot be used.

---

parseStorySnapshot()

function parseStorySnapshot(value): StorySnapshot | null;

Validate a bounded, versioned story snapshot.

Parameters

Parameter	Type	Description
value	unknown	Untrusted message data.

Returns

StorySnapshot | null

Owned snapshot, or null for invalid data.

---

serverActsLive()

function serverActsLive(hello): boolean;

Does this session's hello say the service finds the actions itself? Only a plain true counts.

Parameters

Parameter	Type	Description
hello	unknown	The session message's data.

Returns

boolean

Whether features.gesture_gen is true.

Example

import { serverActsLive } from 'gameable/conversation';
console.log(serverActsLive({ features: { gesture_gen: true } })); // true

---

talkingCharacters()

function talkingCharacters(options): TalkingCharacters;

Wire talking characters to the page and the game.

Parameters

Parameter	Type	Description
options	TalkingCharactersOptions	The audio, the character bridge, where game events go, and the characters' names.

Returns

TalkingCharacters

The handle the engine adapter's conversation option calls.

Example

import { talkingCharacters } from 'gameable/conversation';

declare const engine: { get(id: 'audio'): { context: AudioContext; buses: { voice: AudioNode } } };
declare const events: { tag: 'conversation-event'; val: { entity: number; kind: 'input' | 'status' | 'subtitle' | 'error' | 'story'; text: string } }[];

const talk = talkingCharacters({
  audio: engine.get('audio'),
  emit: (event) => events.push(event),
  names: { elora: 'Elora' },
});
// createEngineAdapter(engine, { ..., conversation: (command) => talk.handle(command) })

---

toPcm16()

function toPcm16(
   chunks, 
   samples, 
   rate
): Uint8Array;

Float samples at rate to 16 kHz mono little-endian PCM16.

Parameters

Parameter	Type	Description
chunks	readonly Float32Array<ArrayBufferLike>[]	The recorded blocks.
samples	number	How many samples they hold.
rate	number	Their sample rate.

Returns

Uint8Array

PCM16 bytes.

---

travels()

function travels(tracks, min?): boolean;

Does the clip go somewhere? (start to end over the floor, metres): the studio's travels.

Parameters

Parameter	Type	Default value	Description
tracks	ActionTracks	undefined	The clip.
min	number	0.6	Metres that count as travel.

Returns

boolean

Whether it travels.

Example

import { travels } from 'gameable/conversation';
console.log(travels({ fps: 30, frames: 2, bones: { Hips: [] }, root: [0, 1, 0, 2, 1, 0] })); // true

---

voiceLoudness()

function voiceLoudness(pcm): Float32Array;

A sentence's loudness from its sound, fifty a second (the studio's loudness): the RMS of each 20 ms, on the meter's own scale (0 at a whisper's floor, 1 at a raised voice).

Parameters

Parameter	Type	Description
pcm	Uint8Array	Mono PCM16 at 16 kHz, little-endian.

Returns

Float32Array

One level per 20 ms, 0..1.

Example

import { voiceLoudness } from 'gameable/conversation';
console.log(voiceLoudness(new Uint8Array(3200)).length); // 5