Live analysis

Send samples you hold (a microphone, a telephony bridge, a file) over the WebSocket. Audio already in a LiveKit room is analyzed by the agent worker, which publishes events into the room. See LiveKit.

Realtime WebSocket

const session = prosody.stream.session({
  encoding: 'pcm16',
  sampleRate: 16_000,
  source: 'microphone',
  onUpdate: (s) => render(s.snapshot()),
});

await session.start();     // resolves on config_ack
session.send(pcmChunk);    // Int16Array | Uint8Array | ArrayBuffer | Buffer
await session.stop();      // asks for session_end, then closes

snapshot() returns the same Transcription that transcribe returns: turns, speakers, and prosody, rebuilt from everything received so far. The same rendering code works on both surfaces.

Audio format

The server buffers incoming chunks and analyzes them on a one-second cadence.

Pacing a file

When you replay faster than real time, wait for each analysis second to be acknowledged instead of flooding the socket:

for (const chunk of chunks) {
  session.send(chunk);
  await session.waitForFrame();
}

Events

speaker_update carries the model's committed lane for an analyzed interval. LiveSession applies it to its Conversation, so snapshot() returns the current attributed turn set.

If you want the raw events without the conversation spine, prosody.stream.connect hands you each event as it arrives. Build a VoiceFrame from a directive to read the measured state, the movement, and the affect:

import { VoiceFrame } from '@prosodyai/sdk';

const stream = prosody.stream.connect(
  {
    onDirective: (event) => {
      const frame = VoiceFrame.fromDirective(event);
      log(frame.speakerId, frame.state.intonation.pitch, frame.vad);
    },
    onTranscriptUpdate: (event) => log(event.segments),
  },
  { encoding: 'pcm16', sampleRate: 16_000 },
);

await stream.connect();
stream.sendAudio(pcmChunk);
stream.end();

LiveKit

For audio already in a LiveKit room, use the livekit-plugins-prosodyai plugin. It owns the full-duplex speech loop, speaker lanes, identity, and saved conversation context on the agent worker, and publishes typed events into the room. See LiveKit plugin for install, the realtime model, and the committed event stream.