Browser SDK

The browser SDK connects a web page to a voice. It sets up the audio, tries WebRTC first and falls back to a WebSocket by itself, and gives you events to drive your page. It has no runtime dependencies.

The SDK is proprietary and shared privately. Install the package you received from the Kheelona team, for example npm install ./kheelona-voice-<version>.tgz. It is not on a public registry. Contact the Kheelona team for access. The browser quickstart runs the whole example. Every call is listed in the SDK reference.

The flow

Your server creates a session with your API key and sends the page the session ID, the client secret and the connection addresses. Never send the key. See Sessions.
The page calls KheelonaVoice.connect(...) from a click.
The page listens to the session’s events.

Here is the example page doing steps 1 and 2. Call connect from a click or a tap: the browser starts the audio and the microphone only after a user gesture.

async function connectToVoice() {
  // Your server creates the session with your API key. The page only ever gets a single-use client secret.
  const res = await fetch('/api/voice-session', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: '{}',
  });
  if (!res.ok) {
    const body = await res.json().catch(() => ({}));
    const retryAfter = Number(res.headers.get('Retry-After'));
    throw new MintError(body.error?.code ?? 'mint_failed', res.status, retryAfter > 0 ? retryAfter * 1000 : undefined);
  }
  const { sessionId, clientSecret, endpoints } = await res.json();
  // Call this from a click: the browser starts the audio and the microphone only after a user gesture.
  // A failed connect needs a new session (a new secret), so every attempt mints again.
  return KheelonaVoice.connect({ sessionId, clientSecret, endpoints, workletBaseUrl: '/sdk/' });
}

connect returns a session once the voice is ready. If it fails, it rejects with a KheelonaVoiceError. A failed connect needs a new session: create a new one on your server and try again. The client secret may be spent or expired. The one exception is capacity_exhausted with a retryAfterMs: you may retry with the same secret after that wait, while the secret is under 60 seconds old. When the wait is a second or two (one server is busy), connect already retries by itself, up to 3 times, before it rejects.

Options

These are the options of connect. The full list is in the ConnectOptions reference.

Option	What it does
`sessionId`, `clientSecret`	Required. From your server’s `POST /v1/sessions` answer. The secret is never your API key.
`endpoints`	`{ webrtcUrl, websocketUrl }`: the `connect` addresses from the same answer. Without it the SDK uses the production service.
`transport`	`auto` (the default), `webrtc` or `websocket`.
`microphone`	`{ deviceId }` to choose an input device, or `false` for typed turns only: no microphone is opened.
`workletBaseUrl`	Where the audio worklet files are served from. See Content Security Policy.

Transports

With auto the SDK tries WebRTC first. If WebRTC cannot connect, it opens a WebSocket with the same client secret. This happens when:

the browser has no WebRTC, or it fails to set up the connection;
the network fails before the connection is up;
the connection is not ready within 3 seconds.

It does not happen when the service answers with an error (an HTTP error answer rejects connect with that error), and it does not happen after the connection has opened. Both transports give you the same events. After connect, session.transport is webrtc or websocket.

transport: 'webrtc' or 'websocket' forces one and never falls back. The WebSocket path loads two small audio worklets.

Subscribe after `await connect()`

Attach your listeners right after await connect(), as the example does:

function listen(voice) {
  // Attach after `await connect()`: nothing is missed, and `state` first gives the current state.
  voice.on('state', (state) => {
    $('state').textContent = state;
  });
  voice.on('transcript', ({ text }) => addCaption('You', text));
  voice.on('replySentence', ({ text }) => addCaption('Voice', text));
  voice.on('error', (e) => {
    if (e.code === 'audio_unavailable') {
      // The browser refused to play the voice: ask for a tap (see the "Tap to allow sound" button).
      $('allow-sound').hidden = false;
    } else {
      showError(e);
    }
  });
  voice.on('ended', () => finish());
}

Nothing is missed. Events that arrived while connect was running, including the ready event, are delivered to you in a moment after it resolves. A new state listener first gets the current state, once, and then every change. At that point session.state is listening. on returns a function that removes the listener.

Events

Event	When	Payload
`state`	The session changes state: `listening`, `thinking`, `speaking`. It is `ended` at the end.	the state
`transcript`	What the person said, once the turn is understood.	`turn`, `text`, `language`, `languageStage`, `languageBeta`
`replySentence`	One sentence of the reply, as it starts to play.	`replyId`, `index`, `text`, `emotion`
`interrupted`	The reply was cut off.	`replyId`, `heardMs`
`safety`	A safety rule acted on a turn. See Safety levels.	`turn`, `stage`, `category`, `action`
`warning`	The session will end soon.	`code` (`max_duration_soon` or `idle_soon`), `secondsLeft`
`error`	A problem. If `fatal` is false the session goes on.	`code`, `message`, `retryable`, `fatal`
`ended`	The session is over. Fires once.	`reason`, `durationMs`, `turns`
`event`	Every event from the service, as received.	the raw event

Captions and transcripts are for your page to show. Kheelona does not keep them. See Captions. The names and types are in the SessionEvents reference.

Mute and interrupt

mute(true) stops sending the microphone. mute(false) starts again. While you are muted, the idle timer is paused. interrupt() stops the voice now, as if the person had started to talk. While the voice is speaking, the reply is cut and you get interrupted. When it is not speaking, nothing is cut and you get no interrupted event. Use it for a Stop button or a push-to-talk page.

/** Wires a mute toggle and an interrupt button. `getSession` returns the live session, or null when there is none. */
export function wireControls(getSession, { muteButton, interruptButton }) {
  muteButton.addEventListener('click', () => {
    const session = getSession();
    if (!session) return;
    // The button holds the state: aria-pressed is "true" while the microphone is muted.
    const muted = muteButton.getAttribute('aria-pressed') !== 'true';
    session.mute(muted);
    muteButton.setAttribute('aria-pressed', String(muted));
    muteButton.textContent = muted ? 'Unmute' : 'Mute';
  });
  interruptButton.addEventListener('click', () => {
    // Stops the voice now, as if the person had started speaking.
    getSession()?.interrupt();
  });
}

Typed turns, loudness and ending

Three more calls have no snippet here: they are one line each. See the reference for each.

sendText(text) sends a typed turn of at most 2,000 characters. It goes through the same safety checks. On a child voice, what is said aloud is checked a little more strictly than typed text, because speech recognition can change the words. A typed turn ends whatever the voice is doing. Longer text throws a RangeError. With microphone: false it is the only way to talk.
levels() returns { input, output }, each from 0 to 1, for a meter. Call it from requestAnimationFrame.
end() ends the session and releases the microphone. It returns a promise that resolves when the service has confirmed, or after 5 seconds at most. Call it when your page closes the conversation. A session also ends by itself at its time limit or after a long silence.

SDK version

The SDK has its own version number, which is the number in the package file’s name (kheelona-voice-<version>.tgz). It follows semantic versioning: a major version can change how your page calls the SDK. The changelog lists new versions. Test your page after you update.

No sound

If the browser refuses to play the voice, for example because connect was not called from a click, the SDK sends a non-fatal error event with the code audio_unavailable. The session goes on, and captions still work. The example page shows a Tap to allow sound button. When the person taps it, call session.resumeAudio() synchronously in the click handler, with no await before it. The example does it like this:

// A tap is the gesture the browser needs: call resumeAudio() directly in the handler, with no await before it.
$('allow-sound').addEventListener('click', () => {
  session?.resumeAudio();
  $('allow-sound').hidden = true;
});

The call resumes the audio and tries to play the voice again inside the tap. If the browser still refuses, you get audio_unavailable again.

Content Security Policy

If your page has a Content Security Policy, three things matter:

Connections. connect-src must allow the origins of the connect URLs your server got, for example https://api.voice.kheelona.com and wss://api.voice.kheelona.com.
Audio worklets on the WebSocket path. The SDK loads two worklet files. Either host them yourself, from your own origin, and pass workletBaseUrl: the files are capture.worklet.js and playback.worklet.js in the SDK package’s dist folder, and the example does this with workletBaseUrl: '/sdk/'. Or allow script-src blob:, and the SDK loads them from its own bundle. If a worklet cannot load, you get audio_unavailable rather than a hang.
Scripts and styles. The example needs only 'self' for both. It runs with no inline script and no inline style.

The complete policy of the example is on the Security page.

Errors

connect rejects, and the error event carries, a code and two flags:

code: branch on this, never on the message. The error list has every code. The SDK adds four of its own: microphone_unavailable (missing, blocked or in use), audio_unavailable (see above), network (the network failed) and transport_failed (neither transport could be set up).
retryable: trying the same thing again later can work.
retryAfterMs: when the service said how long to wait, in milliseconds. It is set only when connect fails on the WebRTC path.

Do not log error.cause as it is. When a browser’s own error is kept as the cause, its text can contain connection details. Log the code and the message; both are safe.

A connection that drops after it was up ends the session: you get an error with the code network and fatal: true, then ended with the reason disconnected. Start a new session to continue. The new one has no memory of the old one.

Supported browsers

The SDK is built for the current versions of Chrome, Edge, Firefox and Safari, on desktop and on phones. We test it in Chromium, Firefox and WebKit. It needs:

getUserMedia, which browsers allow only on https pages and on localhost;
AudioWorklet;
WebRTC, or a WebSocket for the fallback.

Safari can refuse to play the voice when microphone is false: treat audio_unavailable as above.