# Browser SDK

> Connect a web page to a Kheelona voice. Options, transports, events, controls, security policy and errors.

Source: https://docs.voice.kheelona.com/guides/browser-sdk/

The browser SDK connects a web page to a voice. It sets up the audio, tries WebRTC first and falls back to a
WebSocket by itself, and gives you events to drive your page. It has no runtime dependencies.

The SDK is proprietary and shared privately. Install the package you received from the Kheelona team, for example
`npm install ./kheelona-voice-<version>.tgz`. It is not on a public registry. Contact the Kheelona team for access.
The [browser quickstart](https://docs.voice.kheelona.com/get-started/quickstart-browser.md) runs the whole example. Every call is listed in the
[SDK reference](https://docs.voice.kheelona.com/reference/sdk/classes/kheelonavoice.md).

## The flow

1. Your **server** creates a session with your API key and sends the page the session ID, the client secret and the
   connection addresses. Never send the key. See [Sessions](https://docs.voice.kheelona.com/concepts/sessions.md).
2. The page calls `KheelonaVoice.connect(...)` from a click.
3. The page listens to the session's events.

Here is the example page doing steps 1 and 2. Call `connect` from a click or a tap: the browser starts the audio and
the microphone only after a user gesture.

```js title="public/main.js"
async function connectToVoice() {
  // Your server creates the session with your API key. The page only ever gets a single-use client secret.
  const res = await fetch('/api/voice-session', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: '{}',
  });
  if (!res.ok) {
    const body = await res.json().catch(() => ({}));
    const retryAfter = Number(res.headers.get('Retry-After'));
    throw new MintError(body.error?.code ?? 'mint_failed', res.status, retryAfter > 0 ? retryAfter * 1000 : undefined);
  }
  const { sessionId, clientSecret, endpoints } = await res.json();
  // Call this from a click: the browser starts the audio and the microphone only after a user gesture.
  // A failed connect needs a new session (a new secret), so every attempt mints again.
  return KheelonaVoice.connect({ sessionId, clientSecret, endpoints, workletBaseUrl: '/sdk/' });
}
```

`connect` returns a session once the voice is ready. If it fails, it rejects with a `KheelonaVoiceError`. **A failed
`connect` needs a new session**: create a new one on your server and try again. The client secret may be spent or
expired. The one exception is `capacity_exhausted` with a `retryAfterMs`: you may retry with the same secret after that
wait, while the secret is under 60 seconds old. When the wait is a second or two (one server is busy), `connect` already
retries by itself, up to 3 times, before it rejects.

## Options

These are the options of `connect`. The full list is in the
[ConnectOptions reference](https://docs.voice.kheelona.com/reference/sdk/interfaces/connectoptions.md).

| Option | What it does |
|---|---|
| `sessionId`, `clientSecret` | Required. From your server's `POST /v1/sessions` answer. The secret is never your API key. |
| `endpoints` | `{ webrtcUrl, websocketUrl }`: the `connect` addresses from the same answer. Without it the SDK uses the production service. |
| `transport` | `auto` (the default), `webrtc` or `websocket`. |
| `microphone` | `{ deviceId }` to choose an input device, or `false` for typed turns only: no microphone is opened. |
| `workletBaseUrl` | Where the audio worklet files are served from. See [Content Security Policy](#content-security-policy). |

## Transports

With `auto` the SDK tries **WebRTC** first. If WebRTC cannot connect, it opens a **WebSocket** with the same client
secret. This happens when:

- the browser has no WebRTC, or it fails to set up the connection;
- the network fails before the connection is up;
- the connection is not ready within **3 seconds**.

It does not happen when the service answers with an error (an HTTP error answer rejects `connect` with that error), and
it does not happen after the connection has opened. Both transports give you the same events. After `connect`,
`session.transport` is `webrtc` or `websocket`.

`transport: 'webrtc'` or `'websocket'` forces one and never falls back. The WebSocket path loads two small audio worklets.

## Subscribe after `await connect()`

Attach your listeners right after `await connect()`, as the example does:

```js title="public/main.js"
function listen(voice) {
  // Attach after `await connect()`: nothing is missed, and `state` first gives the current state.
  voice.on('state', (state) => {
    $('state').textContent = state;
  });
  voice.on('transcript', ({ text }) => addCaption('You', text));
  voice.on('replySentence', ({ text }) => addCaption('Voice', text));
  voice.on('error', (e) => {
    if (e.code === 'audio_unavailable') {
      // The browser refused to play the voice: ask for a tap (see the "Tap to allow sound" button).
      $('allow-sound').hidden = false;
    } else {
      showError(e);
    }
  });
  voice.on('ended', () => finish());
}
```

Nothing is missed. Events that arrived while `connect` was running, including the ready event, are delivered to you in a
moment after it resolves. A new `state` listener first gets the current state, once, and then every change. At that point
`session.state` is `listening`. `on` returns a function that removes the listener.

## Events

| Event | When | Payload |
|---|---|---|
| `state` | The session changes state: `listening`, `thinking`, `speaking`. It is `ended` at the end. | the state |
| `transcript` | What the person said, once the turn is understood. | `turn`, `text`, `language`, `languageStage`, `languageBeta` |
| `replySentence` | One sentence of the reply, as it starts to play. | `replyId`, `index`, `text`, `emotion` |
| `interrupted` | The reply was cut off. | `replyId`, `heardMs` |
| `safety` | A safety rule acted on a turn. See [Safety levels](https://docs.voice.kheelona.com/concepts/safety-levels.md). | `turn`, `stage`, `category`, `action` |
| `warning` | The session will end soon. | `code` (`max_duration_soon` or `idle_soon`), `secondsLeft` |
| `error` | A problem. If `fatal` is false the session goes on. | `code`, `message`, `retryable`, `fatal` |
| `ended` | The session is over. Fires once. | `reason`, `durationMs`, `turns` |
| `event` | Every event from the service, as received. | the raw event |

Captions and transcripts are for your page to show. Kheelona does not keep them. See [Captions](https://docs.voice.kheelona.com/guides/captions.md).
The names and types are in the [SessionEvents reference](https://docs.voice.kheelona.com/reference/sdk/interfaces/sessionevents.md).

## Mute and interrupt

`mute(true)` stops sending the microphone. `mute(false)` starts again. While you are muted, the idle timer is paused.
`interrupt()` stops the voice now, as if the person had started to talk. While the voice is speaking, the reply is cut
and you get `interrupted`. When it is not speaking, nothing is cut and you get no `interrupted` event. Use it for a Stop button or a push-to-talk page.

```js title="public/controls.js"
/** Wires a mute toggle and an interrupt button. `getSession` returns the live session, or null when there is none. */
export function wireControls(getSession, { muteButton, interruptButton }) {
  muteButton.addEventListener('click', () => {
    const session = getSession();
    if (!session) return;
    // The button holds the state: aria-pressed is "true" while the microphone is muted.
    const muted = muteButton.getAttribute('aria-pressed') !== 'true';
    session.mute(muted);
    muteButton.setAttribute('aria-pressed', String(muted));
    muteButton.textContent = muted ? 'Unmute' : 'Mute';
  });
  interruptButton.addEventListener('click', () => {
    // Stops the voice now, as if the person had started speaking.
    getSession()?.interrupt();
  });
}
```

## Typed turns, loudness and ending

Three more calls have no snippet here: they are one line each. See the reference for each.

- [`sendText(text)`](https://docs.voice.kheelona.com/reference/sdk/interfaces/voicesession.md#sendtext) sends a typed turn of at most 2,000 characters. It
  goes through the same safety checks. On a `child` voice, what is said aloud is checked a little more strictly than
  typed text, because speech recognition can change the words. A typed turn ends whatever the voice is doing. Longer text throws a `RangeError`.
  With `microphone: false` it is the only way to talk.
- [`levels()`](https://docs.voice.kheelona.com/reference/sdk/interfaces/voicesession.md#levels) returns `{ input, output }`, each from 0 to 1, for a
  meter. Call it from `requestAnimationFrame`.
- [`end()`](https://docs.voice.kheelona.com/reference/sdk/interfaces/voicesession.md#end) ends the session and releases the microphone. It returns a
  promise that resolves when the service has confirmed, or after 5 seconds at most. Call it when your page closes the
  conversation. A session also ends by itself at its time limit or after a long silence.

## SDK version

The SDK has its own version number, which is the number in the package file's name (`kheelona-voice-<version>.tgz`). It
follows semantic versioning: a major version can change how your page calls the SDK. The
[changelog](https://docs.voice.kheelona.com/policies/changelog.md) lists new versions. Test your page after you update.

## No sound

If the browser refuses to play the voice, for example because `connect` was not called from a click, the SDK sends a
non-fatal `error` event with the code `audio_unavailable`. The session goes on, and captions still work. The example
page shows a **Tap to allow sound** button. When the person taps it, call `session.resumeAudio()` **synchronously in the
click handler**, with no `await` before it. The example does it like this:

```js title="public/main.js"
// A tap is the gesture the browser needs: call resumeAudio() directly in the handler, with no await before it.
$('allow-sound').addEventListener('click', () => {
  session?.resumeAudio();
  $('allow-sound').hidden = true;
});
```

The call resumes the audio and tries to play the voice again inside the tap. If the browser still refuses, you get
`audio_unavailable` again.

## Content Security Policy

If your page has a Content Security Policy, three things matter:

- **Connections.** `connect-src` must allow the origins of the `connect` URLs your server got, for example
  `https://api.voice.kheelona.com` and `wss://api.voice.kheelona.com`.
- **Audio worklets on the WebSocket path.** The SDK loads two worklet files. Either host them yourself, from your own
  origin, and pass `workletBaseUrl`: the files are `capture.worklet.js` and `playback.worklet.js` in the SDK package's
  `dist` folder, and the example does this with `workletBaseUrl: '/sdk/'`. Or allow `script-src blob:`, and the SDK loads
  them from its own bundle. If a worklet cannot load, you get `audio_unavailable` rather than a hang.
- **Scripts and styles.** The example needs only `'self'` for both. It runs with no inline script and no inline style.

The complete policy of the example is on the [Security](https://docs.voice.kheelona.com/guides/security.md#a-content-security-policy-for-your-page) page.

## Errors

`connect` rejects, and the `error` event carries, a code and two flags:

- `code`: branch on this, never on the message. The [error list](https://docs.voice.kheelona.com/guides/errors-and-retries.md) has every code. The SDK adds
  four of its own: `microphone_unavailable` (missing, blocked or in use), `audio_unavailable` (see above), `network`
  (the network failed) and `transport_failed` (neither transport could be set up).
- `retryable`: trying the same thing again later can work.
- `retryAfterMs`: when the service said how long to wait, in milliseconds. It is set only when `connect` fails on the
  WebRTC path.

**Do not log `error.cause` as it is.** When a browser's own error is kept as the cause, its text can contain connection
details. Log the `code` and the `message`; both are safe.

A connection that drops after it was up ends the session: you get an `error` with the code `network` and `fatal: true`,
then `ended` with the reason `disconnected`. Start a new session to continue. The new one has no memory of the old one.

## Supported browsers

The SDK is built for the current versions of Chrome, Edge, Firefox and Safari, on desktop and on phones. We test it in
Chromium, Firefox and WebKit. It needs:

- `getUserMedia`, which browsers allow only on `https` pages and on `localhost`;
- `AudioWorklet`;
- WebRTC, or a WebSocket for the fallback.

Safari can refuse to play the voice when `microphone` is `false`: treat `audio_unavailable` as above.

## Next

- [Interruptions and echo](https://docs.voice.kheelona.com/guides/interruptions-and-echo.md)
- [Captions](https://docs.voice.kheelona.com/guides/captions.md)
- [Security](https://docs.voice.kheelona.com/guides/security.md)