# Server events

> Server events: sent by Kheelona Voice to the client.

Source: https://docs.voice.kheelona.com/reference/generated/events/server/

Server events: sent by Kheelona Voice to the client, as JSON text frames. Every event has a `type` field.

Every server event also carries:

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `event_id` | [EventId](#eventid) | yes | Server-generated ID. | pattern `^evt_[0-9A-HJKMNP-TV-Z]{26}$` |
| `ts` | string | yes | When the event was created: RFC 3339 UTC with milliseconds. | format `date-time` |

| Event | When |
| --- | --- |
| [`session.ready`](#sessionready) | The connection is accepted. |
| [`state`](#state) | The conversation state changed. |
| [`input.speech_started`](#inputspeech_started) | Voice activity detection heard speech. |
| [`input.speech_stopped`](#inputspeech_stopped) | The turn detector decided the turn is complete. |
| [`transcript.final`](#transcriptfinal) | What was heard (live text; never stored). |
| [`reply.started`](#replystarted) | The voice began a reply. |
| [`reply.sentence`](#replysentence) | A sentence's audio started being delivered. `text` has no emotion tags (live text; never stored). |
| [`reply.done`](#replydone) | The reply finished. |
| [`interrupted`](#interrupted) | The person talked over the voice. The client MUST flush queued reply audio immediately. |
| [`reply.paused`](#replypaused) | Only for a client that declared `playback_pause` (`client.capabilities`), and only for a reply it saw start: the voice paused its own playback because the person started to speak. The client SHOULD pause its playback buffer now. Not a listening cue. The pause ends at the first of `reply.resumed`, `interrupted`, `reply.done` or the next `reply.started`. |
| [`reply.resumed`](#replyresumed) | Only after `reply.paused`: the pause ended and the same reply plays on. The client resumes its playback buffer. |
| [`safety`](#safety) | A safety rule fired. Category only, never text. |
| [`session.warning`](#sessionwarning) | A limit is close: 60 s before max duration, 20 s before the idle timeout. |
| [`error`](#error) | A problem. If `fatal`, the session ends next. |
| [`session.ended`](#sessionended) | The last event before the connection closes. |
| [`pong`](#pong) | Answer to `ping`. |

## `session.ready`

The connection is accepted.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `session_id` | [SessionId](#sessionid) | yes | The session. | pattern `^ses_[0-9A-HJKMNP-TV-Z]{26}$` |
| `voice` | [VoiceInfo](#voiceinfo) | yes | The voice and the version this session pinned. |  |
| `languages` | array of [Lang](#lang) | yes | The session's declared languages (default `["en","hi"]`). |  |
| `audio` | [AudioInfo](#audioinfo) | yes | Audio formats in each direction. |  |
| `limits` | [Limits](#limits) | yes | Duration and idle limits. |  |
| `interruptions` | boolean | yes | Whether the person can interrupt the voice. |  |

## `state`

The conversation state changed.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `value` | [StateValue](#statevalue) | yes | The new state. | one of `listening`, `thinking`, `speaking` |

## `input.speech_started`

Voice activity detection heard speech.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `turn` | integer | yes | The turn number. | format `uint32`; min 0 |

## `input.speech_stopped`

The turn detector decided the turn is complete.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `turn` | integer | yes | The turn number. | format `uint32`; min 0 |

## `transcript.final`

What was heard (live text; never stored).

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `turn` | integer | yes | The turn number. | format `uint32`; min 0 |
| `text` | string | yes | The transcript. |  |
| `language` | [Lang](#lang) | yes | The language it was heard in. | one of `en`, `hi`, `mr`, `bn`, `as`, `gu`, `pa`, `ta`, `te`, `kn`, `ml` |
| `language_beta` | boolean | yes | True for every language that is not supported (beta or alpha; kept for older clients). |  |
| `language_stage` | [LangStage](#langstage) | yes | The language's stage: `supported`, `beta` or `alpha`. | one of `supported`, `beta`, `alpha` |

## `reply.started`

The voice began a reply.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `turn` | integer | yes | The turn being answered. | format `uint32`; min 0 |
| `reply_id` | [ReplyId](#replyid) | yes | The reply. | pattern `^rep_[0-9A-HJKMNP-TV-Z]{26}$` |

## `reply.sentence`

A sentence's audio started being delivered. `text` has no emotion tags (live text; never stored).

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `reply_id` | [ReplyId](#replyid) | yes | The reply. | pattern `^rep_[0-9A-HJKMNP-TV-Z]{26}$` |
| `index` | integer | yes | The sentence index within the reply (also in the audio frame header). | format `uint16`; min 0; max 65535 |
| `text` | string | yes | The sentence as caption text. |  |
| `emotion` | [Emotion](#emotion) | yes | The emotion it's spoken with. | one of `excited`, `happy`, `giggly`, `calm`, `soft`, `sad`, `surprised` |

## `reply.done`

The reply finished.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `reply_id` | [ReplyId](#replyid) | yes | The reply. | pattern `^rep_[0-9A-HJKMNP-TV-Z]{26}$` |
| `reason` | [ReplyDoneReason](#replydonereason) | yes | Why it finished. | one of `completed`, `interrupted`, `safety`, `error` |
| `sentences_spoken` | integer | yes | Sentences actually spoken. | format `uint16`; min 0; max 65535 |

## `interrupted`

The person talked over the voice. The client MUST flush queued reply audio immediately.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `reply_id` | [ReplyId](#replyid) | yes | The interrupted reply. | pattern `^rep_[0-9A-HJKMNP-TV-Z]{26}$` |
| `heard_ms` | integer | yes | Milliseconds of it the person heard. | format `uint32`; min 0 |

## `reply.paused`

Only for a client that declared `playback_pause` (`client.capabilities`), and only for a reply it saw start: the voice paused its own playback because the person started to speak. The client SHOULD pause its playback buffer now. Not a listening cue. The pause ends at the first of `reply.resumed`, `interrupted`, `reply.done` or the next `reply.started`.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `reply_id` | [ReplyId](#replyid) | yes | The paused reply. | pattern `^rep_[0-9A-HJKMNP-TV-Z]{26}$` |

## `reply.resumed`

Only after `reply.paused`: the pause ended and the same reply plays on. The client resumes its playback buffer.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `reply_id` | [ReplyId](#replyid) | yes | The resumed reply. | pattern `^rep_[0-9A-HJKMNP-TV-Z]{26}$` |

## `safety`

A safety rule fired. Category only, never text.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `turn` | integer | yes | The turn. | format `uint32`; min 0 |
| `stage` | [SafetyStage](#safetystage) | yes | Where the rule fired. | one of `input`, `output`, `provider` |
| `category` | string | yes | What kind of content fired, as a stable `snake_case` label: from the classifier `violent`, `illegal`, `sexual`, `pii`, `self_harm`, `unethical`, `political`, `copyright`, `jailbreak`, `unknown`, `unsafe` (no category named); from the blocklists also `profanity`, `hate`, `romance`, `scary`, `dangerous_activity`, `substances`, `personal_data`. A free string: new labels may be added, so treat an unknown one as "other". |  |
| `action` | [SafetyAction](#safetyaction) | yes | What the voice did. | one of `redirected`, `sentence_dropped` |

## `session.warning`

A limit is close: 60 s before max duration, 20 s before the idle timeout.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `code` | [WarningCode](#warningcode) | yes | Which limit. | one of `max_duration_soon`, `idle_soon` |
| `seconds_left` | integer | yes | Seconds until it's reached. | format `uint32`; min 0 |

## `error`

A problem. If `fatal`, the session ends next.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `code` | [ErrorCode](#errorcode) | yes | The error code. | one of `invalid_request`, `invalid_audio`, `unauthenticated`, `secret_expired`, `forbidden`, `origin_not_allowed`, `voice_not_found`, `session_not_found`, `capacity_exhausted`, `partner_limit_reached`, `rate_limited`, `provider_unavailable`, `internal`, `not_found`, `conflict`, `unavailable` |
| `message` | string | yes | A human-readable explanation (no conversation content). |  |
| `retryable` | boolean | yes | Whether retrying can help. |  |
| `fatal` | boolean | yes | Whether the session ends next. |  |

## `session.ended`

The last event before the connection closes.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `reason` | [EndReason](#endreason) | yes | Why it ended. | one of `client_ended`, `partner_ended`, `max_duration`, `idle_timeout`, `disconnected`, `server_shutdown`, `provider_failure`, `suspended`, `service_stopped`, `error` |
| `duration_ms` | integer | yes | Connected time in milliseconds. | format `uint64`; min 0 |
| `turns` | integer | yes | Turns in the session. | format `uint32`; min 0 |

## `pong`

Answer to `ping`.

No fields besides `type`.

## Types

### AudioFormat

PCM16 little-endian mono at one of three sample rates.

| Value | Description |
| --- | --- |
| `pcm16_16k` | 16,000 Hz (the input default). |
| `pcm16_24k` | 24,000 Hz (the output default). |
| `pcm16_48k` | 48,000 Hz. |

### AudioInfo

Audio formats chosen at session creation.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `input_format` | [AudioFormat](#audioformat) | yes | Client → server audio. | one of `pcm16_16k`, `pcm16_24k`, `pcm16_48k` |
| `output_format` | [AudioFormat](#audioformat) | yes | Server → client audio. | one of `pcm16_16k`, `pcm16_24k`, `pcm16_48k` |

### Emotion

The emotion a sentence is spoken with.

| Value | Description |
| --- | --- |
| `excited` | Excited |
| `happy` | Happy: the first sentence's default when the model gives no tag. |
| `giggly` | Giggly: only when the person is laughing. |
| `calm` | Calm |
| `soft` | Soft |
| `sad` | Sad |
| `surprised` | Surprised |

### EndReason

Why a session ended.

| Value | Description |
| --- | --- |
| `client_ended` | The client sent `session.end`. |
| `partner_ended` | The partner called `POST /v1/sessions/{id}/end`. |
| `max_duration` | The maximum duration was reached. |
| `idle_timeout` | Nobody spoke for the idle timeout. |
| `disconnected` | The connection dropped. |
| `server_shutdown` | The server is shutting down. |
| `provider_failure` | Three provider failures in a row. |
| `suspended` | The partner was suspended mid-session. |
| `service_stopped` | The voice service was switched off. |
| `error` | Another error. |

### ErrorCode

A machine-readable error code.

| Value | Description |
| --- | --- |
| `invalid_request` | Fix the request. |
| `invalid_audio` | A malformed audio frame; the frame is dropped. |
| `unauthenticated` | Missing or invalid API key or client secret. |
| `secret_expired` | The client secret is older than 60 s or already used: create a new session. |
| `forbidden` | The key lacks the scope, or the partner is suspended. |
| `origin_not_allowed` | The web origin isn't allowed for this partner. |
| `voice_not_found` | Unknown or retired voice, or not enabled for this partner. |
| `session_not_found` | No such session. |
| `capacity_exhausted` | The whole service is full; retry after the hinted seconds. |
| `partner_limit_reached` | The partner is at its maximum concurrent sessions. |
| `rate_limited` | Too many API calls. |
| `provider_unavailable` | A processing step failed. Non-fatal for one turn; fatal after 3 failures in a row. |
| `internal` | Our bug; the request ID finds it. |
| `not_found` | REST only: the resource doesn't exist, or isn't visible to your API key (a resource that belongs to someone else is `not_found`, so its existence isn't revealed). |
| `conflict` | REST only: the request conflicts with the current state of the resource; change the request and retry. |
| `unavailable` | REST only: the API is temporarily unavailable; retry later. |

### EventId

string, pattern `^evt_[0-9A-HJKMNP-TV-Z]{26}$`.

### Lang

A supported language, as its ISO 639-1 code. Odia is not supported.

| Value | Description |
| --- | --- |
| `en` | English |
| `hi` | Hindi |
| `mr` | Marathi (beta) |
| `bn` | Bengali (beta) |
| `as` | Assamese (alpha) |
| `gu` | Gujarati |
| `pa` | Punjabi (beta) |
| `ta` | Tamil |
| `te` | Telugu (beta) |
| `kn` | Kannada (beta) |
| `ml` | Malayalam (alpha) |

### LangStage

How far a language is tested: supported, beta or alpha.

| Value | Description |
| --- | --- |
| `supported` | Tested end to end. |
| `beta` | Works, with more recognition mistakes. |
| `alpha` | Early: spoken turns are often not recognised. |

### Limits

Session limits.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `max_duration_s` | integer | yes | Maximum session length in seconds. | format `uint32`; min 0 |
| `idle_timeout_s` | integer | yes | Idle timeout in seconds. | format `uint32`; min 0 |

### ReplyDoneReason

Why a reply finished.

| Value | Description |
| --- | --- |
| `completed` | Every sentence was spoken. |
| `interrupted` | The person talked over it. |
| `safety` | A safety rule ended it. |
| `error` | A provider failed. |

### ReplyId

string, pattern `^rep_[0-9A-HJKMNP-TV-Z]{26}$`.

### SafetyAction

What the voice did when a safety rule fired.

| Value | Description |
| --- | --- |
| `redirected` | Spoke the voice's fallback line instead. |
| `sentence_dropped` | Dropped the sentence. |

### SafetyStage

Where a safety rule fired.

| Value | Description |
| --- | --- |
| `input` | On the person's turn, before the reply model. |
| `output` | On a reply sentence, before speech. |
| `provider` | Reported by a provider. |

### SessionId

string, pattern `^ses_[0-9A-HJKMNP-TV-Z]{26}$`.

### StateValue

Conversation state.

| Value | Description |
| --- | --- |
| `listening` | Waiting for the person. |
| `thinking` | From the end of the person's turn until the first reply audio. |
| `speaking` | Reply audio is being delivered. |

### VoiceInfo

The voice a session uses.

| Field | Type | Required | Description | Rules |
| --- | --- | --- | --- | --- |
| `id` | [VoiceSlug](#voiceslug) | yes | Public voice ID, e.g. `lumi-dragon`. | pattern `^[a-z][a-z0-9-]{2,39}$` |
| `version` | integer | yes | The pinned version (from 1). | format `uint32`; min 0 |
| `name` | string | yes | Display name. |  |

### VoiceSlug

string, pattern `^[a-z][a-z0-9-]{2,39}$`.

### WarningCode

Which limit a `session.warning` is about.

| Value | Description |
| --- | --- |
| `max_duration_soon` | Maximum duration in 60 s. |
| `idle_soon` | Idle timeout in 20 s. |