Interruptions and echo

A person can talk while the voice is speaking, and the voice stops. This is called a barge-in. This page explains what happens, what your client does, and how to switch it off for devices that cannot tell the voice from the person.

What happens in a barge-in

By default a session has interruptions on. When the person starts to speak while the voice is speaking, the voice pauses at once: Kheelona stops sending audio. What happens next depends on how long the sound lasts and what was said.

The 1 s rule. If the person is still talking a second later, it is a barge-in. You get interrupted with the reply_id and heard_ms, how much of the reply the person heard. The reply then ends with reply.done and the reason interrupted. The voice forgets the part of the reply the person did not hear, and the person’s speech is a new turn.

A shorter sound is listened to. Kheelona works out what it was, and the words decide:

What the person said	What the voice does
Nothing useful: a cough, a bang, a noise, or the voice’s own echo	Carries on where it paused. No event.
A short word of agreement or surprise (“hmm”, “okay”, “wow”)	Carries on.
A stop word (“stop”, “wait”)	Stops. The person’s words become their turn, and the voice answers briefly, in character, then listens.
A worry (“help”, “I’m scared”)	Stops. The words go through the safety checks like any turn.
Anything, when the voice has just asked a question, or has just checked in, nudged or asked the person to repeat	Stops. The words are the person’s answer.
Words that the safety checks block	Stops. The words are handled like any blocked turn.

When the voice stops this way you get interrupted and reply.done with the reason interrupted, as above. The words are never lost: they are the next turn.

A long acknowledgement. If the person says only “haan, haan” for a second or more, or laughs, the voice does not answer it as a turn. The old reply ends with interrupted and reply.done with the reason interrupted, as above (flush as usual), and the voice carries on in a new reply: a short cue line, with a new reply.started and no new transcript.final. It never continues the old reply, so you never get reply.resumed for this. This happens at most once in a row, and only after an ordinary reply. It never happens over an invitation: after a question, a check-in, a nudge or a request to repeat, a long “haan haan” is the person’s answer and is answered as a normal turn.

Other words said over the voice. When the voice carries on after a short sound, those other words are not used today: the voice does not take them into account in its next reply. Stop words, worries, blocked words and answers over a question always stop the voice and become the person’s turn (the table above), so nothing that needs an answer is dropped this way.

Your client does not need to do anything for this. The voice pauses and resumes inside Kheelona. The only events you see are the ones above, plus the optional pause events below.

Your client must stop playing at once on interrupted and drop any reply audio it has queued. The browser SDK does this for you on both transports. A client on the WebSocket path, like a toy relay, flushes its own queue when interrupted arrives, and drops frames of the interrupted reply that arrive after it.

If the person keeps talking after a pause and the voice has not started to answer yet, the new speech joins the same turn. Nothing is cut off, because nothing was playing.

You can also stop the voice yourself, for a Stop button or a push-to-talk page. The browser SDK has session.interrupt() and the WebSocket has the input.interrupt event. It works whether interruptions is on or off.

Pause events (optional)

When the voice pauses, a WebSocket client keeps playing whatever audio it has already queued, up to about 300 ms. A relay can do better by pausing its own buffer. This is opt-in, so a client that does nothing behaves as before.

Declare support. After session.ready, send once:

{ "type": "client.capabilities", "playback_pause": true }

Only declare it if your client really can pause and resume its playback. Unknown fields are ignored. Without the declaration you never get the events below.

Then handle two events, each with a reply_id:

reply.paused: the voice paused its own playback because the person started to speak. Pause your playback buffer now. Keep the audio you hold.
reply.resumed: the pause ended and the same reply plays on. Resume your buffer.

reply.paused describes the voice’s playback. It is not a cue that the person is speaking, and it does not mean the voice has stopped for good. You only get these events for a reply you have already seen start. Ignore a reply.paused, reply.resumed or reply.done that names a reply other than the current one.

A pause ends in one of four ways. The first of these ends it:

reply.resumed: the same reply plays on.
interrupted: the barge-in was confirmed. Flush your queue, as always.
reply.done: the reply ended during the pause (for example after an error or a safety redirect).
The next reply.started: a new reply begins.

Your client must end its pause on any of the four, so it can never stay paused. While a pause lasts, Kheelona stops its own playback clock, so heard_ms and caption timing stay right.

How a reply is spoken, and what gets cut

The voice may speak several sentences of a reply in one go, so that the intonation flows. The first sentence is spoken first and alone, so the reply starts quickly. If the person interrupts, the voice still forgets exactly the part of the reply that was not heard, as above. reply.sentence events for the later sentences are sent when each sentence is estimated to start playing, to within a fraction of a second, so captions stay close to the audio.

Echo cancellation

A microphone near a loudspeaker hears the voice. If that sound goes back to Kheelona as if it were the person, the voice interrupts itself.

In a browser, the SDK asks the browser for echo cancellation and noise suppression when it opens the microphone, so this is handled for you. Headphones remove the problem completely. A phone on loudspeaker or a laptop with the volume up can still leak: a short leak only pauses the voice for a moment, a longer one interrupts it.
On your own server or device, echo cancellation is required, on every device, toys included: the voice’s audio must not come back in the microphone audio you send. Kheelona still checks for its own echo, but only as a safety net. Because words now count when they are said over the voice, a clean microphone matters more than before: a device that leaks a lot makes the voice more careful with short sounds for the rest of the session.

`interruptions: false`

Create the session with "interruptions": false:

{ "voice_id": "lumi-dragon", "interruptions": false }

Then the voice is never cut off by sound. While the voice is speaking, anything the microphone sends is ignored, and the person’s turn starts after the voice has finished. This also means that nothing said over the voice is heard, not even a worry. Use it for content that must play through, and keep a button for interrupt(). session.ready says which setting the session runs with. The example relay takes --no-interruptions to try it.

The trade-off is that people cannot interrupt by talking. Give them a button that calls interrupt() if they need one.

`playback.position` for WebSocket clients

On the WebSocket path the voice’s audio is sent in small steps ahead of playback, and Kheelona estimates how much of it was heard. When you cut the reply, you can send {"type":"playback.position","reply_id":"…","played_ms":1234} with how much you actually played. Kheelona uses it to forget exactly the part that was not heard. It is optional: without it Kheelona uses its own estimate. It is for WebSocket clients only. The browser SDK sends it for you on the WebSocket path and sends nothing on WebRTC.

Turn-taking: how the voice decides the person has finished, and the slow-reply line.
Server-to-server: flushing audio in a relay.
Client events: input.interrupt, playback.position and client.capabilities.