Turn-taking

The person does not press a button. The voice listens all the time, notices when they start to speak and when they have finished, and answers.

States

A session is always in one of three states, sent in state events.

State	Meaning
listening	Waiting for the person to speak.
thinking	The person finished and the reply is being prepared. No voice yet.
speaking	The reply is playing.

A session also has the state ended in the browser SDK, at the end.

How the end of a turn is found

The voice listens for speech, and when the person goes quiet it decides whether they have finished or are only pausing. A short pause in the middle of a sentence is usually not the end of a turn. If the person only paused and carries on, see “When the person carries on” below. After about 3 seconds of silence a turn ends in any case. One spoken turn is at most 60 seconds of speech: a longer one is cut and processed.

You get input.speech_started when speech is detected and input.speech_stopped when the turn ends. A turn that turns out to hold no words goes from thinking straight back to listening with no transcript and no reply.

When the person pauses in the middle of a thought

Sometimes a person stops in the middle of an idea (“I want to go to the…”) or says “wait, let me think”. Where the voice supports it, it notices this from the words, not only from the sound, and does not answer yet. What you see: input.speech_stopped, state: thinking, transcript.final, then state: listening, with no reply. If the person goes on, the voice understands both parts together as one turn. If they stay quiet, about 5 seconds after they stopped the voice says a short line in character, like “Go on, I’m listening!”, with no new transcript before it. Treat that line like a greeting.

A typed turn that comes after such a pause is understood together with the first part too. The voice says the line only once per pause, and after it the next turn is a new one.

This is on by default where it is supported and can be switched off per deployment. A typed turn is never treated as unfinished.

When the words were not understood

If the person interrupts the voice and what they said could not be understood, the voice asks them in character to say it again, and repeats its last question if they may have missed it. It also comes with no new transcript. It happens at most once in a row.

Early replies

Sometimes the voice is not sure the person has finished, and would wait a few seconds to be sure. After half a second of quiet it can start preparing a reply in the background. Nothing is said and you see nothing: no event, no caption, no audio. The turn’s events come only when the turn is confirmed, in the usual order. If the person goes on, the prepared reply is thrown away and the whole turn is understood again. If they stay quiet, the reply starts at once when the turn is confirmed. The words are checked first, as always.

The slow-reply line

If the answer is not ready about 2.5 seconds after the person finished, the voice says one short line so they are never left in silence: “Hmm, let me think!”, “One moment…”, and similar, in the session’s language. It arrives as an ordinary first reply.sentence of the reply, with its audio. The answer follows as the next sentences of the same reply.

• It never comes before the turn is confirmed.
• It does not come if the answer is ready by then.
• It never comes after a sad or worried turn, or after a turn that the safety checks flagged. Then the voice answers when it is ready.
• It is not used for a greeting or a short in-character line, and not on two turns in a row.
• If the person speaks during the line, that reply ends with interrupted and reply.done with the reason interrupted (flush as usual), and their speech joins the same turn. In the last moment of the line, once the answer itself is about to start, speech is treated like speech over the answer (see Interruptions and echo): the voice pauses and the same rules apply.

Treat it like any other sentence. It can be switched off per deployment.

When the person carries on

• While the voice is thinking, before any reply audio has played, new speech joins the same turn. The reply being prepared is dropped, and the whole turn is understood again as one. This is so that “Hi Lumi!” followed by a question gets one answer, not two.
• While the voice is speaking, new speech pauses it, and the words decide whether it stops. See Interruptions and echo.
• A typed turn (input.text) is a turn of its own, and it ends whatever is in progress first: a reply being spoken is interrupted, and a reply being prepared is cancelled.

Replies never overlap: a new reply starts only after the previous one has ended.

The order of events in one turn

1. input.speech_started
2. input.speech_stopped
3. state: thinking
4. transcript.final
5. reply.started
6. state: speaking, together with the first reply.sentence and the first audio
7. more reply.sentence events, one per sentence
8. reply.done
9. state: listening

When the person interrupts, interrupted and a reply.done with the reason interrupted take the place of the end, and the next turn’s input.speech_started follows. Every reply.started gets exactly one reply.done.

Some replies answer no turn, so no transcript.final comes before them: a greeting, the “Go on, I’m listening!” line, a request to say it again, and the check-in and goodbye of a quiet session. Each is an ordinary reply, from reply.started to reply.done.

Sessions that end the turn for you

A session also ends on its own: at its time limit, or when it stays quiet. A quiet session gets a check-in and then a goodbye before it ends. See Quiet sessions and Limits.