Agent text input

Overview

In a voice agent, Speech-to-Text only hears the caller. The agent's own voice is played to the caller and removed from the microphone by echo cancellation, so the model never hears the question the caller is answering.

Agent text input fixes that. While your agent speaks, you send its words to Soniox on the same WebSocket as the caller's audio. The model then knows what was just said and transcribes the caller's reply with that in mind.

Agent text input is available on stt-rt-v6. See models.

Why use it

The caller's replies are recognized better

A person on the line knows what the caller is answering, because they heard the question. Agent text gives the model the same advantage. A few examples:

Agent says	Caller says
"There are two nearby: Maple Cafe and River Roast. Which one would you like?"	"River Roast, please."
"What type of assistance do you need for your stranded car?"	"I need a tow."
"Which Thai soup would you like: tom yum or coconut soup?"	"Tom yum with shrimp."

Without the question, "River Roast" can easily become "River Road", "a tow" can become "a tour", and a dish like "tom yum" is hard to get right from sound alone.

These are just a few cases. Whenever a reply depends on what the agent asked, offered or read back, knowing the agent's words helps the model get it right. Short answers gain the most, since a word or two on its own gives the model little to go on. The caller's first words, before the agent has said anything, get no help from agent text; context helps there.

The agent's voice isn't transcribed as the caller

Echo cancellation is never perfect, and parts of the agent's voice often leak into the microphone, especially when it starts talking. Because the model knows what the agent is saying, it can tell that leftover echo apart from the caller and leave it out of the transcript.

Keep your echo cancellation on. Agent text input handles what it leaves behind; it doesn't replace it.

The agent's text never appears in the transcript. You only get the caller's speech back.

How it works

Enable it in the start request

{
  "model": "stt-rt-v6",
  "audio_format": "auto",
  "enable_agent_text_input": true
}

Stream the caller's audio as usual

Send the caller's microphone audio as binary frames, the same as in any real-time session.

Send the agent's text as it's spoken

While the agent speaks, send its words a few at a time as text frames on the same WebSocket:

{"type": "agent_text", "text": "There are "}
{"type": "agent_text", "text": "two nearby: "}
{"type": "agent_text", "text": "Maple Cafe "}
{"type": "agent_text", "text": "and River Roast. "}
{"type": "agent_text", "text": "Which one "}
{"type": "agent_text", "text": "would you like?", "end_of_segment": true}

At the end of each agent turn, send end_of_segment: true, either on its own or with the last piece of text.

Message fields

typeRequiredstring

Must be agent_text.

textstring

The next piece of the agent's text. Can be empty.

end_of_segmentboolean

Set to true when the agent's turn ends or is interrupted. Defaults to false.

Send the text exactly as the agent says it, spaces included. Soniox joins the pieces of a turn just as you sent them, so don't strip or add spaces.

Always end a turn with end_of_segment.

Send the text in time with the audio

Agent text messages don't carry timestamps. The model places each one at the point in the audio stream where you sent it. So when you send the text matters as much as what you send.

Send each piece of text a little before the caller hears it, about 500 ms to 1 second ahead of playback:

Follow the playback, not the LLM. Your LLM writes the reply much faster than it's spoken. Don't forward the text as it comes out of the LLM, and don't send the whole reply at once.
Follow the playback, not the TTS. TTS also generates audio faster than real time. What counts is when the audio is actually played to the caller.
Use TTS timestamps. Character or word timestamps from your TTS tell you when each word is spoken. With Soniox Text-to-Speech, set return_timestamps to get them.

A good place to send it is right after each chunk of caller audio. A separate timer every 50 ms or so works too.

Here's the idea in Python, using the character timestamps from Soniox Text-to-Speech. After each audio chunk, send the characters that will be played within the next 700 ms:

import json

LEAD_S = 0.7  # send text 700 ms before it's played

async def stream_caller_audio(ws, mic, agent):
    async for chunk in mic:
        await ws.send(chunk)
        if agent.reply:  # the agent is speaking
            await send_due_text(ws, agent.reply, agent.player.position_s())

async def send_due_text(ws, reply, position_s):
    # reply.characters and reply.start_times grow as TTS timestamps arrive
    # (characters and character_start_times_seconds). reply.tts_done is set
    # once the TTS has sent the whole reply.
    due = reply.sent
    while due < len(reply.characters) and reply.start_times[due] <= position_s + LEAD_S:
        due += 1

    if due > reply.sent:
        await ws.send(json.dumps({
            "type": "agent_text",
            "text": "".join(reply.characters[reply.sent:due]),
            "end_of_segment": reply.tts_done and due == len(reply.characters),
        }))
        reply.sent = due

agent.player.position_s() stands for however your app tracks how much of the reply has been played. If your TTS gives word timestamps instead, the same code works with words.

If your agent runs on a server and doesn't get playback reports from the caller's device, you can estimate the position from the time since you started sending the reply's audio to the caller. Network delays are small enough not to matter. Just never count more than the audio you've actually sent, so the text doesn't run ahead when the TTS falls behind:

import time

def position_s(self):
    elapsed = time.monotonic() - self.playback_started_at
    return min(elapsed, self.audio_sent_s)

You can split the text anywhere, even in the middle of a word. Soniox joins the pieces back together.

When the caller interrupts

If the caller starts talking and you stop the agent's playback:

Send any text that was already played but not sent yet.
Send end_of_segment: true.
Don't send the rest of the reply. The caller never heard it.

Text you sent ahead of the interruption is fine. It stays where it is.

What to send

Only what the agent says out loud. Remove anything the TTS doesn't speak, like SSML, markdown or expression tags such as [laughs].
One segment per agent turn. Mark the end of a turn, not the end of each sentence.
Only agent speech. Don't use agent text for instructions, notes or background information. The model expects the agent's words in time with the audio. For background information, use context.

Agent text and context work well together: context covers the names and words of your business for the whole call, and agent text covers what the agent just said.

Errors

These errors end the stream:

enable_agent_text_input on a model that doesn't support it.
An agent_text message in a session without enable_agent_text_input: Agent text requires enable_agent_text_input in the start request.
An agent_text message that is too long: Agent text is too long. Maximum length is N characters.

See error handling.

Pricing

Agent text is billed per token, at the same rate as other input text tokens, like context. Only text that reaches the model is billed.

Agent text input

On this page