Voice Agents SDK for Rust

sentdm-voice-agents lets a Rust process serve calls as a Sent voice agent: it connects to Sent with your account API key, receives each call as a session, and gives your code the ways to act on it. Speech, heartbeats, reconnects and resumes are the SDK's; your code sees calls. The agent record itself, its tools, transfer targets and fallback, is set up in Voice agents; what the SDK exchanges with Sent is the voice agent protocol, which the SDK implements at version 0.2.0 and passes the protocol's conformance scenarios for before every release.

The protocol is in Preview and so is this SDK. Within sent.voice.v1 Sent only adds frames and fields, so an agent built today keeps working.

Installation

cargo add sentdm-voice-agents tokio --features tokio/full

Rust 1.85 or later. The runtime is Tokio, the connection is tokio-tungstenite with rustls, and the frames are serde_json.

Quick Start

Create the agent on Sent

Create a voice agent with POST /v3/voice/agents and bind a number to it, as Voice agents describes, and leave websocket_url out: the SDK connects to Sent, and an agent created with a websocket_url is dialled by Sent instead, so it never reaches your process. Give it the tools it may use, ["hangup", "transfer"] for the sample below, with the sample's number in transfer_targets. Note its agent_id. Choose a text session, where Sent runs speech-to-text and text-to-speech, or an audio session, where your agent handles raw audio.

Serve calls

use sentdm_voice_agents::{Agent, Handler, Options, Session, UserSaid};
use std::sync::Arc;

struct Support;

impl Handler for Support {
    async fn on_session_start(&self, session: Arc<Session>) {
        session.say("Hi, this is Acme support. How can I help?").ok();
    }

    async fn on_user_said(&self, session: Arc<Session>, said: UserSaid) {
        if said.text.contains("person") {
            session.say("Transferring you now.").ok();
            session.transfer_number("+14155550100").await.ok();
        } else {
            session.say(&format!("You said {}", said.text)).ok();
        }
    }
}

#[tokio::main]
async fn main() {
    let options = Options::new(std::env::var("SENT_API_KEY").unwrap(), "agent_...");
    Agent::new(options, Support).serve().await.unwrap();
}

serve() connects to Sent and serves calls until SIGINT or SIGTERM; run() does the same without the signal handlers, and stop() ends it. Keep the API key on the server the agent runs on, never in a client.

Handlers

Implement Handler for your type and write the methods you need; each one has a default that does nothing. Each session's handlers run in order on their own task, so sessions never wait for one another, and a slow handler delays its own session only.

Handler	When
`on_session_start(session)`	A call reached the agent. In a text session, `say` to speak; an audio session waits for audio
`on_user_said(session, said)`	Text sessions: the caller said something. `said.correction` is true when it replaces the previous utterance; `clear()` and answer again
`on_audio(session, samples)`	Audio sessions: the caller's audio, 16-bit little-endian mono PCM at `session.sample_rate`
`on_tool_result(session, result)`	Sent answered a tool call. The waiting tool call gets the same result
`on_session_end(session, end)`	The call left the agent; `end.reason` says why
`on_session_lost(session)`	The connection was lost and the session could not be resumed; Sent ends the call with `agent_unreachable`
`on_error(error)`	Sent refused a frame; the connection stays open

The session

Method	What it does
`say(text)`, `say_with(text, last, interruptible)`	Text sessions: speak to the caller. Stream a reply in pieces with `last` false and finish with `last` true
`clear()`	Drop the speech or audio the agent sent that Sent has not played yet
`send_audio(samples)`	Audio sessions: play 16-bit mono PCM at the session's sample rate; 20 ms per call, at most 200 ms
`transfer(party)`, `transfer_number(number)`	Move the call to a number, a user of your app, or another agent; resolves with Sent's `ToolResult`
`hangup()`	End the call, after the speech queued before it
`send_dtmf(digits)`	Play digits on the call: `0` to `9`, `*`, `#`, `A` to `D`, and `w` to wait half a second
`hold(held)`	Put the caller on hold, or take them off it

session.from, to, direction, session_type, language, context (what your backend passed when it placed an outbound call) and sample_rate carry what the call started with. A tool refused or failed resolves with a result whose ok is false and an error.code such as tool_not_allowed or target_not_allowed; the agent's tools and transfer targets settings decide what it may do. Every method returns Error::SessionEnded once the call has left the agent.

Resilience

The SDK answers Sent's heartbeats, opens a new connection when Sent drains one, reconnects with backoff when a connection is lost, and resumes the sessions that can move so the callers stay on the line. It keeps the last frames it sent per session and sends again the ones Sent did not read. A session that cannot be resumed reaches on_session_lost. Nothing in your handlers needs to know.

Frames

sentdm_voice_agents::frames holds every frame of the protocol as a type, with decode and encode that check the protocol's rules. It is generated from the protocol and regenerated on every protocol release.

Other languages

The Python, TypeScript, .NET and Go SDKs are the same agent in those languages; Java follows, on the same protocol and the same conformance scenarios. Any language with a WebSocket client can build an agent on the protocol directly.

On this page