Voice Agents SDK for Java

dm.sent:voice-agents lets a Java process serve calls as a Sent voice agent: it connects to Sent with your account API key, receives each call as a session, and gives your code the ways to act on it. Speech, heartbeats, reconnects and resumes are the SDK's; your code sees calls. The agent record itself, its tools, transfer targets and fallback, is set up in Voice agents; what the SDK exchanges with Sent is the voice agent protocol, which the SDK implements at version 0.2.0 and passes the protocol's conformance scenarios for before every release.

The protocol is in Preview and so is this SDK. Within sent.voice.v1 Sent only adds frames and fields, so an agent built today keeps working.

Installation

// Gradle
implementation("dm.sent:voice-agents:0.1.0")

<!-- Maven -->
<dependency>
  <groupId>dm.sent</groupId>
  <artifactId>voice-agents</artifactId>
  <version>0.1.0</version>
</dependency>

Java 17 or later. The connection is the JDK's java.net.http.WebSocket; the one dependency is jackson-databind, for the frames.

Quick Start

Create the agent on Sent

Create a voice agent with POST /v3/voice/agents and bind a number to it, as Voice agents describes, and leave websocket_url out: the SDK connects to Sent, and an agent created with a websocket_url is dialled by Sent instead, so it never reaches your process. Give it the tools it may use, ["hangup", "transfer"] for the sample below, with the sample's number in transfer_targets. Note its agent_id. Choose a text session, where Sent runs speech-to-text and text-to-speech, or an audio session, where your agent handles raw audio.

Serve calls

import dm.sent.voice.agents.Agent;
import dm.sent.voice.agents.Handler;
import dm.sent.voice.agents.Options;
import dm.sent.voice.agents.Session;
import dm.sent.voice.agents.frames.UserSaid;

public class Support implements Handler {
    @Override
    public void onSessionStart(Session session) {
        session.say("Hi, this is Acme support. How can I help?");
    }

    @Override
    public void onUserSaid(Session session, UserSaid said) {
        if (said.text().contains("person")) {
            session.say("Transferring you now.");
            session.transfer("+14155550100").join();
        } else {
            session.say("You said " + said.text());
        }
    }

    public static void main(String[] args) {
        Options options = new Options(System.getenv("SENT_API_KEY"), "agent_...");
        new Agent(options, new Support()).serve();
    }
}

serve() connects to Sent and serves calls until SIGINT or SIGTERM; run() does the same and returns a future that completes on stop(). Keep the API key on the server the agent runs on, never in a client.

Handlers

Implement Handler and write the methods you need; each one has a default that does nothing. Each session's handlers run in order on their own chain, so sessions never wait for one another, and a slow handler delays its own session only. A handler may block on a tool's future with join() without holding the other sessions up.

Handler	When
`onSessionStart(session)`	A call reached the agent. In a text session, `say` to speak; an audio session waits for audio
`onUserSaid(session, said)`	Text sessions: the caller said something. `said.correction()` is true when it replaces the previous utterance; `clear()` and answer again
`onAudio(session, samples)`	Audio sessions: the caller's audio, 16-bit little-endian mono PCM at `session.sampleRate()`
`onToolResult(session, result)`	Sent answered a tool call. The waiting future gets the same result
`onSessionEnd(session, end)`	The call left the agent; `end.reason()` says why
`onSessionLost(session)`	The connection was lost and the session could not be resumed; Sent ends the call with `agent_unreachable`
`onError(error)`	Sent refused a frame; the connection stays open

The session

Method	What it does
`say(text)`, `say(text, last, interruptible)`	Text sessions: speak to the caller. Stream a reply in pieces with `last` false and finish with `last` true
`clear()`	Drop the speech or audio the agent sent that Sent has not played yet
`sendAudio(samples)`	Audio sessions: play 16-bit mono PCM at the session's sample rate; 20 ms per call, at most 200 ms
`transfer(party)`, `transfer(number)`	Move the call to a number, a user of your app, or another agent; a `CompletableFuture` of Sent's `ToolResult`
`hangup()`	End the call, after the speech queued before it
`sendDtmf(digits)`	Play digits on the call: `0` to `9`, `*`, `#`, `A` to `D`, and `w` to wait half a second
`hold(held)`	Put the caller on hold, or take them off it

session.from(), to(), direction(), sessionType(), language(), context() (what your backend passed when it placed an outbound call) and sampleRate() carry what the call started with. A tool refused or failed completes with a result whose ok() is false and an error().code() such as tool_not_allowed or target_not_allowed; the agent's tools and transfer targets settings decide what it may do. Every method throws SessionEndedException, or fails its future with it, once the call has left the agent.

Resilience

The SDK answers Sent's heartbeats, opens a new connection when Sent drains one, reconnects with backoff when a connection is lost, and resumes the sessions that can move so the callers stay on the line. It keeps the last frames it sent per session and sends again the ones Sent did not read. A session that cannot be resumed reaches onSessionLost. Nothing in your handlers needs to know.

Frames

The dm.sent.voice.agents.frames package holds every frame of the protocol as a record, with Protocol.decode() and Protocol.encode() that check the protocol's rules. It is generated from the protocol and regenerated on every protocol release.

Other languages

The Python, TypeScript, .NET, Go and Rust SDKs are the same agent in those languages, each on the same protocol and the same conformance scenarios. Any language with a WebSocket client can build an agent on the protocol directly.

On this page