Connect a voice agent (Preview)

A voice agent is on a call as a session over a WebSocket. Either your agent opens the connection to Sent and holds it, or Sent opens one to your agent for each call. The frames are the same either way: Sent tells the agent a call has started, the two exchange text or audio, the agent acts on the call with tools, and Sent tells the agent when the call has left it. This guide covers getting connected and keeping the connection healthy. The frames themselves are in the voice agent protocol.

Prerequisites

• A voice agent, created with POST /v3/voice/agents. See Voice agents.
• Phone calls enabled for your account, and a number bound to the agent, so that a call can reach it.
• A WebSocket client, or a WebSocket server when Sent connects to you, in your agent's language. Frames are JSON text messages, one per message, plus binary messages for audio. The sample agents in the protocol repository connect in, answer pings, play the caller's audio back in an audio session, and move their sessions when a connection drains.

Your agent connects to Sent

This is how an agent without a websocket_url works. Open one WebSocket to Sent with your account API key as a bearer token, and ask for the sent.voice.v1 subprotocol:

GET wss://voice.sent.dm/v1/agents/connect?agent_id=agent_9f2ab000-0000-4000-8000-000000000001
Authorization: Bearer sk_live_...
Sec-WebSocket-Protocol: sent.voice.v1

Never put the key in the URL. Sent refuses the upgrade with:

Once connected, the first frame Sent sends is connection.ready. It carries the protocol_version Sent speaks, heartbeat_interval_ms, and max_sessions, the most sessions Sent puts on this connection at once. From then on, every call to a number bound to the agent arrives as a session.start frame.

import WebSocket from 'ws';

const ws = new WebSocket(
  'wss://voice.sent.dm/v1/agents/connect?agent_id=agent_9f2ab000-0000-4000-8000-000000000001',
  'sent.voice.v1',
  { headers: { Authorization: `Bearer ${process.env.SENT_API_KEY}` } },
);

const send = (frame) => ws.send(JSON.stringify(frame));

ws.on('message', (data, isBinary) => {
  if (isBinary) return; // audio sessions only, see the protocol reference
  const frame = JSON.parse(data.toString());

  switch (frame.type) {
    case 'ping':
      send({ type: 'pong', ts: frame.ts });
      break;
    case 'session.start':
      // A text session. With a greeting set on the agent, Sent speaks it first.
      send({ type: 'agent.say', session_id: frame.session_id, text: 'Hi, this is Acme. How can I help?' });
      break;
    case 'user.said':
      // Hand frame.text to your model, then speak the reply
      send({ type: 'agent.say', session_id: frame.session_id, text: replyTo(frame.text) });
      break;
    case 'session.end':
      console.log(`Session ${frame.session_id} ended: ${frame.reason}`);
      break;
  }
});

import asyncio, json, os
import websockets

URL = "wss://voice.sent.dm/v1/agents/connect?agent_id=agent_9f2ab000-0000-4000-8000-000000000001"

async def run():
    async with websockets.connect(
        URL,
        subprotocols=["sent.voice.v1"],
        additional_headers={"Authorization": f"Bearer {os.environ['SENT_API_KEY']}"},
    ) as ws:
        async for message in ws:
            if isinstance(message, bytes):
                continue  # audio sessions only, see the protocol reference
            frame = json.loads(message)
            if frame["type"] == "ping":
                await ws.send(json.dumps({"type": "pong", "ts": frame["ts"]}))
            elif frame["type"] == "session.start":
                await ws.send(json.dumps({"type": "agent.say", "session_id": frame["session_id"],
                                          "text": "Hi, this is Acme. How can I help?"}))
            elif frame["type"] == "user.said":
                await ws.send(json.dumps({"type": "agent.say", "session_id": frame["session_id"],
                                          "text": reply_to(frame["text"])}))
            elif frame["type"] == "session.end":
                print(f"Session {frame['session_id']} ended: {frame['reason']}")

asyncio.run(run())

One connection, many sessions

Sessions on one connection run at the same time, each named by its session_id, and their frames interleave. Frames for one session arrive in the order they were sent. To take more calls than max_sessions, open more connections, for example one per process. Sent spreads calls across the connections you hold open, and the agent's limits.max_concurrent_calls caps the total.

Sent checks the key behind every open connection again about once a minute. A key you revoke, an agent you delete, or phone calls turned off for the account close the connection with code 4003 within about two minutes, and its sessions end.

Answer every ping

Sent sends a ping frame on every connection at heartbeat_interval_ms, 5,000 ms. Answer each one with a pong carrying the same ts. When three pings in a row go unanswered, Sent closes the connection with code 4008 and its sessions detach, as Resume a session after a lost connection describes. When no ping reaches you for three intervals, treat the connection as dead, open a new one, and resume your sessions there. Heartbeats also keep idle proxies and load balancers from closing the connection between calls.

When Sent drains the connection

A connection.draining frame means Sent starts no more sessions on this connection, while the sessions already on it go on. Sent sends it before it restarts the side that holds your connection, so an agent that handles it keeps its callers through a Sent deploy.

1. Open a new connection for the same agent_id with the same key, straight away, so that new calls keep reaching the agent.
2. After its connection.ready, send session.resume on the new connection for each session on the old one, with the session's latest token, from its session.start or its last session.resumed, and received, how many of Sent's counted frames for the session you've read. See Resume a session after a lost connection.
3. Once session.resumed arrives, the session is on the new connection, and the old one carries nothing more for it.

Sent closes the drained connection with code 1001 once no session is left on it. The sample agents do this.

Close a connection with code 1000 only when the agent means to stop serving its calls: its sessions end at once and the calls follow the agent's fallback. Any other close is treated as a loss, and the sessions wait for a resume.

Sent connects to your agent

This is how an agent with a websocket_url works. For each call, Sent opens one WebSocket to that URL, offering the sent.voice.v1 subprotocol, sends connection.ready with room for one session, then session.start, and closes the connection with code 1000 after session.end. Your server may select the subprotocol or none.

The URL must use wss and resolve to a public address. Sent resolves the host when it connects, refuses a private, loopback, or link-local address, follows no redirect, and uses no proxy. A handshake that isn't accepted within 5 seconds declines the call, which then follows the agent's fallback.

Verify the handshake

Sent signs the handshake the way it signs every webhook, with the agent's signing_secret:

The signature is HMAC-SHA256 of {X-Webhook-ID}.{X-Webhook-Timestamp}. followed by an empty payload, because a handshake has no body. The key is the signing_secret with its whsec_ prefix removed, then Base64-decoded. The verifier you run for webhooks works unchanged with an empty payload. Refuse the handshake with 401 when the signature doesn't match, when the timestamp is more than 5 minutes from your clock, or when you've seen the X-Webhook-Event-ID before.

import { createHmac, timingSafeEqual } from 'node:crypto';
import { WebSocketServer } from 'ws';

const TOLERANCE_SECONDS = 300;
const seen = new Set<string>(); // session ids already accepted; bound its size in a real server

function verifySentHandshake(headers, secret) {
  const id = headers['x-webhook-id'];
  const eventId = headers['x-webhook-event-id'];
  const timestamp = headers['x-webhook-timestamp'];
  const signature = headers['x-webhook-signature'];
  if (!id || !eventId || !timestamp || !signature) return false;

  const skew = Math.abs(Math.floor(Date.now() / 1000) - Number(timestamp));
  if (!Number.isFinite(skew) || skew > TOLERANCE_SECONDS) return false;
  if (seen.has(eventId)) return false;

  const key = Buffer.from(secret.replace(/^whsec_/, ''), 'base64');
  const expected = createHmac('sha256', key).update(`${id}.${timestamp}.`).digest();

  const ok = signature
    .split(/\s+/)
    .filter((part) => part.startsWith('v1,'))
    .some((part) => {
      const provided = Buffer.from(part.slice(3), 'base64');
      return provided.length === expected.length && timingSafeEqual(provided, expected);
    });
  if (ok) seen.add(eventId);
  return ok;
}

// Behind TLS: the agent's websocket_url is wss://, so terminate TLS in front of this server
const server = new WebSocketServer({
  port: 8080,
  handleProtocols: (protocols) => (protocols.has('sent.voice.v1') ? 'sent.voice.v1' : false),
  verifyClient: ({ req }, done) => {
    const ok = verifySentHandshake(req.headers, process.env.SENT_AGENT_SIGNING_SECRET);
    done(ok, ok ? 200 : 401, ok ? undefined : 'Invalid signature');
  },
});

server.on('connection', (ws) => {
  // One session per connection: connection.ready, session.start, the call, session.end, then Sent closes with 1000
});

POST /v3/voice/agents/{id}/rotate-secret answers with the new signing_secret and the old one stops signing at once, so install the new secret on the server straight after the call. Handshakes in between fail verification, so rotate at a quiet moment, and let the verifier accept both secrets while the switch is under way if calls can't wait. A session on a connection Sent opened can't be resumed: when the connection is lost, the session ends with agent_unreachable at once and the call follows the agent's fallback.

A session, start to end

For each session, Sent's first frame is session.start and its last is session.end. After session.end the agent sends nothing more for that session.

1. session.start names the session, the call, the agent, the parties and the session_type. On a text session it carries the language from the agent's speech. On an audio session it carries audio.stream and audio.sample_rate.
2. The greeting, when the agent has one, is spoken by Sent right after session.start. Otherwise the agent speaks first or waits for the caller.
3. Text sessions exchange user.said and agent.say. Audio sessions exchange binary messages of 16-bit little-endian mono PCM at audio.sample_rate, each with a 2-byte big-endian stream id in front, 20 ms of audio per message. Both can send agent.clear to drop what Sent hasn't played yet.
4. Tools act on the call: tool.call from the agent, answered by exactly one tool.result.
5. session.end says why the call left the agent: the caller hung up, the agent hung up or transferred, the call reached max_call_seconds, or the agent became unreachable.

Every frame and field is in the voice agent protocol, with a worked text session.

Resume a session after a lost connection

The calls on a connection your agent opened outlive the connection. A network fault, your own deploy or a load balancer closing a long-lived socket ends the connection while the callers are still on the line. The agent picks their sessions up on a new connection, within a window.

A session that can be resumed has resume in its session.start: a token and window_ms, how long Sent keeps the session after its connection is lost. Each session.resumed brings the next token. The token is a secret that resumes this one session, once, on a connection opened for the same agent_id with the same key. Keep it out of URLs and logs.

While a session is detached:

• Speech and audio the agent sent, and Sent hasn't played yet, keep playing, and a tool queued behind them still runs.
• The caller's audio is dropped and isn't replayed.
• Sent keeps the frames it would have sent, such as user.said and tool.result, to replay on the new connection. A call that ends meanwhile keeps its session.end for the replay too.

When window_ms passes with the session still detached, Sent ends it with agent_unreachable, and the call follows the agent's fallback.

To resume, open a new connection for the same agent_id with the same key. After connection.ready, send one session.resume per session, with its latest token and received, how many of Sent's frames for the session you've read:

{ "type": "session.resume", "session_id": "vs_7Hq2Lm9xQa", "token": "vsr_example_token_000000000000000001", "received": 4 }

Sent answers each one with:

session.resumed carries received, how many of your frames Sent read. Send the later ones again if they still matter: a tool.call Sent never read gets no tool.result. It carries the next resume token while the session is live, and on an audio session the stream number to use on this connection. Once you've sent session.resume, read and send nothing more for the session on the old connection, and nothing on the new one before session.resumed.

Each side counts the frames it sends for a session, and keeps its last 32 to send again. The agent's counted frames are agent.say, tool.call and agent.clear. Sent's are user.said, tool.result and session.end. session.start, session.resume, session.resumed, error and binary audio aren't counted. A frame type you don't know counts when it carries your session's session_id. An agent that streams one reply in more than 32 pieces can outrun the window.

To resume after your own restart, the new process needs each session's token and count, and what the agent knows about the call. An agent that ignores resume keeps working, but when its connection is lost its calls wait window_ms before they follow fallback.

Next steps