Realtime | AI SDK

Realtime is an experimental feature.

This guide covers legacy token-based, turn-based realtime conversations over WebSockets. These sessions run in the browser and connect directly to the provider using a short-lived token that you create on your server. You can also route the connection through AI Gateway.

For OpenAI Live's continuous JSON/PCM16 WSS relay runtime and application-handled client delegation, see experimental_useRealtime.

The typical flow is:

The browser calls your setup endpoint.
Your server creates a short-lived realtime token with experimental_realtime.getToken().
The browser opens a WebSocket connection to the provider or AI Gateway.
The model streams audio, text, and tool calls back to the browser.
Tool calls are handled by your application with onToolCall.

For continuous OpenAI Live conversations, use an application-owned WebSocket relay or optional WebRTC via api.session. Live uses client delegation: your application handles delegated work and submits context rather than using the turn-based tool loop below. See the experimental_useRealtime reference for SDP setup, server-owned permissions, capture ownership, and graceful close.

Setup Endpoint

Create a setup endpoint that returns a short-lived token for the realtime provider. This endpoint can also attach tool definitions to the session.

app/api/realtime/setup/route.ts

import { openai } from '@ai-sdk/openai';
import { experimental_getRealtimeToolDefinitions, tool } from 'ai';
import { z } from 'zod';

const tools = {
  getWeather: tool({
    description: 'Get the current weather for a city',
    inputSchema: z.object({
      city: z.string().describe('The city to get weather for'),
    }),
  }),
};

export async function POST(request: Request) {
  const body = await request.json().catch(() => ({}));
  const toolDefinitions = await experimental_getRealtimeToolDefinitions({
    tools,
  });

  const token = await openai.experimental_realtime.getToken({
    model: 'gpt-realtime',
    sessionConfig: {
      ...body.sessionConfig,
      tools: toolDefinitions,
    },
  });

  return Response.json({
    ...token,
    tools: toolDefinitions,
  });
}

In production, authenticate and rate-limit your setup endpoint. It creates realtime sessions using your server-side API key.

AI Gateway

Use AI Gateway when you want the same realtime client code to work across supported upstream providers. The Gateway normalizes realtime events server-side, and the browser still receives only a short-lived client secret.

Create the short-lived Gateway realtime token from a server-side setup endpoint:

app/api/realtime/setup/route.ts

import { gateway } from 'ai';

export async function POST() {
  const token = await gateway.experimental_realtime.getToken({
    model: 'openai/gpt-realtime-2',
  });

  return Response.json(token);
}

Then use the matching Gateway realtime model in the browser:

app/realtime/page.tsx

'use client';

import { experimental_useRealtime } from '@ai-sdk/react';
import { gateway } from 'ai';

const model = gateway.experimental_realtime('openai/gpt-realtime-2');
const sessionConfig = {
  instructions: 'You are a helpful assistant. Be concise.',
  inputAudioTranscription: {},
  voice: 'alloy',
  turnDetection: { type: 'server-vad' as const },
};

export default function RealtimePage() {
  const realtime = experimental_useRealtime({
    model,
    api: {
      token: '/api/realtime/setup',
    },
    sessionConfig,
  });

  // ...
}

gateway.experimental_realtime.getToken() must run on your server because it uses your Gateway credential to mint a vcst_ client secret. Creating the realtime model with gateway.experimental_realtime() is safe in the browser.

Tool definitions work the same way with AI Gateway: convert AI SDK tools with experimental_getRealtimeToolDefinitions() in your setup endpoint and return the definitions alongside the token. The hook includes them in the session update after the WebSocket opens.

Client Session

Use the experimental_useRealtime hook to connect to a realtime model, capture microphone audio, play model audio, send text messages, and render messages.

app/realtime/page.tsx

'use client';

import { openai } from '@ai-sdk/openai';
import { experimental_useRealtime } from '@ai-sdk/react';

const model = openai.experimental_realtime('gpt-realtime');
const sessionConfig = {
  instructions: 'You are a helpful assistant. Be concise.',
  inputAudioTranscription: {},
  voice: 'alloy',
  turnDetection: { type: 'server-vad' as const },
};

export default function RealtimePage() {
  const realtime = experimental_useRealtime({
    model,
    api: {
      token: '/api/realtime/setup',
    },
    sessionConfig,
  });

  return (
    <div>
      <button onClick={realtime.connect}>Connect</button>
      <button onClick={realtime.disconnect}>Disconnect</button>

      {realtime.messages.map(message => (
        <div key={message.id}>
          <strong>{message.role}</strong>
          {message.parts.map((part, index) =>
            part.type === 'text' ? <span key={index}>{part.text}</span> : null,
          )}
        </div>
      ))}
    </div>
  );
}

Keep model and session configuration objects stable across renders. Use module scope as above, or useMemo when configuration depends on props. Replacing either object replaces the hook's session.

Tool Calling

Realtime tool execution is client-driven. The provider sends tool calls over the WebSocket, and your application handles them with onToolCall. If the result is available immediately, return it from onToolCall. The SDK sends it back to the provider as tool output.

For server-backed tools, call an app-specific API endpoint from onToolCall. Avoid generic "execute tool by name" routes. App-specific endpoints are easier to secure because they can use your normal authentication, authorization, validation, and rate limiting rules.

Server-Backed Tool Endpoint

app/api/weather/route.ts

import { z } from 'zod';

const inputSchema = z.object({
  city: z.string(),
});

export async function POST(request: Request) {
  const input = inputSchema.safeParse(await request.json());

  if (!input.success) {
    return Response.json({ error: 'Invalid input' }, { status: 400 });
  }

  return Response.json({
    city: input.data.city,
    temperature: 72,
    condition: 'sunny',
  });
}

Client Tool Handler

app/realtime/page.tsx

import { openai } from '@ai-sdk/openai';
import { experimental_useRealtime } from '@ai-sdk/react';

const model = openai.experimental_realtime('gpt-realtime');

export default function RealtimePage() {
  const realtime = experimental_useRealtime({
    model,
    api: {
      token: '/api/realtime/setup',
    },
    onToolCall: async ({ toolCall }) => {
      if (toolCall.toolName === 'getWeather') {
        const response = await fetch('/api/weather', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify(toolCall.args),
        });

        if (!response.ok) {
          throw new Error('Weather lookup failed');
        }

        return response.json();
      }
    },
  });

  // ...
}

You can also submit tool output manually with addToolOutput when the tool requires user interaction or another asynchronous process:

realtime.addToolOutput(toolCallId, {
  approved: true,
});

Supported Providers

Realtime models are available on providers that expose realtime WebSocket APIs:

import { openai } from '@ai-sdk/openai';
import { google } from '@ai-sdk/google';
import { xai } from '@ai-sdk/xai';

const openaiModel = openai.experimental_realtime('gpt-realtime');
const googleModel = google.experimental_realtime(
  'gemini-3.1-flash-live-preview',
);
const xaiModel = xai.experimental_realtime('grok-voice-latest');

You can also route realtime through the AI Gateway, which normalizes the session so the same client code works across upstream providers:

import { gateway } from '@ai-sdk/gateway';

const gatewayModel = gateway.experimental_realtime('openai/gpt-realtime-2');

gateway.experimental_realtime.getToken() mints a short-lived Gateway client secret on your server. The browser uses that token to open the Gateway WebSocket; the SDK handles the Gateway-specific WebSocket subprotocols for you. See the AI Gateway realtime docs for Gateway-specific token and provider option details.