# Realtime

<Note type="warning">Realtime is an experimental feature.</Note>

The AI SDK provides realtime models for bidirectional audio and text
conversations over WebSockets. Realtime sessions run in the browser and connect
directly to the provider using a short-lived token that you create on your
server.

The typical flow is:

1. The browser calls your setup endpoint.
1. Your server creates a short-lived provider token with `experimental_realtime.getToken()`.
1. The browser opens a WebSocket connection to the provider.
1. The model streams audio, text, and tool calls back to the browser.
1. Tool calls are handled by your application with `onToolCall`.

## Setup Endpoint

Create a setup endpoint that returns a short-lived token for the realtime
provider. This endpoint can also attach tool definitions to the session.

```ts filename='app/api/realtime/setup/route.ts'
import { openai } from '@ai-sdk/openai';
import { experimental_getRealtimeToolDefinitions, tool } from 'ai';
import { z } from 'zod';

const tools = {
  getWeather: tool({
    description: 'Get the current weather for a city',
    inputSchema: z.object({
      city: z.string().describe('The city to get weather for'),
    }),
  }),
};

export async function POST(request: Request) {
  const body = await request.json().catch(() => ({}));
  const toolDefinitions = await experimental_getRealtimeToolDefinitions({
    tools,
  });

  const token = await openai.experimental_realtime.getToken({
    model: 'gpt-realtime',
    sessionConfig: {
      ...body.sessionConfig,
      tools: toolDefinitions,
    },
  });

  return Response.json({
    ...token,
    tools: toolDefinitions,
  });
}
```

<Note>
  In production, authenticate and rate-limit your setup endpoint. It creates
  provider sessions using your server-side API key.
</Note>

## Client Session

Use the `experimental_useRealtime` hook to connect to the provider, capture microphone audio,
play model audio, send text messages, and render messages.

```tsx filename='app/realtime/page.tsx'
'use client';

import { openai } from '@ai-sdk/openai';
import { experimental_useRealtime } from '@ai-sdk/react';

export default function RealtimePage() {
  const realtime = experimental_useRealtime({
    model: openai.experimental_realtime('gpt-realtime'),
    api: {
      token: '/api/realtime/setup',
    },
    sessionConfig: {
      instructions: 'You are a helpful assistant. Be concise.',
      inputAudioTranscription: {},
      voice: 'alloy',
      turnDetection: { type: 'server-vad' },
    },
  });

  return (
    <div>
      <button onClick={realtime.connect}>Connect</button>
      <button onClick={realtime.disconnect}>Disconnect</button>

      {realtime.messages.map(message => (
        <div key={message.id}>
          <strong>{message.role}</strong>
          {message.parts.map((part, index) =>
            part.type === 'text' ? <span key={index}>{part.text}</span> : null,
          )}
        </div>
      ))}
    </div>
  );
}
```

## Tool Calling

Realtime tool execution is client-driven. The provider sends tool calls over the
WebSocket, and your application handles them with `onToolCall`. If the result is
available immediately, return it from `onToolCall`. The SDK sends it back to the
provider as tool output.

For server-backed tools, call an app-specific API endpoint from `onToolCall`.
Avoid generic "execute tool by name" routes. App-specific endpoints are easier
to secure because they can use your normal authentication, authorization,
validation, and rate limiting rules.

### Server-Backed Tool Endpoint

```ts filename='app/api/weather/route.ts'
import { z } from 'zod';

const inputSchema = z.object({
  city: z.string(),
});

export async function POST(request: Request) {
  const input = inputSchema.safeParse(await request.json());

  if (!input.success) {
    return Response.json({ error: 'Invalid input' }, { status: 400 });
  }

  return Response.json({
    city: input.data.city,
    temperature: 72,
    condition: 'sunny',
  });
}
```

### Client Tool Handler

```tsx filename='app/realtime/page.tsx' highlight="10-24"
import { openai } from '@ai-sdk/openai';
import { experimental_useRealtime } from '@ai-sdk/react';

export default function RealtimePage() {
  const realtime = experimental_useRealtime({
    model: openai.experimental_realtime('gpt-realtime'),
    api: {
      token: '/api/realtime/setup',
    },
    onToolCall: async ({ toolCall }) => {
      if (toolCall.toolName === 'getWeather') {
        const response = await fetch('/api/weather', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify(toolCall.args),
        });

        if (!response.ok) {
          throw new Error('Weather lookup failed');
        }

        return response.json();
      }
    },
  });

  // ...
}
```

You can also submit tool output manually with `addToolOutput` when the tool
requires user interaction or another asynchronous process:

```tsx
realtime.addToolOutput(toolCallId, {
  approved: true,
});
```

## Supported Providers

Realtime models are available on providers that expose realtime WebSocket APIs:

```ts
import { openai } from '@ai-sdk/openai';
import { google } from '@ai-sdk/google';
import { xai } from '@ai-sdk/xai';

const openaiModel = openai.experimental_realtime('gpt-realtime');
const googleModel = google.experimental_realtime(
  'gemini-3.1-flash-live-preview',
);
const xaiModel = xai.experimental_realtime('grok-voice-latest');
```

You can also route realtime through the [AI Gateway](/providers/ai-sdk-providers/ai-gateway#realtime), which normalizes the
session so the same client code works across upstream providers:

```ts
import { gateway } from '@ai-sdk/gateway';

const gatewayModel = gateway.experimental_realtime('openai/gpt-realtime-2');
```

Gateway realtime v0 is server-side only and throws if used in a browser. It uses
the resolved Gateway auth token for the WebSocket upgrade rather than a
provider-minted short-lived client secret. See the
[AI Gateway realtime docs](/providers/ai-sdk-providers/ai-gateway#realtime) for
the Gateway-specific token and subprotocol details.


## Navigation

- [Overview](/v7/docs/ai-sdk-core/overview)
- [Generating Text](/v7/docs/ai-sdk-core/generating-text)
- [Generating Structured Data](/v7/docs/ai-sdk-core/generating-structured-data)
- [Tool Calling](/v7/docs/ai-sdk-core/tools-and-tool-calling)
- [Model Context Protocol (MCP)](/v7/docs/ai-sdk-core/mcp-tools)
- [MCP Apps](/v7/docs/ai-sdk-core/mcp-apps)
- [Runtime and Tool Context](/v7/docs/ai-sdk-core/runtime-and-tool-context)
- [Prompt Engineering](/v7/docs/ai-sdk-core/prompt-engineering)
- [Settings](/v7/docs/ai-sdk-core/settings)
- [Reasoning](/v7/docs/ai-sdk-core/reasoning)
- [Embeddings](/v7/docs/ai-sdk-core/embeddings)
- [Reranking](/v7/docs/ai-sdk-core/reranking)
- [Image Generation](/v7/docs/ai-sdk-core/image-generation)
- [Realtime](/v7/docs/ai-sdk-core/realtime)
- [Transcription](/v7/docs/ai-sdk-core/transcription)
- [Speech](/v7/docs/ai-sdk-core/speech)
- [Video Generation](/v7/docs/ai-sdk-core/video-generation)
- [File Uploads](/v7/docs/ai-sdk-core/file-uploads)
- [Language Model Middleware](/v7/docs/ai-sdk-core/middleware)
- [Skill Uploads](/v7/docs/ai-sdk-core/skill-uploads)
- [Provider & Model Management](/v7/docs/ai-sdk-core/provider-management)
- [Error Handling](/v7/docs/ai-sdk-core/error-handling)
- [Testing](/v7/docs/ai-sdk-core/testing)
- [Telemetry](/v7/docs/ai-sdk-core/telemetry)
- [DevTools](/v7/docs/ai-sdk-core/devtools)
- [Lifecycle Callbacks](/v7/docs/ai-sdk-core/lifecycle-callbacks)


[Full Sitemap](/sitemap.md)