`experimental_useRealtime()`

experimental_useRealtime is an experimental feature.

Creates a browser-side realtime session for bidirectional audio and text conversations with a realtime provider model.

The hook connects to a realtime WebSocket using a short-lived token from your setup endpoint, returns messages as UIMessage[], and provides controls for audio capture, playback, text input, and tool output.

import { openai } from '@ai-sdk/openai';
import { experimental_useRealtime } from '@ai-sdk/react';

const realtime = experimental_useRealtime({
  model: openai.experimental_realtime('gpt-realtime'),
  api: {
    token: '/api/realtime/setup',
  },
});

For AI Gateway, pass gateway.experimental_realtime(...) as the model and point api.token at a server-side setup endpoint that calls gateway.experimental_realtime.getToken().

Import

import { experimental_useRealtime } from "@ai-sdk/react"

API Signature

Parameters

model:

Experimental_RealtimeModel

api:

{ token: string }

Object

token:

string

sessionConfig?:

Partial<Experimental_RealtimeSessionConfig>

sampleRate?:

number

maxEvents?:

number

onToolCall?:

(options: { toolCall: { toolCallId: string; toolName: string; args: unknown } }) => unknown | Promise<unknown> | undefined

onEvent?:

(event: Experimental_RealtimeServerEvent) => void

onError?:

(error: Error) => void

Returns

status:

'disconnected' | 'connecting' | 'connected' | 'error'

messages:

UIMessage[]

events:

Experimental_RealtimeServerEvent[]

isCapturing:

boolean

isPlaying:

boolean

connect:

() => Promise<void>

disconnect:

() => void

addToolOutput:

(callId: string, result: unknown) => void

sendEvent:

(event: Experimental_RealtimeClientEvent) => void

sendTextMessage:

(text: string) => void

sendAudio:

(base64Audio: string) => void

commitAudio:

() => void

clearAudioBuffer:

() => void

requestResponse:

(options?: { modalities?: string[] }) => void

cancelResponse:

() => void

startAudioCapture:

(stream: MediaStream) => void

stopAudioCapture:

() => void

stopPlayback:

() => void

Tool Calling

Realtime tool execution is client-driven. Use onToolCall to handle tool calls and return the tool output:

const realtime = experimental_useRealtime({
  model: openai.experimental_realtime('gpt-realtime'),
  api: {
    token: '/api/realtime/setup',
  },
  onToolCall: async ({ toolCall }) => {
    if (toolCall.toolName === 'getWeather') {
      const response = await fetch('/api/weather', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify(toolCall.args),
      });

      return response.json();
    }
  },
});

For tools that require user interaction, return undefined from onToolCall and call addToolOutput later.

See Realtime for a complete example with server-backed app-specific tool endpoints.