HeyGen

The HeyGen provider supports heygen-video-1, a video generation model with text-to-video, image-to-video, and reference-to-video modes. Generated clips include audio. It also supports HeyGen Decision models, which classify, check, and score text, images, and video. You can use HeyGen models through AI Gateway (recommended) or connect directly with @ai-sdk/heygen.

import { experimental_generateVideo as generateVideo } from 'ai';
import { writeFile } from 'node:fs/promises';

const { video } = await generateVideo({
  model: "heygen/heygen-video-1",
  prompt: 'A paper boat drifting across a calm lake at sunrise.',
  duration: 5,
  aspectRatio: '16:9',
});

await writeFile('video.mp4', video.uint8Array);

Using AI Gateway

The AI SDK uses AI Gateway by default for model strings. Gateway provides access to models across providers through a single API key, with usage and cost tracking. Install ai:

$ pnpm add ai

Set an AI Gateway API key in your environment:

export AI_GATEWAY_API_KEY=your_api_key_here

Vercel deployments can also use OIDC authentication. See the AI Gateway provider documentation for configuration and the model catalog for available models.

Gateway feature and provider-option support depends on the model and the provider serving the request. See Gateway provider options.

Using @ai-sdk/heygen

For direct access to HeyGen, install ai and @ai-sdk/heygen:

$ pnpm add ai @ai-sdk/heygen

Set your HeyGen API credential in the HEYGEN_API_KEY environment variable:

export HEYGEN_API_KEY=your_api_key_here

AI Gateway does not expose every API in @ai-sdk/heygen. Use the direct provider for provider-specific APIs or custom connection settings.

The following sections document the direct @ai-sdk/heygen provider, including its model factories and settings. Gateway model and feature availability can differ.

This provider supports video generation and decision models. HeyGen's avatar, voice, translation, and video editing APIs are not included.

Provider Instance

Import the default instance:

import { heygen } from '@ai-sdk/heygen';

Or create a customized instance:

import { createHeyGen } from '@ai-sdk/heygen';

const heygen = createHeyGen({
  apiKey: process.env.HEYGEN_API_KEY,
});

The following settings are optional:

  • apiKey string | PromiseLike<string> | (() => string | PromiseLike<string>): The key sent in the x-api-key header. Defaults to the HEYGEN_API_KEY environment variable.

    Pass a function to resolve credentials before each request, including when keys rotate. The function can return a string or a promise. Credential caching and refresh are managed by the function.

  • baseURL string: The API URL prefix. Defaults to https://api.heygen.com.

  • headers Record<string, string>: Additional request headers.

  • fetch typeof fetch: A custom fetch implementation.

Video Models

Create a model with heygen.video('heygen-video-1') or heygen.videoModel('heygen-video-1').

Text to Video

import { heygen } from '@ai-sdk/heygen';
import { experimental_generateVideo as generateVideo } from 'ai';

const { videos } = await generateVideo({
  model: heygen.video('heygen-video-1'),
  prompt:
    'A paper boat floats down a quiet stream. Static camera, soft morning light, the sound of flowing water. No music.',
  duration: 5,
  aspectRatio: '16:9',
});

Image to Video

Pass an image in the prompt to use it as the first frame. The video follows the image's aspect ratio; a supplied aspectRatio is ignored with a warning.

const { videos } = await generateVideo({
  model: heygen.video('heygen-video-1'),
  prompt: {
    text: 'The subject moves slowly. Preserve the composition and natural lighting.',
    image: 'https://example.com/first-frame.jpg',
  },
  duration: 5,
});

A single first_frame in frameImages is also supported. Last-frame conditioning and multiple first frames are rejected.

Reference to Video

Use inputReferences to condition a new scene on images or videos. Include a mediaType for explicit URL references so the provider can select the reference kind.

const { videos } = await generateVideo({
  model: heygen.video('heygen-video-1'),
  prompt: 'The product in <Picture 1> sits on a desk beside a sunlit window.',
  inputReferences: [
    {
      data: 'https://example.com/product.jpg',
      mediaType: 'image/jpeg',
    },
  ],
  duration: 5,
  aspectRatio: '16:9',
});

References are grouped by media type, preserving order within each group. Address them in the prompt as <Picture 1>, <Video 1>, and <Audio 1>. Provider-option references are appended after standard references of the same kind.

Reference-to-video requires at least one image or video. Audio-only conditioning is not supported. The adapter accepts up to 9 image references, 3 video references, and 3 audio references, with a combined maximum of 12. Reference videos share a 15-second input budget across the request. Footage within that budget is used in full; longer references are allocated and trimmed by HeyGen. See the reference-video rules.

Provider Options

import { heygen, type HeyGenVideoModelOptions } from '@ai-sdk/heygen';
import { experimental_generateVideo as generateVideo } from 'ai';

const { videos } = await generateVideo({
  model: heygen.video('heygen-video-1'),
  prompt: 'The product in <Picture 1> rotates slowly on a table. No music.',
  aspectRatio: '16:9',
  providerOptions: {
    heygen: {
      resolution: '1080p',
      promptEnhancement: 'disabled',
      referenceImages: [{ type: 'asset_id', assetId: 'YOUR_ASSET_ID' }],
    } satisfies HeyGenVideoModelOptions,
  },
});
  • mode 'text_to_video' | 'image_to_video' | 'reference_to_video': Inferred from the inputs when omitted. Explicit modes must match the supplied inputs. First-frame images cannot be combined with references.
  • resolution '480p' | '768p' | '1080p' | '2k': Output size class. Defaults to 768p. Takes precedence over the standard resolution option.
  • promptEnhancement 'turbo' | 'quality' | 'ultraquality' | 'disabled': HeyGen's prompt enhancement setting. The API defaults to quality.
  • image HeyGenVideoAsset: First-frame input, including an existing asset ID. Cannot be combined with a standard image or frameImages.
  • referenceImages HeyGenVideoAsset[]: Additional image references.
  • referenceVideos HeyGenVideoAsset[]: Additional video references.
  • referenceAudio HeyGenVideoAsset[]: Additional audio references.

HeyGenVideoAsset supports these representations:

{ type: 'url', url: 'https://example.com/input.jpg' }
{ type: 'asset_id', assetId: 'YOUR_ASSET_ID' }
{ type: 'base64', mediaType: 'image/jpeg', data: 'BASE64_DATA' }

Asset IDs must already exist in the workspace associated with the API key. This provider does not upload or manage HeyGen assets. Standard file inputs are sent inline as base64. HeyGen permits up to 5 MB per inline image and 16 MB per inline video or audio input. URL inputs must use HTTPS and be directly accessible without redirects; HeyGen permits up to 16 MB per image URL and 32 MB per video or audio URL.

Resolution and Aspect Ratio

Prefer providerOptions.heygen.resolution to select a size class. The standard resolution option accepts the exact frame sizes below and infers the corresponding aspect ratio when aspectRatio is omitted:

Aspect ratio480p768p1080p2k
21:9960x4161536x672——
16:9832x4801344x7681890x10802688x1536
4:3640x4801024x768——
1:1480x480768x768——
3:4480x640768x1024——
9:16480x832768x13441080x18901536x2688

Text-to-video defaults to 16:9. Reference-to-video defaults to adaptive, inheriting the first reference image or video's shape. At 1080p or 2k, text-to-video and reference-to-video require 16:9 or 9:16. Image-to-video always inherits the first frame's shape, even when a standard frame size is supplied; only its size class is used.

Other Constraints

  • Prompts must contain between 1 and 32,000 characters.
  • The SDK accepts positive whole-second durations and forwards them unchanged. HeyGen currently supports 5–30 seconds for text-to-video and image-to-video with quality or turbo prompt enhancement, and 5–15 seconds for reference-to-video or ultraquality/disabled enhancement. The default duration is 5. The API enforces these limits; see duration limits.
  • Seeds must be unsigned 32-bit integers.
  • One video is generated per provider call.
  • Output is MP4 at 24 fps with generated audio. Unsupported fps values and generateAudio: false produce warnings.

Asynchronous Generation

The provider implements doStart and doStatus. The AI SDK orchestrates polling for generateVideo. Native webhook completion is not implemented; polling is used.

HeyGen jobs continue running after submission. Aborting a local request does not cancel the remote generation. The returned video URL is signed; retrieving the same operation's status again refreshes the URL.

Pass an Idempotency-Key through the standard headers option when retrying the same submission. HeyGen retains keys for 24 hours; concurrent duplicates return an HTTP 409 error.

Provider Metadata

providerMetadata.heygen.videos contains one entry per completed video. The AI SDK combines these entries when multiple videos are requested. Each entry contains:

  • videoId: The HeyGen video identifier.
  • mode: The generation mode submitted to HeyGen.
  • resolution: The resolved size class submitted to HeyGen, retained across asynchronous status calls. This is separate from the actual output dimensions.
  • duration, width, height, aspectRatio, and seed: Values reported by HeyGen on completion, when available.
  • timings.inference: The inference timing reported by HeyGen, when available.

Submission and in-progress operation metadata exposes videoId, mode, and resolution directly under providerMetadata.heygen. Failed or cancelled operations also expose failureCode when HeyGen reports it.

The current HeyGen video endpoint does not document token counts or credit consumption in its generation response. The provider returns the reported duration and dimensions without estimating those quantities. Reported inference timing is preserved per video.

Decision Models

Use heygen.decisionModel() with experimental_decide to call the HeyGen Decision API. This API is experimental and may change in patch releases.

import { heygen } from '@ai-sdk/heygen';
import { experimental_decide as decide } from 'ai';

const result = await decide({
  model: heygen.decisionModel('heygen-decision-1'),
  state:
    'Video export fails for everyone on our team after today’s update. We have no workaround.',
  questions: {
    team: {
      type: 'choice',
      instructions: 'Which support team should handle this request?',
      criteria: {
        video: 'Video creation, rendering, or export problems',
        account: 'Sign-in, access, or workspace administration',
        billing: 'Payments, invoices, or subscription charges',
      },
    },
    blocked: {
      type: 'boolean',
      instructions: 'Is the customer unable to work without a workaround?',
    },
    impact: {
      type: 'score',
      instructions: 'How much does this issue affect the customer’s work?',
      criteria: ['Cosmetic', 'Slower, with a workaround', 'Blocked'],
    },
  },
});

result.answers.blocked.probability; // P(true), from 0 to 1
result.answers.team.choice; // 'video' | 'account' | 'billing'
result.answers.impact.score; // Expected zero-based level, from 0 to 2

heygen-decision-1 and heygen-decision-1-turbo are available. The provider maps SDK boolean questions to HeyGen's native noul type and returns Choice and Score probability distributions without rounding. Box questions are unsupported.

Images and video

Add image and video file parts to state. The provider sends them as HeyGen assets and keeps text and JSON parts in the native state. HeyGen reads images first, then videos, then the state, each in request order, so describe each input's role in the text by its order:

import { readFileSync } from 'node:fs';

const result = await decide({
  model: heygen.decisionModel('heygen-decision-1'),
  state: [
    {
      type: 'text',
      text: 'The first image is the reference product; the video is the unboxing.',
    },
    {
      type: 'file',
      mediaType: 'image/png',
      data: readFileSync('reference.png'),
    },
    {
      type: 'file',
      mediaType: 'video/mp4',
      data: readFileSync('unboxing.mp4'),
    },
  ],
  questions: {
    matches: {
      type: 'boolean',
      instructions: 'Does the unboxed product match the reference product?',
    },
  },
});

Inline file data is sent as base64. When calling the model directly, HTTPS URL file data is passed to HeyGen as a URL, and a provider reference such as { type: 'reference', reference: { heygen: assetId } } is sent as an uploaded HeyGen asset. experimental_decide downloads URL file state before calling the provider.

Confidence, legends, and response metadata

HeyGen's confidence values and Score legends are available in providerMetadata.heygen, keyed by question ID. Confidence is distinct from the selected option's probability. Per-answer diagnostic warnings returned by HeyGen are available at providerMetadata.heygen.answerWarnings.

const metadata = result.providerMetadata?.heygen;
console.log(metadata?.confidence); // Choice and Score confidence by question ID
console.log(metadata?.legend); // Score index-to-description maps by question ID
console.log(metadata?.answerWarnings); // Diagnostic warnings by question ID
console.log(metadata?.requestId); // HeyGen request ID for support
console.log(result.response.id); // HeyGen decision ID
console.log(result.usage.inputTokens);

Decision limits

The provider validates these limits before sending the decision request:

InputLimit
Questions1–400; names nonempty, without colons, newlines, or surrounding whitespace
Choice options1–255
Score levels1–10, all non-null
MediaAt most 4 images and 1 video; HTTPS URLs only

See HeyGen's media limits and context limits for API-enforced file-size and token limits. HeyGen decision models do not accept provider-specific options, maxOutputTokens, or reasoning; these produce unsupported warnings and are not sent to the API.

Decision Model Capabilities

ModelBooleanChoiceScoreBoxImage InputReasoningMax Output Tokens
heygen-decision-1
heygen-decision-1-turbo

Model Capabilities

ModelImage InputFirst FrameLast FrameReference ImagesReference VideoAudioAspect RatioResolutionDurationFPSSeed
heygen-video-1Conditional: Audio is always generated; disabling it is unsupported.Conditional: Supported outside reference-to-video mode.Conditional: Supported outside reference-to-video mode.Conditional: Fixed at 24 fps.

Provider Option Reference

The following option definitions are generated from the provider catalog.

OperationOptionDescription
videomodeInferred from the supplied image and references when omitted.
videoresolutionOutput size class. Takes precedence over the standard resolution option.
videopromptEnhancementDefaults to quality. Use disabled to preserve the original prompt.
videoimageFirst frame, including an existing HeyGen asset. Cannot be combined with a standard image.
videoreferenceImagesAdditional references, appended after standard inputReferences of the same kind.
videoreferenceVideosreferenceVideos
videoreferenceAudioreferenceAudio