Heroku Provider

Heroku is a cloud platform that allows you to deploy and run applications, including AI models with OpenAI API compatibility. You can deploy models that are OpenAI API compatible and use them with the AI SDK.

Setup

The Heroku provider is available via the @ai-sdk/openai-compatible module as it is compatible with the OpenAI API. You can install it with

pnpm

npm

yarn

pnpm add @ai-sdk/openai-compatible

Heroku Setup

Create a test app in Heroku:

heroku create

Inference using claude-3-5-haiku:

heroku ai:models:create -a $APP_NAME claude-3-5-haiku

Export Variables:

export INFERENCE_KEY=$(heroku config:get INFERENCE_KEY -a $APP_NAME)
export INFERENCE_MODEL_ID=$(heroku config:get INFERENCE_MODEL_ID -a $APP_NAME)
export INFERENCE_URL=$(heroku config:get INFERENCE_URL -a $APP_NAME)

Provider Instance

To use Heroku, you can create a custom provider instance with the createOpenAICompatible function from @ai-sdk/openai-compatible:

import { createOpenAICompatible } from '@ai-sdk/openai-compatible';

const heroku = createOpenAICompatible({
  name: 'heroku',
  baseURL: process.env.INFERENCE_URL + '/v1',
  apiKey: process.env.INFERENCE_KEY,
});

Be sure to have your INFERENCE_KEY, INFERENCE_MODEL_ID, and INFERENCE_URL set in your environment variables.

Language Models

You can create Heroku models using a provider instance. The first argument is the served model name, e.g. claude-3-5-haiku.

const model = heroku('claude-3-5-haiku');

Example

You can use Heroku language models to generate text with the generateText function:

import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
import { generateText } from 'ai';

const heroku = createOpenAICompatible({
  name: 'heroku',
  baseURL: process.env.INFERENCE_URL + '/v1',
  apiKey: process.env.INFERENCE_KEY,
});

const { text } = await generateText({
  model: heroku('claude-3-5-haiku'),
  prompt: 'Tell me about yourself in one sentence',
});

console.log(text);

Heroku language models are also able to generate text in a streaming fashion with the streamText function:

import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
import { streamText } from 'ai';

const heroku = createOpenAICompatible({
  name: 'heroku',
  baseURL: process.env.INFERENCE_URL + '/v1',
  apiKey: process.env.INFERENCE_KEY,
});

const result = streamText({
  model: heroku('claude-3-5-haiku'),
  prompt: 'Tell me about yourself in one sentence',
});

for await (const message of result.textStream) {
  console.log(message);
}

Heroku language models can also be used in the generateObject, and streamObject functions.