AWS Bedrock

Anthropic, Amazon Titan, A121 LLMs are Supported on Bedrock

Pre-Requisites

LiteLLM requires boto3 to be installed on your system for Bedrock requests

pip install boto3>=1.28.57

Required Environment Variables

os.environ["AWS_ACCESS_KEY_ID"] = ""  # Access key
os.environ["AWS_SECRET_ACCESS_KEY"] = "" # Secret access key
os.environ["AWS_REGION_NAME"] = "" # us-east-1, us-east-2, us-west-1, us-west-2

Usage

import os 
from litellm import completion

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

response = completion(
  model="anthropic.claude-instant-v1", 
  messages=[{ "content": "Hello, how are you?","role": "user"}]
)

Usage - Streaming

import os 
from litellm import completion

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

response = completion(
  model="anthropic.claude-instant-v1", 
  messages=[{ "content": "Hello, how are you?","role": "user"}],
  stream=True
)
for chunk in response:
  print(chunk)

Example Streaming Output Chunk

{
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": "ase can appeal the case to a higher federal court. If a higher federal court rules in a way that conflicts with a ruling from a lower federal court or conflicts with a ruling from a higher state court, the parties involved in the case can appeal the case to the Supreme Court. In order to appeal a case to the Sup"
      }
    }
  ],
  "created": null,
  "model": "anthropic.claude-instant-v1",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}

Boto3 - Authentication

Passing credentials as parameters - Completion()

Pass AWS credentials as parameters to litellm.completion

import os 
from litellm import completion

response = completion(
            model="anthropic.claude-instant-v1", 
            messages=[{ "content": "Hello, how are you?","role": "user"}],
            aws_access_key_id="",
            aws_secret_access_key="",
            aws_region_name="",
)

Passing an external BedrockRuntime.Client as a parameter - Completion()

Pass an external BedrockRuntime.Client object as a parameter to litellm.completion. Useful when using an AWS credentials profile, SSO session, assumed role session, or if environment variables are not available for auth.

Create a client from session credentials:

import boto3
from litellm import completion

bedrock = boto3.client(
            service_name="bedrock-runtime",
            region_name="us-east-1",
            aws_access_key_id="",
            aws_secret_access_key="",
            aws_session_token="",
)

response = completion(
            model="anthropic.claude-instant-v1",
            messages=[{ "content": "Hello, how are you?","role": "user"}],
            aws_bedrock_client=bedrock,
)

Create a client from AWS profile in ~/.aws/config:

import boto3
from litellm import completion

dev_session = boto3.Session(profile_name="dev-profile")
bedrock = dev_session.client(
            service_name="bedrock-runtime",
            region_name="us-east-1",
)

response = completion(
            model="anthropic.claude-instant-v1",
            messages=[{ "content": "Hello, how are you?","role": "user"}],
            aws_bedrock_client=bedrock,
)

Supported AWS Bedrock Models

Here's an example of using a bedrock model with LiteLLM

Model Name	Command
Anthropic Claude-V2	completion(model='anthropic.claude-v2', messages=messages)
Anthropic Claude-Instant V1	completion(model='anthropic.claude-instant-v1', messages=messages)
Anthropic Claude-V1	completion(model='anthropic.claude-v1', messages=messages)
Amazon Titan Lite	completion(model='amazon.titan-text-lite-v1', messages=messages)
Amazon Titan Express	completion(model='amazon.titan-text-express-v1', messages=messages)
Cohere Command	completion(model='cohere.command-text-v14', messages=messages)
AI21 J2-Mid	completion(model='ai21.j2-mid-v1', messages=messages)
AI21 J2-Ultra	completion(model='ai21.j2-ultra-v1', messages=messages)
Meta Llama 2 Chat 13b	completion(model='meta.llama2-13b-chat-v1', messages=messages)
Meta Llama 2 Chat 70b	completion(model='meta.llama2-70b-chat-v1', messages=messages)

Bedrock Embedding

API keys

This can be set as env variables or passed as params to litellm.embedding()

import os
os.environ["AWS_ACCESS_KEY_ID"] = ""        # Access key
os.environ["AWS_SECRET_ACCESS_KEY"] = ""    # Secret access key
os.environ["AWS_REGION_NAME"] = ""           # us-east-1, us-east-2, us-west-1, us-west-2

Usage

from litellm import embedding
response = embedding(
    model="amazon.titan-embed-text-v1",
    input=["good morning from litellm"],
)
print(response)

Supported AWS Bedrock Embedding Models

Model Name	Function Call
Titan Embeddings - G1	embedding(model="amazon.titan-embed-text-v1", input=input)