{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Module 1 - Inference API Basics\n", "\n", "In this notebook we'll introduce how to use the different APIs available to invoke AI models on Amazon Bedrock - programmatically using Python." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Contents\n", "\n", "1. [Introduction](#intro)\n", "2. [Prerequisites](#prereqs)\n", "3. [Regions and Cross-Region Inference](#regions)\n", "4. [Anthropic Messages API](#anthropic)\n", "5. [OpenAI APIs](#openai)\n", "6. [Amazon Bedrock Converse APIs](#converse)\n", "7. [Amazon Bedrock InvokeModel APIs](#invoke)\n", "8. [Service Tiers](#tiers)\n", "9. [Runtime and Mantle Endpoints: Understanding the Difference](#endpoints)\n", "10. [From APIs to AI Frameworks](#frameworks)\n", "11. [Conclusion and Key Takeaways](#conclusion)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Introduction \n", "\n", "Amazon Bedrock offers a diverse range of AI foundation models:\n", "- From frontier general intelligence models like Anthropic Claude or OpenAI GPT, to smaller, open weight, and specialized models\n", "- From text-based chat, to other data types like audio, images, video, and embedding vectors\n", "- With advanced configuration parameters and model-specific features, to get the best performance for different use cases\n", "\n", "Because of this diversity and the fast pace of frontier AI development, Bedrock offers **multiple APIs** for invoking AI models:\n", "\n", "| API Endpoint | Description |\n", "|:-------------|:------------|\n", "| **Anthropic Messages** | Invoke Claude models through APIs compatible with Anthropic's own [first-party services](https://platform.claude.com/docs/en/api/overview) |\n", "| **OpenAI Responses** | Invoke OpenAI **and** open weight models through OpenAI-compatible [APIs](https://developers.openai.com/api/reference/resources/responses/methods/create) (broadly used as a compatability standard in open source projects like LiteLLM, vLLM, SGLang) |\n", "| **OpenAI Chat Completions** | Invoke open weight models through OpenAI's older [chat API](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create) (also broadly used as a compatability standard) |\n", "| **Bedrock Converse** | Invoke a range of Anthropic and open weight models through standardized [APIs](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html) created by AWS, with support for additional Bedrock features like Guardrails |\n", "| **Bedrock InvokeModel** | Direct, [non-standardized access](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html) to call individual models - useful for advanced model capabilities that aren't offered through more standard APIs |\n", "\n", "All of these API groups support **streaming** responses as they're generated, as well as non-streaming calls that deliver the whole response at once.\n", "\n", "In this module we'll practice the basics of using these different APIs, and also explore three important features for managing response speed and cost:\n", "\n", "| Feature | Description |\n", "|:-------------|:------------|\n", "| **[Cross-Region Inference](https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html)** | Allow Amazon Bedrock to process your request in other AWS Regions under the hood, for more stable latency and in some cases discounted pricing |\n", "| **[Service Tiers](https://docs.aws.amazon.com/bedrock/latest/userguide/service-tiers-inference.html)** | Prioritize speed or cost of inference by choosing different service tiers for each request |\n", "| **[Batch Inference](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html)** | Run a batch job of many inference requests, letting Amazon Bedrock orchestrate the process |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Prerequisites \n", "\n", "You should already have set up your **AWS Account Access** and AWS CLI credentials as described in the workshop prerequisites instructions. The required **Python libraries** for this notebook are detailed in [../pyproject.toml](../pyproject.toml).\n", "\n", "If you're at an event providing temporary AWS Accounts and using the cloud IDE, this setup should already be done for you and the libraries installed at a [uv](https://docs.astral.sh/uv/)-managed virtual environment in the `.venv` folder: Go ahead and run the below cells as-is and when prompted to **select kernel** choose `Python Environments > .venv`.\n", "\n", "If you're using your own local IDE, check you've followed the *Install dependencies* instructions in [../README.md](../README.md).\n", "\n", "Once you're ready, run the cell below to get started:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2\n", "\n", "# Python Built-Ins:\n", "import json\n", "import os\n", "import sys\n", "\n", "# External Dependencies:\n", "import boto3 # AWS SDK for Python\n", "from anthropic import Anthropic\n", "from aws_bedrock_token_generator import provide_token # API key-based auth helper\n", "from dotenv import load_dotenv # .env configuration support for running locally\n", "from IPython.display import Markdown, display # Notebook display utilities\n", "from langchain.agents import create_agent\n", "from langchain.tools import tool\n", "from langchain_openai import ChatOpenAI\n", "from openai import OpenAI\n", "from openai import PermissionDeniedError as OpenAIPermissionDeniedError\n", "\n", "load_dotenv() # Load extra environment variables from .env fle, if present" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Regions and Cross-Region Inference \n", "\n", "The availability of different models and inference APIs varies depending on the [AWS Region](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/) you target, and the [model cards](https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards.html) in the Amazon Bedrock User Guide each have a handy reference for which regions and API endpoints are supported for your target model.\n", "\n", "Some models and APIs also support **[cross-Region inference](https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html)** - specified by adding a prefix (like `global.`, `eu.`, or `us.`) at the start of the model ID. With these cross-Region inference \"profiles\" you still choose a specific \"source\" AWS Region to send your request to... but under the hood, Amazon Bedrock can securely and automatically route your request to be processed in another commercial \"destination\" AWS Region within your target geography.\n", "\n", "For models/APIs that support it, and use-cases whose governance requirements can permit it, **cross-Region inference is generally recommended** above in-Region - because it enables Amazon Bedrock to transparently route requests around local busy periods, delivering faster and more consistent response speeds overall. Many models also offer [discounted pricing](https://aws.amazon.com/bedrock/pricing/) or higher quota limits for cross-Region requests.\n", "\n", "For this notebook, we'll mainly be using models that are available *either* through `us-east-1` or `us-west-2` - including a couple of cross-Region requests identified by their geographical prefix.\n", "\n", "> ℹ️ **Top tip:** By default, boto3 references the `AWS_DEFAULT_REGION` environment variable if a region is not configured in your AWS CLI Profile... ***Not*** `AWS_REGION` as used by some other SDKs!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "AWS_REGION = os.environ.get(\n", " \"AWS_REGION\", os.environ.get(\"AWS_DEFAULT_REGION\", \"us-west-2\")\n", ")\n", "print(f\"Using AWS Region: {AWS_REGION}\")\n", "\n", "if AWS_REGION not in (\"us-east-1\", \"us-west-2\"):\n", " print(\n", " \"⚠️ WARNING: The models and endpoints chosen in this notebook assume that you're \"\n", " \"working in us-east-1 or us-west-2. If you choose to continue with this setting, you \"\n", " \"may see errors.\"\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Anthropic Messages API\n", "\n", "Amazon Bedrock's [Anthropic Messages API endpoint](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-messages-api.html) is compatible with Anthropic's own first-party Messages API, enabling integration with tools like [Claude Code](https://claude.com/product/claude-code) which expect this interface.\n", "\n", "Programmatically, you can use it through the [`anthropic` Python SDK](https://platform.claude.com/docs/en/cli-sdks-libraries/sdks/python) with the base `Anthropic` client (recommended) - by configuring the `base_url` to point to Amazon Bedrock, and providing an appropriate `api_key`.\n", "\n", "Generally, AWS APIs use [AWS Signature v4](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv.html) for authentication as it offers security benefits over plain API keys. Bedrock does provide features to [create short- or long-lived API keys](https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys-generate.html), but ephemeral credentials should be preferred where possible.\n", "\n", "For Python apps, you can automatically generate an ephemeral **API key-like token** from the current AWS credentials using the [`aws-bedrock-token-generator` library](https://pypi.org/project/aws-bedrock-token-generator/) - giving a convenient compromise which supports clients that need API key-style credentials, without needing to create and manage API keys separate from your standard AWS access.\n", "\n", "> **Note:** As documented in the [\"platform integrations\" section](https://platform.claude.com/docs/en/cli-sdks-libraries/sdks/python#platform-integrations) of the Anthropic Python Client SDKs guide, there are actually other Bedrock-connecting clients too:\n", ">\n", "> - `AnthropicBedrock` transparently collects AWS CLI / boto3 credentials, but uses the Amazon Bedrock `InvokeModel` API under the hood, so is less-recommended.\n", "> - `AnthropicBedrockMantle` uses the Anthropic Messages API but under the [Mantle endpoint](https://docs.aws.amazon.com/bedrock/latest/userguide/endpoints.html), which is missing some features like cross-Region inference and structured outputs (with `output_config.format`).\n", "> - `AnthropicAWS` targets [Claude Platform on AWS](https://docs.aws.amazon.com/claude-platform/latest/userguide/welcome.html), which offers additional Anthropic features but operated directly by Anthropic - rather than Amazon Bedrock" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "client = Anthropic(\n", " # Point to Amazon Bedrock:\n", " base_url=f\"https://bedrock-runtime.{AWS_REGION}.amazonaws.com/anthropic\",\n", " # Create temporary key-like token from AWS CLI credentials:\n", " # (More secure than creating a long-lived Amazon Bedrock API key)\n", " api_key=provide_token(region=AWS_REGION),\n", ")\n", "\n", "resp = client.messages.create(\n", " model=\"global.anthropic.claude-haiku-4-5-20251001-v1:0\",\n", " max_tokens=1024,\n", " messages=[\n", " {\"role\": \"user\", \"content\": \"Can you explain the features of Amazon Bedrock?\"},\n", " ],\n", ")\n", "\n", "print(resp)\n", "print(\"----------\")\n", "display(Markdown(\"\\n\\n\".join([c.text for c in resp.content if hasattr(c, \"text\")])))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This same pattern also works with equivalent Anthropic wrappers in frameworks like LangChain (e.g. with [`ChatAnthropic`](https://docs.langchain.com/oss/python/integrations/chat/anthropic)) and Strands ([`AnthropicModel`](https://strandsagents.com/docs/user-guide/concepts/model-providers/anthropic/))." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. OpenAI APIs \n", "\n", "OpenAI's original [Chat Completions API](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create) was adopted as an interoperability standard by many open source LLM gateway and hosting projects - like LiteLLM, vLLM, and SGLang. Since then, OpenAI have moved towards their newer [Responses API](https://developers.openai.com/api/reference/resources/responses/methods/create) - with Chat Completions now positioned as a legacy API.\n", "\n", "At the time of writing, Amazon Bedrock supports the Chat Completions API for all open weight models added since GPT-OSS (2025-09). The Responses API is supported for OpenAI models and all open weight models added since Gemma 4 (2026-06).\n", "\n", "As we saw with the Anthropic Messages API, this means you can call Amazon Bedrock directly from SDKs or tools that support OpenAI - just by configuring the API base URL to Amazon Bedrock instead of the OpenAI default, and providing either an Amazon Bedrock API Key or a generated token from your AWS credentials:\n", "\n", "> ⚠️ **Note for AWS-hosted events:** Unfortunately, temporary AWS Accounts provided for AWS-hosted events may not be able to access the OpenAI GPT models as shown below. If you get a permission denied error here, just carry on to the next cell where we'll show an alternative model instead." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "oai_client = OpenAI(\n", " api_key=provide_token(region=AWS_REGION),\n", " base_url=f\"https://bedrock-runtime.{AWS_REGION}.amazonaws.com/openai/v1\",\n", ")\n", "\n", "try:\n", " resp = oai_client.responses.create(\n", " model=\"us.openai.gpt-5.6-terra\",\n", " input=\"Howdy\",\n", " )\n", "\n", " print(resp)\n", " print(\"----------\")\n", " display(Markdown(resp.output_text))\n", "except OpenAIPermissionDeniedError:\n", " print(\n", " \"❌ Sorry - it looks like this AWS Account doesn't have access to this model.\\n\"\n", " \"Please move along to the next section to see alternatives.\"\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Beyond OpenAI models, many open weight models can also be invoked through Bedrock's OpenAI APIs - creating opportunities to optimize solution cost without needing to switch APIs.\n", "\n", "Note though that:\n", "1. Some open weight models like [Gemma 4](https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards-google.html) may *only* be available on the alternative Mantle endpoint, not the standard Bedrock Runtime.\n", "2. On Mantle endpoints, some older models use a plain `/v1` base URL instead of the full `/openai/v1`\n", "3. Some older models support the older OpenAI Chat Completions API but *not* the newer Responses API. \n", "\n", "For full details on which models are available on the Mantle versus the Runtime endpoints, and which base URL should be used, and which APIs are supported - refer to the [model cards](https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards.html) in the Amazon Bedrock User Guide." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "oai_mantle_client = OpenAI(\n", " api_key=provide_token(region=AWS_REGION),\n", " base_url=f\"https://bedrock-mantle.{AWS_REGION}.api.aws/openai/v1\",\n", ")\n", "\n", "# For this example, let's summarize a paragraph from an AWS Blog post:\n", "# Source: https://aws.amazon.com/jp/blogs/machine-learning/announcing-new-tools-for-building-with-generative-ai-on-aws/\n", "text_to_summarize = \"\"\"\n", "AWS took all of that feedback from customers, and today we are excited to announce Amazon Bedrock, \\\n", "a new service that makes FMs from AI21 Labs, Anthropic, Stability AI, and Amazon accessible via an \\\n", "API. Bedrock is the easiest way for customers to build and scale generative AI-based applications \\\n", "using FMs, democratizing access for all builders. Bedrock will offer the ability to access a range \\\n", "of powerful FMs for text and images—including Amazons Titan FMs, which consist of two new LLMs \\\n", "we're also announcing today—through a scalable, reliable, and secure AWS managed service. With \\\n", "Bedrock's serverless experience, customers can easily find the right model for what they're trying \\\n", "to get done, get started quickly, privately customize FMs with their own data, and easily \\\n", "integrate and deploy them into their applications using the AWS tools and capabilities they are \\\n", "familiar with, without having to manage any infrastructure (including integrations with Amazon \\\n", "SageMaker ML features like Experiments to test different models and Pipelines to manage their FMs \\\n", "at scale).\n", "\"\"\"\n", "\n", "resp = oai_mantle_client.responses.create(\n", " model=\"google.gemma-4-31b\",\n", " input=f\"Provide a concise, 2-3 sentence summary of the following text:\\n{text_to_summarize}\",\n", " store=True,\n", ")\n", "\n", "print(resp)\n", "print(\"----------\")\n", "display(Markdown(resp.output_text))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that while other APIs generally require you to pass in the full conversation history each time for multi-turn conversations, the **Responses API is stateful**: If you configure responses to be stored (as we did above with `store=True`), you can directly continue a conversation by referencing the previous response - as shown below:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "resp2 = oai_mantle_client.responses.create(\n", " model=\"google.gemma-4-31b\",\n", " input=\"Can you make it even shorter?\",\n", " previous_response_id=resp.id,\n", ")\n", "\n", "print(f\"Original: ~{len(resp.output_text.split(' '))} words\")\n", "print(resp)\n", "print(f\"Shortened: ~{len(resp2.output_text.split(' '))} words\")\n", "print(\"----------\")\n", "display(Markdown(resp2.output_text))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The older **Chat Completions API**, by contrast, is **stateless** - like other APIs on Amazon Bedrock. Here, multi-turn conversations require the client to remember the history and pass it back to the model each time - as shown in the example below using the [Kimi K2.5](https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-moonshot-ai-kimi-k2-5.html) model:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "oaic_client = OpenAI(\n", " api_key=provide_token(region=AWS_REGION),\n", " base_url=f\"https://bedrock-mantle.{AWS_REGION}.api.aws/v1\",\n", ")\n", "\n", "resp = oaic_client.chat.completions.create(\n", " model=\"moonshotai.kimi-k2.5\",\n", " messages=[\n", " {\n", " \"role\": \"user\",\n", " \"content\": \"Hi, I've got a maths problem here I'm struggling a bit with - can you help?\",\n", " },\n", " {\n", " \"role\": \"assistant\",\n", " \"content\": \"Hello! I'd be happy to help. Please share, and I'll do my best to assist.\",\n", " },\n", " {\n", " \"role\": \"user\",\n", " \"content\": (\n", " \"Ella sells magazine subscriptions and earns $4 for each new subscriber she signs \"\n", " \"up, plus a fixed $25 per week. If Ella wants to earn at least $58 this week, how \"\n", " \"many subscriptions does she need to sell?\"\n", " ),\n", " },\n", " ],\n", ")\n", "\n", "print(resp)\n", "print(\"----------\")\n", "display(Markdown(resp.choices[0].message.content))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Amazon Bedrock Converse APIs \n", "\n", "Amazon Bedrock's [Converse](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html) and [ConverseStream](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ConverseStream.html) APIs provide a common interface for chat-like models across a range of providers. As native AWS APIs, you can use them through the AWS SDK for your language of choice.\n", "\n", "The Converse API provides some additional features like [integrated guardrail checks](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-input-tagging-base-inference.html) and [cross-region inference](https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html) that aren't currently available through OpenAI and Anthropic-compatible endpoints. However, it's missing support for some models. \n", "\n", "Let's see it in action with a basic request against a global cross-Region inference endpoint:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "bedrock_runtime = boto3.client(\"bedrock-runtime\", region_name=AWS_REGION)\n", "\n", "code_generation_prompt = \"\"\"\n", "Create a Python function called get_weather that accepts a location as a string parameter.\n", "The function should return a dictionary containing weather data (condition, temperature, humidity)\n", "using hardcoded mock values for predefined cities. Include New York, San Francisco, Miami, and\n", "Seattle as default cities. The return statement must be:\n", "`return weather_data.get(location, {\"condition\": \"Unknown\", \"temperature\": 0, \"humidity\": 0})`\n", "\n", "Rules:\n", "- Return ONLY the function definition, nothing else\n", "- No imports of any kind\n", "- No main() function\n", "- No API calls, no SDK usage, no class definitions\n", "- NO preamble, explanation, or markdown formatting (e.g. ``` fences)\n", "\"\"\"\n", "\n", "resp = bedrock_runtime.converse(\n", " modelId=\"global.anthropic.claude-sonnet-4-6\", # Note cross-Region 'global.' prefix\n", " messages=[\n", " {\n", " \"role\": \"user\",\n", " \"content\": [{\"text\": code_generation_prompt}],\n", " }\n", " ],\n", ")\n", "\n", "generated_code = \"\"\n", "for block in resp[\"output\"][\"message\"][\"content\"]:\n", " if \"text\" in block:\n", " generated_code += block[\"text\"]\n", " print(block[\"text\"])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "try:\n", " print(\"Trying to call your LLM-generated function:\\n\")\n", " exec(generated_code + \"\\n\\nprint(get_weather('New York'))\", {})\n", " print(\"✅ Success!\")\n", "except Exception as e:\n", " print(f\"❌ Your LLM-generated code failed to run with error:\\n{e}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Like Anthropic and OpenAI APIs, Amazon Bedrock Converse has a streaming equivalent [ConverseStream](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ConverseStream.html), which you can use to display long outputs live as they're generated, rather than waiting for completion first. For example:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "resp = bedrock_runtime.converse_stream(\n", " modelId=\"global.anthropic.claude-sonnet-4-6\",\n", " messages=[\n", " {\n", " \"role\": \"user\",\n", " \"content\": [{\"text\": \"Tell me a happy story about a cloud?\"}],\n", " }\n", " ],\n", " inferenceConfig={\n", " \"maxTokens\": 1000,\n", " },\n", ")\n", "\n", "# Extract and print the response text in real-time.\n", "for event in resp[\"stream\"]:\n", " if \"contentBlockDelta\" in event:\n", " text = event[\"contentBlockDelta\"].get(\"delta\", {}).get(\"text\", \"\")\n", " # print(text, end=\"\")\n", " sys.stdout.write(text)\n", " sys.stdout.flush()\n", "print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. Amazon Bedrock InvokeModel APIs \n", "\n", "The [InvokeModel](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html) (and equivalent streaming) APIs provide direct access to call models without enforcing a standard API structure, which is particularly useful for models whose use cases don't correspond to typical chat-oriented APIs: Like image or video generation, or speech processing.\n", "\n", "As an example, let's try out [Stability AI Image Services](https://docs.aws.amazon.com/bedrock/latest/userguide/stable-image-services.html) to convert a rough sketch (in [dog-sketch.png](dog-sketch.png)) into a more photorealistic image:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import base64\n", "\n", "from IPython.display import Image\n", "\n", "bedrock_runtime = boto3.client(\"bedrock-runtime\", region_name=AWS_REGION)\n", "\n", "with open(\"dog-sketch.png\", \"rb\") as f:\n", " sketch_bytes = f.read()\n", " sketch_base64 = base64.b64encode(sketch_bytes).decode(\"utf-8\")\n", "resp = bedrock_runtime.invoke_model(\n", " # Note cross-Region 'us.' prefix on model ID:\n", " modelId=\"us.stability.stable-image-control-sketch-v1:0\",\n", " body=json.dumps(\n", " {\n", " \"image\": sketch_base64,\n", " \"prompt\": \"Happy dog with propellor hat and lollipop, in t-shirt and shorts\",\n", " }\n", " ),\n", ")\n", "\n", "resp_body = json.loads(resp[\"body\"].read())\n", "img_b64 = resp_body[\"images\"][0]\n", "image = base64.b64decode(img_b64)\n", "print(f\"Image generated: {len(image):,} bytes ({len(img_b64):,} in base64)\")\n", "display(Image(data=image))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8. Service Tiers \n", "\n", "Amazon Bedrock offers four [service tiers](https://docs.aws.amazon.com/bedrock/latest/userguide/service-tiers-inference.html) to help you optimize for availability, cost, and performance:\n", "\n", "**Reserved tier** provides the ability to reserve prioritized compute capacity for your mission-critical applications that cannot tolerate any downtime - ensuring uninterrupted operations even during local busy periods.\n", "\n", "Within *on-demand inference*, there are also three tiers you can select between on a **per-request basis** with no up-front commitment:\n", "\n", "- **Priority tier** offers queue priority for faster responses during busy periods, in return for a higher price\n", "- **Standard tier** is the default\n", "- **Flex tier** offers discounted pricing, in return for lower queue priority for workloads that can tolerate longer response times\n", "\n", "On-demand service tier is configured by setting the `serviceTier` field on each inference request - meaning a one-line code change to Flex tier can cut 50% (model-dependent) off your inference bill! Here's an example of flex tier in action:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "bedrock_runtime = boto3.client(\"bedrock-runtime\", region_name=AWS_REGION)\n", "\n", "resp = bedrock_runtime.converse_stream(\n", " modelId=\"moonshotai.kimi-k2.5\",\n", " messages=[\n", " {\n", " \"role\": \"user\",\n", " \"content\": [{\"text\": \"What is Amazon Bedrock\"}],\n", " }\n", " ],\n", " inferenceConfig={\n", " \"maxTokens\": 1000,\n", " },\n", " serviceTier={\"type\": \"flex\"},\n", ")\n", "\n", "# Extract and print the response text in real-time.\n", "resolved_tier = None\n", "for event in resp[\"stream\"]:\n", " if \"contentBlockDelta\" in event:\n", " text = event[\"contentBlockDelta\"].get(\"delta\", {}).get(\"text\", \"\")\n", " sys.stdout.write(text)\n", " sys.stdout.flush()\n", " if \"metadata\" in event:\n", " chunk_tier = event[\"metadata\"].get(\"serviceTier\", {}).get(\"type\")\n", " if chunk_tier is not None:\n", " resolved_tier = chunk_tier\n", "print()\n", "print(\"----------\")\n", "print(f\"> Used service tier: {resolved_tier or 'UNKNOWN'}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Best-practices for inference tiers\n", "\n", "For mission-critical applications, Reserved tier provides the best availability and most stable performance in return for an up-front commitment.\n", "\n", "For on-demand inference, Amazon Bedrock manages auto-scaling infrastructure between models in each AWS Region and inference tiers control the *queue priority* of your requests alongside other customers. Some interesting consequences of this include:\n", "\n", "1. During quiet periods, you might see no significant performance benefit from Priority tier!\n", " - Best-practice: Save costs by dynamically dropping back to Standard or Flex when observed latencies are low.\n", "2. When regional demand for a model picks up rapidly, scaling out new copies of a model can still take time and queue priority only helps if there are other requests to de-prioritize.\n", " - Under the hood, each doubling of model capacity in a Region can take on the order of ~15 minutes.\n", " - Especially if you're targeting a less-locally-popular model in your Region, if *your workload* is big enough to be the main source of demand at that time then prioritization can still only work within the constraints of the available capacity during the time it takes for auto-scaling to kick in.\n", " - Best-practice: For big planned demand ramps or less-locally-popular models, consider Reserved tier where possible\n", " - Best-practice: When load testing your applications before production, aim to run for at least 30 minutes to start seeing how latency stabilises as Amazon Bedrock's auto-scaling takes effect.\n", "\n", "For more guidance on load testing, check out our lightweight Python tool [LLMeter from AWSLabs](https://awslabs.github.io/llmeter/)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9. Runtime and Mantle Endpoints: Understanding the Difference \n", "\n", "As shown in some examples above and detailed in the [\"Endpoints\" section](https://docs.aws.amazon.com/bedrock/latest/userguide/endpoints.html) of the Amazon Bedrock User Guide, there are two separate API endpoints available on Amazon Bedrock:\n", "\n", "1. **`bedrock-runtime`** is the recommended endpoint for most new applications, supporting the broadest range of APIs and features like inference logging and cross-Region inference\n", "2. **`bedrock-mantle`** offers some specific models (like [Gemma 4](https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards-google.html)) and in-Region inference options that aren't currently available on Runtime. Amongst other differences, it also has [its own IAM permission space](https://docs.aws.amazon.com/service-authorization/latest/reference/list_bedrock-mantle.html) (`bedrock-mantle:*` instead of `bedrock:*` for Runtime).\n", "\n", "See the [user guide](https://docs.aws.amazon.com/bedrock/latest/userguide/endpoints.html) for full details of supported features that may influence the right choice for your application - and the [Amazon Bedrock model cards](https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards.html) for your target model(s) to understand which endpoints and APIs are supported for each one.\n", "\n", "In particular, note that the \"Mantle endpoint\" is separate from the \"Mantle inference engine\": Both endpoints use the same underlying inference engine and benefit from Mantle's [zero operator access (ZOA)](https://aws.amazon.com/blogs/machine-learning/exploring-the-zero-operator-access-design-of-mantle/) design." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 10. From APIs to AI Frameworks \n", "\n", "Many AI applications are built with higher-level AI frameworks like [Strands Agents](https://strandsagents.com/), [Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview), and [LangGraph](https://www.langchain.com/langgraph) - sometimes through LLM compatability/gateway layers like [LiteLLM](https://pypistats.org/packages/litellm). These frameworks can help to simplify common patterns like [tool use](https://docs.aws.amazon.com/bedrock/latest/userguide/tool-use-client-side.html) and multi-turn memory - rather than working with the inference APIs directly.\n", "\n", "Amazon Bedrock is compatible with all these and more, generally with **multiple possible integrations** depending which underlying API you want to use. For example, with [LangChain](https://docs.langchain.com/build-overview) you can use:\n", "\n", "- [`ChatBedrockConverse`](https://reference.langchain.com/python/langchain-aws/chat_models/bedrock_converse/ChatBedrockConverse) for the Converse API\n", "- [`ChatAnthropic`](https://reference.langchain.com/python/langchain-anthropic/chat_models/ChatAnthropic) with the `base_url` pointing to Bedrock, for the Anthropic Messages API\n", "- [`ChatOpenAI`](https://reference.langchain.com/python/langchain-openai/chat_models/base/ChatOpenAI) with the `base_url` pointing to Bedrock, for the OpenAI Responses or Chat Completions APIs.\n", "\n", "All the APIs support core features like streaming, tool calling, and structured output - so the choice will often depend on which models you're looking to use and whether you need any advanced features (like [integrated guardrail checks](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-input-tagging-base-inference.html)) that are only available through certain APIs.\n", "\n", "In the example **LangChain** code below we'll create and test out a simple agent:\n", "- Using the [Gemma 4 31B model](https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-google-gemma-4-31b.html) by Google...\n", "- ...Through the [OpenAI-compatible Responses API](https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-mantle.html) on Amazon Bedrock...\n", "- ...With custom tools to list what models are available on Amazon Bedrock's endpoints in the current region." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "bedrock = boto3.client(\"bedrock\")\n", "\n", "oai_mantle_client = OpenAI(\n", " api_key=provide_token(region=AWS_REGION),\n", " base_url=f\"https://bedrock-mantle.{AWS_REGION}.api.aws/v1\",\n", ")\n", "\n", "\n", "@tool\n", "def list_runtime_models() -> list[dict]:\n", " \"\"\"List AI models available on the *bedrock-runtime* endpoint in the current AWS Region\n", "\n", " Runtime endpoints operate under the bedrock:* IAM permission namespace, and offer some features\n", " like cross-Region inference and invocation logging that are not available in bedrock-mantle.\n", " \"\"\"\n", " return bedrock.list_foundation_models()[\"modelSummaries\"]\n", "\n", "\n", "@tool\n", "def list_inference_profiles() -> list[dict]:\n", " \"\"\"List AI models available *through cross-Region inference profiles* from this AWS Region\n", "\n", " With cross-Region inference profiles, requests are still submitted against this \"source\" Region,\n", " but are securely routed under-the-hood for processing in one of a range of Regions: Helping\n", " deliver more consistent low latency during locally busy periods.\n", " \"\"\"\n", " return bedrock.list_inference_profiles()[\"inferenceProfileSummaries\"]\n", "\n", "\n", "@tool\n", "def list_mantle_models() -> list[dict]:\n", " \"\"\"List AI models available on the *bedrock-mantle* endpoint in the current AWS Region\n", "\n", " Mantle endpoints operate under the bedrock-mantle:* IAM permission namespace, and offer some\n", " features like server-side tool use, asynchronous inference, and OpenAI-compatible \"projects\"\n", " that are not accessible in bedrock-runtime.\n", " \"\"\"\n", " oai_mantle_client.models.list().to_dict()[\"data\"]\n", "\n", "\n", "llm = ChatOpenAI(\n", " model=\"google.gemma-4-31b\",\n", " base_url=f\"https://bedrock-mantle.{AWS_REGION}.api.aws/openai/v1\",\n", " api_key=provide_token(region=AWS_REGION), # (from AWS CLI/IAM credentials)\n", " use_responses_api=True, # (not Chat Completions)\n", ")\n", "\n", "lc_agent = create_agent(\n", " model=llm, tools=[list_runtime_models, list_mantle_models, list_inference_profiles]\n", ")\n", "\n", "stream = lc_agent.stream_events(\n", " {\n", " \"messages\": [\n", " {\n", " \"role\": \"user\",\n", " \"content\": (\n", " \"Is Kimi K2.5 available in this region? Which endpoints can I use it with?\"\n", " ),\n", " }\n", " ]\n", " },\n", " version=\"v3\",\n", ")\n", "\n", "for message in stream.messages:\n", " for delta in message.text:\n", " print(delta, end=\"\", flush=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 11. Conclusion and Key Takeaways \n", "\n", "Amazon Bedrock exposes AI models through **multiple different APIs**, which enables a wide range of integrations and gives users fine-grained controls to optimize applications:\n", "\n", "- The **[Anthropic-compatible](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-messages-api.html) and [OpenAI-compatible APIs](https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-mantle.html)** are particularly useful when connecting from existing systems that already expect one of those formats - for example using coding assistants like Claude Code or OpenAI Codex with models on Amazon Bedrock.\n", "- The **[native Converse APIs](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html)** offers portability across model providers, and some additional features like guardrail integration.\n", "- The **[native InvokeModel APIs](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html)** offer direct access to underlying model parameters, which is particularly useful for working with data modalities that aren't yet well-standardized in other APIs (like speech models, or image generation).\n", "\n", "The **[model cards](https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards.html)** in the Amazon Bedrock User Guide provide a handy reference for which APIs and AWS Regions are supported for different models. All real-time APIs support streaming options as well as synchronous responses.\n", "\n", "Besides the different API formats themselves, we also showed some important features for optimizing how models are invoked:\n", "\n", "- **[Cross-Region inference](https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html)** allows Amazon Bedrock to offer more consistent low latency during busy periods (and in many cases a pricing discount) by automatically routing request processing to other regions under the hood, controlled by a geographical model ID prefix like `global.`, `us.`, `eu.`, or etc.\n", "- **[Inference tiers](https://docs.aws.amazon.com/bedrock/latest/userguide/service-tiers-inference.html)** (including Reserved tier and the on-demand options Priority, Standard, and Flex) give users control to prioritize between latency-critical and cost-sensitive workloads.\n", "\n", "### Optional Extension: Batch Inference\n", "\n", "As well as the real-time APIs shown in this notebook, Amazon Bedrock also supports **[batch inference jobs](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html)**. You can use these jobs to orchestrate a batch of requests asynchronously, removing the need to manage individual requests and retries.\n", "\n", "We didn't include a batch example in this notebook because **⚠️ batch jobs are not supported in temporary Accounts provided for AWS-hosted events**. However, if you're following along in your own AWS Account and keen to explore the [example notebooks here](https://github.com/aws-samples/amazon-bedrock-samples/tree/main/introduction-to-bedrock/batch_api).\n", "\n", "### Next Steps\n", "\n", "With this core understanding of the different ways to call models and manage cost/speed/availability trade-offs, you're ready to dive deeper into how to build AI-powered applications or manage cross-team platform governance on Amazon Bedrock. Let's continue on to the next module!" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.13" } }, "nbformat": 4, "nbformat_minor": 4 }