Building Generative AI Applications with Bedrock

Introduction#

Amazon Bedrock is a service for utilizing foundation models offered by AWS. This section reviews the basics of Bedrock, as well as general concepts applicable to any large language models.

Bedrock and Foundation Models#

A Foundation Model refers to a large pre-trained neural network, often trained on massive datasets / compute inaccessible to individual developers. Foundation models are typically used as the starting point for a given machine learning project (e.g., image or text generation), and fine-tuned to fit the task at hand as appropriate.

Amazon Bedrock is a unified API for using generative AI foundation models from multiple third-party providers (e.g., Mistral, Anthropic, etc.). It offers four different endpoint types:

• bedrock: manage, deploy, and train models.
• bedrock-runtime: perform inference using these models.
• bedrock-agent: manage, deploy, and train LLM agents using knowledge bases.
• bedrock-agent-runtime: perform inference against LLM agents and knowledge bases.

Bedrock should be used with an IAM role (as opposed to a root account). Note that this is best practice for AWS usage, regardless of the particular application.

Credentials#

There are two primary methods for accessing Amazon Bedrock:

• IAM Roles: preferred for production, built on principle of least privilege.
• API Keys: bearer-token-style method to call Bedrock APIs. Use for exploration / development only; prefer IAM for production.

Fine-Tuning#

Fine-Tuning is the process of adapting a foundation model by continuing training on a smaller, specialized dataset (relative to the original training dataset). Low-Rank Adaptation (LoRA) is one particular strategy which freezes the original model weights and trains smaller, low-rank matrices.

In some cases, fine-tuning may cause a model to lose already-learned capabilities. This phenomenon is known as catastrophic forgetting. This is most likely to happen when the new training data is narrow or very different from the original dataset.

Model Distillation#

Model Distillation transfers skills from a large “teacher” foundation model to a smaller, faster “student” model. The teacher model generates high-quality synthetic responses, which the student is fine-tuned on. The smaller student model may have lower inference cost and latency without sacrifice to task-specific quality.

Evaluation#

How can we evaluate the performance of a foundation model?

• Benchmark Dataset: set of sample prompts and responses generated by SMEs. Used to measure similarity of generated response to expert response.
• Evaluator Model: second model with vetted performance evaluates generated responses.
• Human Feedback: although subjective, human raters may grade LLM responses in terms of various components (e.g., accuracy, relevance, etc.).

What specific evaluation metrics exist for LLMs?

• ROUGE: counts amount of overlapping units between predicted output (generated text) and ground-truth output. Analogous to recall for text.
  • Units may refer to words, n-grams, etc. ROUGE-N is the overlap on n-grams.
  • Primarily used for text summarization and machine translation.
• BLEU: measures precision of n-grams; also incorporates brevity penalty.
  • Mostly used for machine translation.
• BERTScore: matches token-level contextual embeddings, and computes classification metrics based on these matches.

Bedrock Model Evaluations provides automated model evaluations, with many built-in options for metrics and benchmark datasets. Additionally, synthetic user workflows simulate end-to-end usage of your application for deployment validation. Metrics of interest may include hallucination rate, semantic drift, faithfulness, compliance. etc.

Bedrock Guardrails#

Guardrails is a content filtering tool used for prompts and responses in combination with text-based foundation models. It is particularly useful for word / topic filtering, as well as PII removal / masking.

RAG Fundamentals with Bedrock#

RAG Overview#

Retrieval Augmented Generation (RAG) queries an external database for answers to gather context prior to generating output via an LLM. It augments the original prompt by retrieving and inserting relevant information from a database, then submits the modified query to the foundation model. This implies that RAG does NOT involve training a model.

RAG typically relies on a vector database to store specialized information. Recall that an embedding is a semantic representation of some type of data - for example, a word embedding attempts to capture the meaning of the word via latent features. A vector database simply stores your data alongside their computed embedding vectors. Vector dimensionality is a trade-off between cost and representation power; larger vectors may be more effective at capturing information, but require more storage space.

Retrieval from a vector database proceeds as follows:

1. Compute an embedding vector for your prompt $\rightarrow$ query vector.
2. Compute similarity between the query vector and all items in the vector database. Similarity is defined in terms of some mathematical function (e.g., cosine similarity ~ normalized dot product of embeddings).
3. Return the top-n most similar items.

RAG implementations might consider using AWS Lambda functions (with appropriate event triggers / batching) to update the vector database as the underlying knowledge base evolves.

RAG in Bedrock#

What does a RAG pipeline within Bedrock look like? First, we need to establish the knowledge base. After uploading documents or structured data from S3 into a Bedrock Knowledge Base, an embedding model is used to generate vector embeddings, which are then transferred to a vector store. Users control embedding aspects such as…

• Embedding Dimensionality: number of latent features within each vector.
• Chunking: amount of tokens represented by each vector.

OpenSearch is the primary choice of vector store in AWS, but other options exist.

• Amazon S3 Vectors: create an S3 vector bucket and vector index. Store vector embeddings with put_vectors, and query them with query_vectors.
  • Cost-efficient option (reduces total cost by up to 90%), but with high latency.
• Amazon Aurora: pgvector extension creates a new vector column type, adding vector operators to traditional SQL. Appropriate for small / medium RAG systems with primarily structured data.

Advanced RAG Strategies#

We can break up RAG into different components, and optimize each of these components as appropriate.

Pre-Retrieval refers to any methods employed prior to the retrieval stage in a RAG pipeline. This might involve alternative approaches for indexing, data extraction, or chunking. Compared to using a fixed token window, more advanced chunking strategies may be informed via semantics (through the use of a foundation model) or organize chunks in a hierarchical fashion.

Re-Ranker Models attempt to improve the relevance of retrieved results from your knowledge base. After initial retrieval (with semantic search) gathers the initial chunks, the re-ranker model calculates relevance of each chunk to the query and re-orders the results.

Agentic Workflows#

Agentic AI refers to the use of an LLM to automate tasks and decision making. It grants an LLM agent with tools to perform particular actions.

Multi-Agent Systems#

While more basic agentic systems are built around a central LLM, multi-agent systems intertwine multiple different (specialized or parallelized) agents to perform some larger task. Multi-agent systems are useful in the case of 1) many tools which require specialization, or 2) complicated logic warranting a dedicated agentic workflow.

Common multi-agent patterns include:

• Orchestrating: orchestrator LLM is responsible for decomposing tasks and delegating to worker LLMs. Synthesizer LLM combines results across workers into final output.
• Routing: router LLM fields tasks and chooses one of many specialized agents for redirection.
• Parallelizing: similar to orchestrating, but does not have a dedicated orchestrator LLM.
• Chaining: sequence of specialized LLMs; appropriate when a task has a discrete sequence of well-defined steps.

Short and Long-Term Memory#

How do agentic systems maintain memory?

• Short-Term: immediate context; chat history within a given session. Might also be events or actions taken within the session.
• Long-Term: structured information derived from agent interactions and stored in dedicated data store (e.g., DynamoDB, SQLite, RDS).

AgentCore#

Amazon AgentCore is the dedicated AWS service for deployment and operation of AI agents at scale. It is a serverless platform compatible with many agentic frameworks, including the OpenAI Agent SDK, LangGraph / LangChain, and so on.

Other miscellaneous aspects of AgentCore include:

• AgentCore Memory: serverless solution for maintaining memory records, which store information extracted from agent interactions in a structured manner.
• AgentCore Policies: intercept tool calls and evaluate against defined rules, proceeding if reasonable.
• AgentCore Evaluations: measure the performance of your agentic system, and integrate with visualization tools in CloudWatch.
  • Evaluations rely on the use of another “judge” LLM.
  • Built-in prompts for correctness, conciseness, coherence, goal success rate, etc.

AgentCore also has various failure modes:

• Coordination Failures: multi-agent workflow stalls or loses state during handoff.
• Truncated Streaming: streaming response interrupted during generation.
• Tool Failures: gateway tool call errors or times out.

Your implementation should monitor and account for these failure types to ensure proper functionality in production.

MCP#

Model Context Protocol (MCP) is a standardized interface for interactions between LLM agents and tools. In other words, it enables users to access external tools with consistent interaction patterns. Almost all agentic SDKs have the ability to call external MCP servers, where servers may offer particular tools, resources, or prompts.

(all information obtained from AWS Certified Machine Learning Engineer Associate: Hands On! course on Udemy)