Introduction (DRAFT)

Think of Amazon Bedrock Runtime as a chef's kitchen stocked with the finest ingredients—Foundation Models (FMs) are those premium ingredients. But just as a chef needs to understand the unique properties of each ingredient to create a masterpiece, you must understand the distinct capabilities of each FM. This chapter serves as your culinary school, diving deep into the models available in Bedrock Runtime. We'll explore their strengths, weaknesses, and how to choose the right one for your AI creations—much like knowing when to use a delicate truffle versus a robust portobello mushroom.

Model Selection Strategies

Selecting the right Foundation Model (FM) is key for AI applications. Like choosing tools for a job, each FM has specific strengths. This section covers how to pick the best FM by evaluating performance, cost, and speed.

We'll show you how to assess your needs, test models, match them to use cases, analyze costs, understand versions, and work with streaming outputs.

Defining Your Requirements: Identifying Key Performance Indicators (KPIs)

First, define clear goals and success metrics (KPIs) for your project before picking a model. For example, a chatbot might measure response accuracy and customer satisfaction, while financial analysis would focus on accuracy and processing speed.

Well-defined KPIs provide an objective way to evaluate Foundation Models.

Evaluating Model Performance: Accuracy, Latency, and Cost

After defining KPIs, evaluate models on accuracy, latency, and cost. Check AWS docs for baseline metrics, then validate with testing.

BLEU Score Overview: BLEU evaluates machine translation quality by comparing output against reference translations using n-gram matching.

BLEU scores range from 0 to 1 (0-100%), where:

Note: BLEU focuses on word matching but may miss semantic accuracy. Use with other metrics for complete evaluation.

Understanding F1-Score: The F1-score combines precision and recall into a single metric that measures model accuracy. It helps find the optimal balance between these two metrics.