3.4.5. Reproducible GenAI Experiments: MLflow on SageMaker AI, Bedrock Evaluations and Prompt Management
💡 First Principle: An experiment is reproducible when someone else can rerun it and get the same comparison. For a generative AI system that means pinning four things: the prompt version, the model and inference parameters, the dataset, and the scoring method. Change one at a time, record all four every time.
| Tool | What it pins | Key capabilities |
|---|---|---|
| Managed MLflow on SageMaker AI | Runs: parameters, metrics, artifacts, models | Experiment tracking and comparison in the MLflow UI; tracing of inputs, outputs and metadata at each step of a GenAI application; MLflow Model Registry with automatic registration to SageMaker Model Registry; available as MLflow Apps or MLflow Tracking Servers, with artifacts stored in your own S3 bucket |
| Amazon Bedrock Prompt Management | The prompt | Reusable prompts with variables, variants to compare side by side, immutable versions you deploy to applications; the prompt carries its model and inference configuration |
| Amazon Bedrock evaluations | The scoring method and dataset | Repeatable evaluation jobs over a fixed prompt dataset: programmatic metrics, LLM-as-a-judge, human workers, and RAG evaluations; results written to S3 |
A reproducible workflow:
- Save the candidate prompt in Prompt Management and create a version (versions are snapshots; the draft keeps changing, the version does not).
- Run a Bedrock evaluation job with the same prompt dataset for every candidate — prompt variant, model, or knowledge base configuration.
- Log the prompt version, model ID, inference parameters, dataset location and evaluation scores as an MLflow run, so candidates can be compared in one place.
- Promote the winner by pointing the application at the new prompt version — a change you can roll back by pointing at the old one.
MLflow tracking servers come in Small, Medium and Large sizes, chosen by expected users and transactions per second; MLflow Apps are the newer managed option in selected Regions.
⚠️ Exam Trap: Prompt Management stores and versions prompts; it does not score them. Bedrock evaluations score outputs; they do not version prompts. MLflow tracks and compares runs across both. A stem asking how to "roll back to the previous prompt" wants Prompt Management versions; one asking how to "compare accuracy across models on a fixed dataset" wants an evaluation job; one asking how to "track and compare all experiment runs, parameters and metrics" wants MLflow.
⚠️ Exam Trap: Hard-coding prompts in application code is the anti-pattern here: every prompt change becomes a code deployment and there is no record of which prompt produced which evaluation result.
