4.4.6. Managing Prompts and Testing AI Workloads in CI/CD
💡 First Principle: A prompt is code. It changes model behaviour as much as a weight update does, so it needs the same controls: a single source of truth, immutable versions, automated tests before release, and a rollback that does not require an application redeploy.
Amazon Bedrock Prompt Management stores prompts as managed resources:
- A prompt has one or more variants (message, model, inference parameters) and variables written as
{{variable}}. The console can compare up to three variants side by side. - Saving edits the draft. Creating a version takes an immutable snapshot. Applications call a specific version by ARN, for example
arn:aws:bedrock:us-west-2:123456789012:prompt/PROMPT12345:1. - At runtime you pass that version ARN as the
modelIdofConverseorInvokeModeland fill the variables throughpromptVariables. When you do, the request may not includesystem,inferenceConfig,toolConfigoradditionalModelRequestFields, because those are owned by the prompt version. The caller needsbedrock:RenderPrompton the prompt resource. - The
CHATtemplate type (required for prompt caching) works with Converse-capable models;TEXTworks everywhere.
Testing generative AI in a pipeline. Unit tests cannot assert an exact string from a stochastic model, so AI tests assert properties against a fixed evaluation dataset:
| Test layer | Tooling | Gate example |
|---|---|---|
| Prompt regression | Prompt Management variants plus a golden dataset run in CodeBuild | Candidate prompt version must not score below the current version |
| Model quality | Amazon Bedrock evaluations: programmatic, human, or LLM-as-a-judge | Correctness and faithfulness above threshold |
| RAG quality | Bedrock RAG evaluation (retrieve-only or retrieve-and-generate) with ground truth | Context relevance and citation coverage above threshold |
| Agent behaviour | AgentCore Evaluations on sessions, traces and spans | Tool-selection accuracy and goal success above threshold |
AWS CodeConnections (formerly AWS CodeStar Connections) links CodePipeline, CodeBuild and CodeDeploy to GitHub, GitHub Enterprise Server, GitLab (including self-managed), Bitbucket Cloud and Azure DevOps by connection ARN, so no personal access token is stored in the pipeline. A connection created with the AWS CLI or CloudFormation starts in PENDING and must be completed in the console, which performs the OAuth handshake with the provider; only then is it AVAILABLE.
⚠️ Exam Trap: A pipeline that fails at the source stage right after a CloudFormation deployment of a new connection is not an IAM problem. The connection is still PENDING; someone must complete it in the Developer Tools console. The CLI cannot complete a pending connection.
