4.4.7. Automating FM, Agent and Knowledge Base Releases
💡 First Principle: Every layer of a generative AI application must be releasable on its own schedule and reversible on its own. Achieve that by giving each layer an immutable version and a movable pointer, then letting the pipeline move pointers only after tests pass.
| Layer | Immutable unit | Movable pointer | Rollback |
|---|---|---|---|
| Fine-tuned model (SageMaker AI) | Model package version in a model package group | Approval status plus endpoint config | Redeploy the previous approved version |
| Fine-tuned model (Bedrock) | Custom model | Custom model deployment or Provisioned Throughput | Point the application at the previous deployment ARN |
| Prompt | Prompt version | Version ARN held in application config | Change config to the previous version ARN |
| AgentCore agent | Runtime version (created on every update) | Named endpoint; DEFAULT always tracks the latest | Update the production endpoint to the previous version |
| Agents Classic agent | Agent version | Alias | Update the alias to the previous version |
| Knowledge base content | Source documents in S3 (versioned bucket) | Ingestion job | Restore the object version and re-sync |
Fine-tuned model versioning. Register every fine-tuning output as a new version in the SageMaker Model Registry, with evaluation metrics attached, and gate deployment on approval status. MLflow on SageMaker AI can register models automatically, but Model Registry sync is opt-in: enable it on the MLflow tracking server or MLflow App (UpdateMlflowTrackingServer or UpdateMlflowApp), and from then on registering an MLflow model creates the matching model package group and version in the Model Registry.
Agent release pipelines. Updating an AgentCore Runtime (new container image, protocol or network change) creates a new immutable version and moves only the DEFAULT endpoint. A safe pipeline deploys, runs evaluations against the new version through a staging endpoint, then calls UpdateAgentRuntimeEndpoint to move the production endpoint. Existing sessions keep running the code they started with until they end.
Knowledge base refresh orchestration. Syncing is incremental: a StartIngestionJob re-processes only documents added, modified or deleted since the last sync, and a metadata-only change can skip re-embedding. Typical triggers:
- A schedule (EventBridge Scheduler) or an S3 upload event starts a Step Functions workflow that calls
StartIngestionJob, pollsGetIngestionJobuntilCOMPLETE, then runs a RAG evaluation before announcing the refresh. - Urgent single-document fixes use direct ingestion (
IngestKnowledgeBaseDocuments), which indexes documents without a full sync. For an S3 data source, also write the change to S3, or the next sync overwrites it, and never run direct ingestion andStartIngestionJobat the same time. - A change of chunking strategy or embeddings model is not a refresh: it requires a new data source or knowledge base, a full ingestion, evaluation, and a switch of the knowledge base ID the application uses (blue/green for RAG).
⚠️ Exam Trap: If a question says documents were updated in S3 but answers are still stale, the missing step is a sync (ingestion job), not a new embeddings model or a larger vector store. If it says the team changed the chunk size and nothing improved, remember chunking cannot be changed on an existing data source.
