AWS for AI/Agent Developers — Day 6: CI/CD for AI Agents with CodePipeline

Over the last 5 days we built:

✅ Day 1 — MCP server on ECS Fargate (compute)
✅ Day 2 — DynamoDB Global Tables (state)
✅ Day 3 — ElastiCache + Bedrock (caching)
✅ Day 4 — Lambda + Bedrock (serverless)
✅ Day 5 — Route53 multi-region (routing)

Now we tie it together with CI/CD. Because shipping agent updates by hand is not production.

Today: an automated deployment pipeline that takes code from git push to a running agent in multiple regions, with smoke tests, blue/green deployments, and one-click rollback.

1
                    ┌─────────────────────────────────────┐
2
                    │         Source (GitHub)              │
3
                    │         main branch push             │
4
                    └────────────────┬────────────────────┘
5
                                     │
6
                                     ▼
7
                    ┌─────────────────────────────────────┐
8
                    │         CodeBuild                    │
9
                    │  npm ci → npm run build →            │
10
                    │  docker build → docker push ECR      │
11
                    └────────────────┬────────────────────┘
12
                                     │
13
                                     ▼
14
                    ┌─────────────────────────────────────┐
15
                    │         Staging Deploy               │
16
                    │  ECS new task definition             │
17
                    │  → deploy to staging cluster          │
18
                    └────────────────┬────────────────────┘
19
                                     │
20
                                     ▼
21
                    ┌─────────────────────────────────────┐
22
                    │         Smoke Tests                  │
23
                    │  /health → tool call → LLM response  │
24
                    └────────────────┬────────────────────┘
25
                                     │
26
                          ┌──────────┴──────────┐
27
                          ▼                     ▼
28
              ┌────────────────────┐  ┌────────────────────┐
29
              │    us-east-1       │  │   eu-west-1        │
30
              │   Blue/Green ECS   │  │  Blue/Green ECS    │
31
              │   CloudFront inval │  │  Route53 failover  │
32
              └────────────────────┘  └────────────────────┘

The Full Pipeline Architecture#

Beyond basic CI/CD, agent pipelines need:

Agent-specific smoke tests — not just “does it answer 200”, but “is the LLM responding coherently?”
Model/config versioning — LLM models change, prompts change, tool definitions change
Canary deployments — release to 5% of traffic first, observe, then roll out
Automatic rollback — if error rate spikes, revert within minutes
Config promotion — prompt templates, cache policies, model IDs flow through deployment stages

Step 1: buildspec.yml#

1
version: 0.2
2

3
env:
4
  parameter-store:
5
    GITHUB_TOKEN: "/agent/github-token"
6
    SSE_SECRET: "/agent/sse-shared-secret"
7

8
phases:
9
  install:
10
    runtime-versions:
11
      nodejs: 20
12
    commands:
13
      - npm ci
14
      - npm run build
15

16
  pre_build:
17
    commands:
18
      - echo "Logging in to Amazon ECR..."
19
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $ECR_REPOSITORY_URI
20
      - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-8)
21
      - IMAGE_TAG=$COMMIT_HASH-$(date +%Y%m%d%H%M%S)
22

23
  build:
24
    commands:
25
      - echo "Building Docker image..."
26
      - docker build -t $ECR_REPOSITORY_URI:latest .
27
      - docker tag $ECR_REPOSITORY_URI:latest $ECR_REPOSITORY_URI:$IMAGE_TAG
28

29
  post_build:
30
    commands:
31
      - echo "Pushing Docker images..."
32
      - docker push $ECR_REPOSITORY_URI:latest
33
      - docker push $ECR_REPOSITORY_URI:$IMAGE_TAG
34
      - echo "Generating imagedefinitions.json..."
35
      - printf '[
36
          {"name":"mcp-server","imageUri":"%s:%s"},
37
          {"name":"mcp-server-cache","imageUri":"%s:%s"}
38
        ]' $ECR_REPOSITORY_URI $IMAGE_TAG $ECR_REPOSITORY_URI $IMAGE_TAG > imagedefinitions.json
39
      - cat imagedefinitions.json
40
      - echo "Saving image tag for later stages..."
41
      - echo $IMAGE_TAG > /tmp/image-tag.txt
42

43
artifacts:
44
  files:
45
    - imagedefinitions.json
46
    - appspec.yaml
47
    - taskdef.json
48
    - /tmp/image-tag.txt
49
  discard-paths: yes

Step 2: Agent-Specific Smoke Tests#

1
version: 0.2
2

3
phases:
4
  test:
5
    commands:
6
      - echo "Running agent smoke tests..."
7
      - |
8
        set -e
9

10
        # Test 1: Health Check
11
        echo "=== Test 1: Health check ==="
12
        STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://staging.agent.yourdomain.com/health)
13
        if [ "$STATUS" != "200" ]; then
14
          echo "FAIL: Health check returned $STATUS"
15
          exit 1
16
        fi
17
        echo "PASS: Health check OK"
18

19
        # Test 2: Agent responds
20
        echo "=== Test 2: Agent responds to prompt ==="
21
        RESPONSE=$(curl -s -X POST https://staging.agent.yourdomain.com/agent \
22
          -H "Content-Type: application/json" \
23
          -d '{"prompt": "Say hello in one word", "sessionId": "smoke-test-1"}')
24
        echo "Response: $RESPONSE"
25

26
        if ! echo "$RESPONSE" | grep -q "hello\|Hello\|hi\|Hi"; then
27
          echo "FAIL: Agent did not respond to basic prompt"
28
          exit 1
29
        fi
30
        echo "PASS: Agent responds to prompts"
31

32
        # Test 3: Tool execution
33
        echo "=== Test 3: Tool execution ==="
34
        TOOL_RESPONSE=$(curl -s -X POST https://staging.agent.yourdomain.com/agent \
35
          -H "Content-Type: application/json" \
36
          -d '{"prompt": "List issues in ptminh-kmp/test-repo", "sessionId": "smoke-test-2"}')
37
        echo "Tool response: $TOOL_RESPONSE"
38

39
        if echo "$TOOL_RESPONSE" | grep -q "error"; then
40
          echo "WARN: Tool execution had issues, but continuing..."
41
        fi
42

43
        # Test 4: Session persistence
44
        echo "=== Test 4: Session persistence ==="
45
        SESSION_ID="smoke-test-$(date +%s)"
46
        curl -s -X POST https://staging.agent.yourdomain.com/agent \
47
          -d "{\"prompt\":\"My name is TestAgent\",\"sessionId\":\"$SESSION_ID\"}" > /dev/null
48
        PERSIST=$(curl -s -X POST https://staging.agent.yourdomain.com/agent \
49
          -d "{\"prompt\":\"What is my name?\",\"sessionId\":\"$SESSION_ID\"}")
50
        echo "Session test: $PERSIST"
51

52
        if ! echo "$PERSIST" | grep -qi "testagent"; then
53
          echo "WARN: Session persistence may not be working"
54
        else
55
          echo "PASS: Session persistence OK"
56
        fi
57

58
        # Test 5: Cache is working
59
        echo "=== Test 5: Cache hit ==="
60
        START=$(date +%s%N)
61
        curl -s -X POST https://staging.agent.yourdomain.com/agent \
62
          -d '{"prompt":"What is MCP in 10 words?","sessionId":"smoke-test-cache"}' > /dev/null
63
        curl -s -X POST https://staging.agent.yourdomain.com/agent \
64
          -d '{"prompt":"What is MCP in 10 words?","sessionId":"smoke-test-cache"}' > /dev/null
65
        END=$(date +%s%N)
66
        LATENCY=$(( ($END - $START) / 1000000 ))
67
        echo "Second call latency: ${LATENCY}ms"
68

69
        if [ "$LATENCY" -lt 500 ]; then
70
          echo "PASS: Cache appears to be working"
71
        else
72
          echo "INFO: Cache may not be configured, but continuing"
73
        fi
74

75
        echo ""
76
        echo "=== All smoke tests passed ==="

Step 3: appspec.yaml for Blue/Green#

1
version: 0.0
2

3
Resources:
4
  - TargetService:
5
      Type: AWS::ECS::Service
6
      Properties:
7
        TaskDefinition: "<TASK_DEFINITION>"
8
        LoadBalancerInfo:
9
          ContainerName: "mcp-server"
10
          ContainerPort: 3001
11

12
Hooks:
13
  - BeforeInstall: "CodeDeployHook_BeforeInstall"
14
  - AfterInstall: "CodeDeployHook_AfterInstall"
15
  - AfterAllowTestTraffic: "CodeDeployHook_SmokeTest"
16
  - BeforeAllowTraffic: "CodeDeployHook_BeforeAllowTraffic"
17
  - AfterAllowTraffic: "CodeDeployHook_AfterAllowTraffic"
18

19
  - BeforeAllowTraffic: "Lambda-ValidateAgent"

Step 4: Pipeline with Blue/Green#

1
# pipeline.yaml — Full CodePipeline for agent deployments
2

3
Resources:
4
  AgentPipeline:
5
    Type: AWS::CodePipeline::Pipeline
6
    Properties:
7
      Name: agent-deployment-pipeline
8
      RoleArn: !GetAtt PipelineRole.Arn
9
      ArtifactStore:
10
        Type: S3
11
        Location: !Ref ArtifactBucket
12

13
      Stages:
14
        # ── Stage 1: Source ──
15
        - Name: Source
16
          Actions:
17
            - Name: Source
18
              ActionTypeId:
19
                Category: Source
20
                Owner: ThirdParty
21
                Provider: GitHub
22
                Version: 1
23
              Configuration:
24
                Owner: ptminh-kmp
25
                Repo: agent-infrastructure
26
                Branch: main
27
                OAuthToken: !Ref GitHubOAuthToken
28
              OutputArtifacts:
29
                - Name: SourceOutput
30

31
        # ── Stage 2: Build ──
32
        - Name: Build
33
          Actions:
34
            - Name: Build
35
              ActionTypeId:
36
                Category: Build
37
                Owner: AWS
38
                Provider: CodeBuild
39
                Version: 1
40
              Configuration:
41
                ProjectName: !Ref AgentBuildProject
42
              InputArtifacts:
43
                - Name: SourceOutput
44
              OutputArtifacts:
45
                - Name: BuildOutput
46
              RunOrder: 1
47

48
        # ── Stage 3: Deploy to Staging ──
49
        - Name: DeployStaging
50
          Actions:
51
            - Name: DeployToStaging
52
              ActionTypeId:
53
                Category: Deploy
54
                Owner: AWS
55
                Provider: ECS
56
                Version: 1
57
              Configuration:
58
                ApplicationName: agent-app
59
                DeploymentGroupName: staging
60
                DeploymentConfig: CodeDeployDefault.ECSAllAtOnce
61
                AppSpecTemplateArtifact: BuildOutput
62
                AppSpecTemplatePath: appspec.yaml
63
                TaskDefinitionTemplateArtifact: BuildOutput
64
                TaskDefinitionTemplatePath: taskdef.json
65
              InputArtifacts:
66
                - Name: BuildOutput
67
              RunOrder: 1
68

69
        # ── Stage 4: Smoke Tests ──
70
        - Name: SmokeTests
71
          Actions:
72
            - Name: RunSmokeTests
73
              ActionTypeId:
74
                Category: Test
75
                Owner: AWS
76
                Provider: CodeBuild
77
                Version: 1
78
              Configuration:
79
                ProjectName: !Ref SmokeTestProject
80
                EnvironmentVariables: '[{"name":"STAGING_URL","value":"https://staging.agent.yourdomain.com","type":"PLAINTEXT"}]'
81
              InputArtifacts:
82
                - Name: BuildOutput
83
              RunOrder: 1
84

85
        # ── Stage 5: Manual Approval ──
86
        - Name: Approval
87
          Actions:
88
            - Name: ApproveProduction
89
              ActionTypeId:
90
                Category: Approval
91
                Owner: AWS
92
                Provider: Manual
93
                Version: 1
94
              Configuration:
95
                NotificationArn: !Ref SnsTopic
96
                CustomData: "Agent deployment ready for production. Smoke tests passed."
97
              RunOrder: 1
98

99
        # ── Stage 6: Deploy to us-east-1 ──
100
        - Name: DeployUSEast1
101
          Actions:
102
            - Name: BlueGreenDeploy
103
              ActionTypeId:
104
                Category: Deploy
105
                Owner: AWS
106
                Provider: CodeDeploy
107
                Version: 1
108
              Configuration:
109
                ApplicationName: agent-app
110
                DeploymentGroupName: production-us-east-1
111
                DeploymentConfig: CodeDeployDefault.ECSLinear10PercentEvery1Minute
112
                AppSpecTemplateArtifact: BuildOutput
113
                AppSpecTemplatePath: appspec-us-east-1.yaml
114
                TaskDefinitionTemplateArtifact: BuildOutput
115
                TaskDefinitionTemplatePath: taskdef-us-east-1.json
116
              InputArtifacts:
117
                - Name: BuildOutput
118
                - Name: SourceOutput
119
              RunOrder: 1
120

121
        # ── Stage 7: Deploy to eu-west-1 ──
122
        - Name: DeployEUWest1
123
          Actions:
124
            - Name: BlueGreenDeploy
125
              ActionTypeId:
126
                Category: Deploy
127
                Owner: AWS
128
                Provider: CodeDeploy
129
                Version: 1
130
              Configuration:
131
                ApplicationName: agent-app
132
                DeploymentGroupName: production-eu-west-1
133
                DeploymentConfig: CodeDeployDefault.ECSLinear10PercentEvery1Minute
134
                AppSpecTemplateArtifact: BuildOutput
135
                AppSpecTemplatePath: appspec-eu-west-1.yaml
136
                TaskDefinitionTemplateArtifact: BuildOutput
137
                TaskDefinitionTemplatePath: taskdef-eu-west-1.json
138
              InputArtifacts:
139
                - Name: BuildOutput
140
                - Name: SourceOutput
141
              RunOrder: 2  # After us-east-1 is healthy
142

143
        # ── Stage 8: CloudFront Invalidation ──
144
        - Name: InvalidateCloudFront
145
          Actions:
146
            - Name: InvalidateCache
147
              ActionTypeId:
148
                Category: Invoke
149
                Owner: AWS
150
                Provider: Lambda
151
                Version: 1
152
              Configuration:
153
                FunctionName: cloudfront-invalidator
154
                UserParameters: "EDGE1A2B3C4D5E6"
155
              InputArtifacts:
156
                - Name: BuildOutput
157
              RunOrder: 3  # After both regions deployed

Step 5: Deployment Strategies#

Blue/Green (CodeDeploy)#

1
[Blue - Current]  ──▶ routes 100% traffic
2
[Green - New]     ──▶ routes 0% traffic, receives test traffic
3

4
1. Create Green task set with new image
5
2. Route test traffic to Green (internal validation)
6
3. Hook runs: Lambda validates agent responses
7
4. Shift traffic: 10% → 50% → 100%
8
5. If error rate > threshold → rollback to Blue
9
6. Blue drains and terminates

Canary release#

DeploymentConfig: CodeDeployDefault.ECSLinear10PercentEvery1Minute

Route 10% to Green → observe for 1 minute
Route 50% to Green → observe for 1 minute
Route 100% to Green → Blue drains

Rollback#

1
# One-click rollback
2
aws deploy stop-deployment \
3
  --deployment-id d-EXAMPLE \
4
  --auto-rollback-enabled
5

6
# Manual rollback to previous task definition
7
aws ecs update-service \
8
  --cluster mcp-server-cluster \
9
  --service github-issue-mcp \
10
  --task-definition github-issue-mcp:v42

Step 6: Model and Config Versioning#

Agents aren’t just code — they have models, prompts, tool definitions. These need version control too.

1
infrastructure/
2
├── src/                    # Agent source code
3
├── Dockerfile
4
├── buildspec.yml
5
├── config/
6
│   ├── models/
7
│   │   ├── production.yml  # Model ID, temperature, max tokens
8
│   │   ├── staging.yml     # Same but different model
9
│   │   └── canary.yml      # Test new model on 5% traffic
10
│   ├── prompts/
11
│   │   ├── system-prompt-v1.txt
12
│   │   ├── system-prompt-v2.txt
13
│   │   └── system-prompt-v3.txt
14
│   ├── cache/
15
│   │   └── policies.yml    # Similarity thresholds, TTLs
16
│   └── tools/
17
│       ├── github.yml       # Tool definitions
18
│       └── search.yml
19
└── pipeline.yaml

Config promotion through stages:#

1
# Build stage copies the right config
2
cp config/models/$DEPLOYMENT_ENV.yml src/config/model.yml
3
cp config/prompts/system-prompt-$PROMPT_VERSION.txt src/prompts/system.txt

Step 7: Rollback Scenarios#

Scenario	Detection	Action	Time
Error rate spike	CloudWatch alarm on 5XX	Auto-rollback CodeDeploy	<2 min
LLM response quality drop	Agent smoke test fails	Stop deployment, notify SNS, rollback	<5 min
Latency regression	p95 > 10s for 3 consecutive periods	Rollback + scale analysis	<5 min
Config drift	config checksum mismatch	Block pipeline, alert	Immediate
Cache poisoning	Cache hit rate drops >50%	Invalidate Redis, rollback	<3 min

1
# CloudWatch alarm that triggers auto-rollback
2
ErrorRateAlarm:
3
  Type: AWS::CloudWatch::Alarm
4
  Properties:
5
    AlarmName: agent-error-rate-high
6
    AlarmActions:
7
      - !Ref AutoRollbackTopic
8
    MetricName: HTTPCode_Target_5XX_Count
9
    Namespace: AWS/ApplicationELB
10
    Statistic: Sum
11
    Period: 60
12
    EvaluationPeriods: 3
13
    Threshold: 10
14
    ComparisonOperator: GreaterThanThreshold

Cost Breakdown#

Component	Monthly
CodePipeline (2 active pipelines)	~$2
CodeBuild (200 build minutes)	~$5
ECR storage (< 10 images)	~$1
CodeDeploy (ECS blue/green)	~$0 (included with ECS)
S3 artifact storage	<$1
CloudWatch alarms	~$3
Total CI/CD	~$12/mo

Summary#

The pipeline automates everything from code commit to production deployment across 2 regions, with safety controls at every step.

Pipeline flow:#

1
git push main → build Docker image →
2
  → deploy to staging → run smoke tests →
3
  → manual approval → deploy us-east-1 (blue/green) →
4
  → deploy eu-west-1 → invalidate CloudFront → done

Checklist:#

Series Complete 🎉#

Day	Topic	AWS Services
1	Deploy MCP Server on ECS Fargate	ECS, ECR, ALB, Secrets Manager
2	Agent State with DynamoDB	DynamoDB Global Tables, DAX
3	LLM Caching with ElastiCache + Bedrock	ElastiCache, Bedrock
4	Serverless Agent with Lambda + Bedrock	Lambda, API Gateway, Bedrock
5	Multi-Region Agent Routing with Route53	Route53, CloudFront, Global Accelerator
6	CI/CD for AI Agents	CodePipeline, CodeBuild, CodeDeploy

You now have everything needed to run AI agents in production on AWS: compute, state, caching, serverless, routing, and automation.

Series: AWS for AI/Agent Developers. Day 6 (finale): CI/CD pipeline — CodePipeline, blue/green deployments, agent smoke tests, canary releases, automatic rollback. Git push to production.