2.3.1. Analyzing Resource Configuration and Permissions
First Principle: Most logging failures are caused by one of three things: missing IAM permissions, incorrect resource configuration, or disabled features. Systematic diagnosis starts by checking these three areas in order.
Common Logging Failures by Service:
| Service | Common Failure | Root Cause | Fix |
|---|---|---|---|
| Lambda → CloudWatch | No logs appear | Execution role lacks logs:CreateLogGroup, logs:PutLogEvents | Add CloudWatch Logs permissions to execution role |
| API Gateway | No access/execution logs | Logging not enabled in stage settings, or IAM role missing | Enable logging in stage, create API Gateway logging role |
| CloudFront | No access logs | Access logging disabled, or S3 bucket ACL doesn't grant CloudFront write | Enable logging, fix S3 bucket ACL |
| Health checks | False positives | Security group blocking health check traffic | Allow health check source IPs in security group |
| CloudTrail | Missing data events | Data events not enabled for the specific service | Enable data events in Trail configuration |
Systematic Troubleshooting Process:
- Verify the feature is enabled — Is logging turned on for the service?
- Check IAM permissions — Does the service role have write access to the log destination?
- Check the destination — Is the S3 bucket/CloudWatch log group accessible? Any resource policies blocking access?
- Check network path — Can the resource reach the logging endpoint? (VPC endpoints may be needed for private subnets)
- Check for encryption conflicts — If the destination uses KMS, does the source service have
kms:GenerateDataKeypermission?
⚠️ Exam Trap: A VPC-attached Lambda function does NOT need a NAT gateway or VPC endpoint for its own logs — Lambda captures stdout/stderr and delivers it to CloudWatch Logs through the Lambda service, outside your VPC. If a function runs and returns results but no logs appear, check the execution role's logs:CreateLogGroup, logs:CreateLogStream and logs:PutLogEvents permissions first. Network routes (a NAT gateway or a logs interface endpoint) matter when something inside the VPC calls the CloudWatch Logs API itself — the CloudWatch agent on an EC2 instance in a private subnet, or an SDK call from your code.
Scenario: API Gateway access logs stopped appearing in CloudWatch 3 weeks ago. After investigation, you discover that someone rotated the API Gateway's logging IAM role, and the new role doesn't have logs:PutLogEvents permission.
Reflection Question: Why is IAM permission the first thing to check when a service stops sending logs, and how do you prevent recurrence?
