2.2.1. Amazon CloudWatch Logs for Log Collection

💡 First Principle: Amazon CloudWatch Logs provides a centralized, scalable, and durable service for collecting, storing, and monitoring logs from various AWS services and applications, enabling unified log management.

Scenario: You need to collect application logs from a fleet of EC2 instances, Lambda functions, and API Gateway access logs into a single, centralized location for monitoring and troubleshooting.

Amazon CloudWatch Logs is a fully managed service that allows SysOps Administrators to collect and consolidate logs from multiple sources into a single, highly scalable location.

Key Features of CloudWatch Logs:

• Centralized Collection: Gathers logs from:
  • AWS Lambda functions: Automatically pushed.
  • EC2 instances: Using the CloudWatch Agent.
  • Amazon VPC Flow Logs: For network traffic.
  • AWS CloudTrail: For API activity.
  • Other AWS services and custom applications.
• Log Groups and Streams: Logs are organized into log groups (for a specific application or service) and log streams (for specific instances or Lambda invocations).
• Real-time Monitoring: Set up metric filters to create metrics from log events and trigger alarms.
• Log Retention: Configure how long logs are stored to meet compliance or debugging needs.
• Encryption: Encrypt logs at rest using AWS KMS.

⚠️ Common Pitfall: Not configuring appropriate log retention policies, leading to excessive storage costs for old, infrequently accessed logs.

Key Trade-Offs: Real-time log ingestion and analysis (higher cost, immediate insight) versus delayed archival to S3 (lower cost, but slower retrieval).

Practical Implementation: Configuring CloudWatch Agent for EC2 (snippet from config.json):

{
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "/var/log/syslog",
                        "log_group_name": "/ec2/syslog",
                        "log_stream_name": "{instance_id}"
                    }
                ]
            }
        }
    }
}

Reflection Question: How does Amazon CloudWatch Logs, by providing centralized collection and organization into log groups and streams, enable you as a SysOps Administrator to effectively collect and manage logs from various AWS services and applications?