4.1.2. VPC Flow Logs for IP Traffic Monitoring

VPC Flow Logs capture detailed IP traffic information for network interfaces in your Amazon VPC, providing essential visibility for network monitoring, troubleshooting, and security analysis.

Scenario: You need to troubleshoot intermittent network connectivity issues between your application's EC2 instances and its database. You also need to monitor for suspicious outbound network activity (e.g., to unusual IP addresses) for security purposes.

VPC Flow Logs are a powerful feature that enables network specialists to monitor the IP traffic going to and from network interfaces in their Amazon VPC. They are crucial for network diagnostics, security incident response, and compliance auditing.

Key Features of VPC Flow Logs:

• Traffic Capture: Records information about IP traffic, including source/destination IP address, port, protocol, packets, bytes, and action (ACCEPT or REJECT). This provides a detailed record of every network "flow."
• Scope: Can be enabled for an entire VPC, a subnet, or a specific Elastic Network Interface (ENI) (attached to EC2 instances, load balancers, NAT Gateways, etc.).
• Destinations: Flow log records can be published to Amazon CloudWatch Logs or Amazon S3 for storage and analysis.
• Use Cases:
  • Network Diagnostics: Debugging connectivity issues between EC2 instances or to external networks.
  • Security Analysis: Identifying unusual traffic patterns, unauthorized access attempts, data exfiltration attempts, or potential DDoS attacks.
  • Compliance Auditing: Providing an audit trail of network traffic for regulatory compliance.
  • Performance Optimization: Identifying high-traffic flows or unexpected data transfer.
• Format: Log records are plain text and can be easily parsed.

Practical Implementation: Querying VPC Flow Logs in CloudWatch Logs Insights

# Example query to find rejected traffic between two IPs
fields @timestamp, @message
| filter action = "REJECT"
| filter srcAddr = "10.0.1.10" and dstAddr = "10.0.2.20"
| sort @timestamp desc
| limit 20

# Example query to find top talkers (most bytes transferred)
fields @timestamp, @message
| parse @message "* * * * * * * * * * * * * * * *" as accountId, interfaceId, srcaddr, dstaddr, srcport, dstport, protocol, packets, bytes, start, end, action, logstatus, vpcId, subnetId, instanceId
| stats sum(bytes) as totalBytes by srcaddr, dstaddr
| sort totalBytes desc
| limit 10

⚠️ Common Pitfall: Not enabling Flow Logs at the correct scope (VPC, subnet, or ENI) or not sending them to an accessible destination for analysis. Without proper configuration, you'll lack the necessary visibility.

Key Trade-Offs:

• Granularity of Logs vs. Cost: Capturing all traffic (ALL) provides the most detailed information but incurs higher costs for log ingestion and storage compared to capturing only rejected traffic (REJECT).

Reflection Question: How do VPC Flow Logs, by capturing detailed IP traffic information for network interfaces and publishing it to CloudWatch Logs or S3, provide essential visibility for network monitoring, enabling you to troubleshoot connectivity issues and analyze traffic for security threats?