EC2 Costs Suddenly Increased Without Traffic Growth

Problem Summary

EC2 cost increases without corresponding traffic growth almost always point to one of a handful of resource lifecycle problems: instances running that should not be, instances of the wrong (larger) type, storage volumes persisting after instance termination, or pricing-model transitions from reserved to on-demand. Unlike a traffic-driven cost increase, these are pure waste the compute is doing nothing productive.

Symptoms

• Cost Explorer shows EC2 spend up 50-200% month-over-month with flat or declining traffic
• The EC2 dashboard shows more running instances than your application inventory accounts for
• Data transfer line item in EC2 bill is larger than expected given your CDN usage
• EBS volume count in the console is much higher than the number of running instances
• Savings Plans or Reserved Instance coverage drops below 70% in the Savings Plans utilization report
• New instance types appear in the Cost Explorer usage type breakdown that were not there before
• Dev or staging environment instances show continuous 24/7 CPU usage
• An autoscaling group is maintaining a higher-than-expected minimum instance count

Business Impact

EC2 is typically the largest single line item in an AWS bill. A 50% increase translates to thousands of dollars per month for most production workloads and tens of thousands for larger infrastructures. Beyond the direct cost, over-provisioned EC2 usually reflects a team that has lost inventory control of its infrastructure a gap that also creates security risk from forgotten instances with potentially open security groups.

Root Cause

The most common EC2 cost spike causes: (1) a developer launched a large instance (r5.4xlarge, m5.2xlarge) for a one-off task and never terminated it, (2) dev/test instances that should be stopped evenings and weekends are running 24/7, (3) EBS volumes were not deleted when instances were terminated, (4) a Reserved Instance or Savings Plan expired and the same instances are now billed at on-demand rates (typically 3-4x higher), (5) data transfer between AZs or out to internet not being routed through a CDN.

Diagnosis

Step 1 Inventory All Running Instances by Type and Age

Get a complete list of running instances across all regions with their type and launch time. Instances more than 30 days old that are not tagged as production are prime candidates for investigation.

#!/bin/bash
# Get all running instances with type, launch time, and Name tag
aws ec2 describe-instances   --filters Name=instance-state-name,Values=running   --query 'Reservations[*].Instances[*].{
    ID:InstanceId,
    Type:InstanceType,
    Launched:LaunchTime,
    Name:Tags[?Key==`Name`].Value|[0],
    Env:Tags[?Key==`Environment`].Value|[0]
  }'   --output table | sort -k4

Step 2 Find and Delete Unattached EBS Volumes

Available (unattached) EBS volumes continue to incur storage charges. gp3 costs $0.08/GB/month; io2 costs $0.125/GB/month. A 1 TB io2 volume unattached for a year costs $1,500.

#!/bin/bash
# Snapshot then delete all unattached EBS volumes
# REVIEW THE LIST BEFORE RUNNING THIS IN PRODUCTION
VOLUMES=$(aws ec2 describe-volumes   --filters Name=status,Values=available   --query 'Volumes[*].VolumeId'   --output text)

for vol_id in $VOLUMES; do
  echo "Snapshotting $vol_id before deletion..."
  aws ec2 create-snapshot     --volume-id "$vol_id"     --description "Pre-deletion snapshot $(date +%Y-%m-%d)"     --tag-specifications "ResourceType=snapshot,Tags=[{Key=AutoCleanup,Value=true}]"
  # Uncomment after reviewing snapshots:
  # aws ec2 delete-volume --volume-id "$vol_id"
done

Step 3 Check Reserved Instance and Savings Plans Coverage

In Cost Explorer go to: Savings Plans → Utilization Report and Reserved Instances → Utilization Report. If utilization is below 100% you are paying for reservations you are not using. If coverage is below 70% some on-demand instances are not covered.

Step 4 Identify Dev/Test Instances Running 24/7

Dev and test instances running overnight and on weekends are pure waste. A t3.medium at $0.0416/hr costs $30/month if stopped at night vs $30/month regardless. For a team of 10 developers each with a dev instance, that is $300/month savings just by scheduling stops.

#!/bin/bash
# Find instances tagged as dev/staging that have been running > 16 hours
aws ec2 describe-instances   --filters     Name=instance-state-name,Values=running     Name=tag:Environment,Values=dev,staging,development   --query 'Reservations[*].Instances[*].{
    ID:InstanceId,
    Type:InstanceType,
    LaunchTime:LaunchTime,
    Name:Tags[?Key==`Name`].Value|[0]
  }'   --output table

Step 5 Audit Data Transfer Charges

Data transfer between Availability Zones costs $0.01/GB in each direction. If your application is chatty across AZs (e.g., app servers in us-east-1a calling databases in us-east-1b) this accumulates rapidly. Check VPC Flow Logs or use the Cost Explorer Usage Type filter for DataTransfer-Regional-Bytes.

Production Failure Scenarios

Scenario 1 GPU Instance Left from ML Experiment

A data scientist launched a p3.8xlarge ($12.24/hr) for a model training experiment three weeks ago. The experiment finished but the instance was only stopped, not terminated. Stopped instances do not charge for compute but the attached 500 GB EBS volume costs $50/month. More critically, if it is started again even briefly, it resumes full hourly charges. Terminate unused instances; do not just stop them.

Scenario 2 Autoscaling Group Minimum Raised During Incident

During a traffic incident, an engineer raised the autoscaling group minimum from 2 to 10 instances. Traffic returned to normal but the minimum was never lowered. The group has been running 10 instances instead of 2 for weeks. Check: aws autoscaling describe-auto-scaling-groups --query 'AutoScalingGroups[*].[AutoScalingGroupName,MinSize,MaxSize,DesiredCapacity]'

Scenario 3 Savings Plans Does Not Cover New Instance Family

Your team bought an EC2 Instance Savings Plan for m5 instances but migrated to m6i instances for better price-performance. EC2 Instance Savings Plans are family-specific they do not cover different families. You are paying for an unused m5 reservation AND on-demand m6i prices. Switch to a Compute Savings Plan which covers all instance families.

Prevention Checklist

• Enforce mandatory tagging (Name, Environment, Owner, Project) via IAM policy or AWS Config rule reject untagged instance launches
• Implement a 'stopped instance reaper' Lambda: instances stopped for more than 7 days generate a Slack alert; stopped for 14 days are auto-terminated (with snapshot)
• Schedule dev/test instance stop/start with EventBridge Scheduler treat 24/7 dev instances as a policy violation
• Set delete-on-termination=true for all non-root EBS volumes during instance launch
• Review Reserved Instance and Savings Plans coverage monthly calendar alert for expiry 60 days in advance
• Use Compute Savings Plans instead of EC2 Instance Savings Plans for flexibility across instance families
• Set autoscaling group minimum back to baseline within 24 hours after any incident response document it as part of post-incident checklist
• Enable AWS Trusted Advisor Low Utilization Amazon EC2 Instances check and review weekly

Resolution Summary

Inventory running instances by type and launch time, terminate anything not in your production inventory, delete unattached EBS volumes (after snapshotting), check Reserved Instance and Savings Plans coverage for expiry or family mismatches, and schedule dev environment stops. In most cases, one or two of these actions account for the majority of the unexpected increase.

Lessons Learned

• Stopped EC2 instances are not free their EBS volumes continue to cost money
• Mandatory resource tagging is the prerequisite to any cost attribution or cleanup program
• Reserved Instance and Savings Plans require active management they are not set-and-forget
• Autoscaling group parameter changes made during incidents must be explicitly reverted document this in runbooks
• Data transfer between AZs is invisible until it shows up on the bill consider co-locating tightly coupled services in the same AZ

Conclusion

EC2 cost spikes without traffic growth are a resource governance problem. The technical investigation is straightforward inventory what is running, compare to what should be running, and terminate the delta. The organizational fix is harder: mandatory tagging policies, scheduled instance cleanup, and treating dev environment costs with the same discipline as production. The teams that do this consistently run EC2 spend 40-60% lower than teams that address it reactively.