Redis Memory Usage Keeps Growing Unexpectedly

Problem Summary

Redis is designed as an in-memory data store, which means every key consumes RAM. When memory usage grows continuously rather than fluctuating around a steady state one of a small set of problems is almost always the cause: keys are being written without TTL (time-to-live) expiry, so they accumulate indefinitely; objects being cached are far larger than necessary (entire API responses, serialised ORM objects, full HTML pages); Redis is being used as a primary database for data that should live in PostgreSQL; no eviction policy is configured so Redis cannot free memory automatically when full; or keyspace notification subscribers are consuming memory through message queues. Each of these is diagnosable with redis-cli commands.

Symptoms

• used_memory in redis-cli info memory grows monotonically over days or weeks
• Redis OOM (out of memory) errors appearing in application logs
• Redis starts evicting keys unexpectedly or returns NODATA errors without explicit expiry
• The number of keys reported by redis-cli dbsize grows continuously
• A small number of keys account for a disproportionate percentage of total memory
• Cache hit rate is low despite high memory usage (wrong keys are being retained)
• AWS ElastiCache FreeableMemory CloudWatch metric trending toward zero
• Application latency spikes when Redis performs memory reclamation

Business Impact

When Redis runs out of memory, the impact is immediate and severe. Depending on the eviction policy, Redis will either refuse all write operations (returning OOM error for every SET command) or start evicting cache keys that the application expects to be present, causing every request to fall back to the database which is exactly the load spike Redis was preventing. If Redis is used for session storage, queue management, or rate limiting in addition to caching, an OOM condition causes authentication failures, job processing halts, and rate limiting bypass simultaneously.

Root Cause

Redis memory growth has five main root causes: persistent keys set without TTL that accumulate as the application grows (the most common cause); caching of excessively large objects like full API response payloads or serialised database result sets; using Redis as a primary datastore without thinking through key lifecycle; no maxmemory or maxmemory-policy configured, so Redis never evicts anything automatically; and keyspace notification publishers writing to channels with no active subscribers, creating backlogs. The fix requires identifying which cause applies before making changes.

Diagnosis

Step 1: Get Memory Overview with INFO MEMORY

The starting point for any Redis memory diagnosis is the INFO MEMORY command, which reports current and peak memory usage, fragmentation ratio, and memory allocation details. A fragmentation ratio above 1.5 suggests memory is fragmented and MEMORY PURGE (Redis 4+) or a rolling restart may reclaim physical RAM.

# Get full memory statistics
redis-cli info memory

# Key fields to check:
# used_memory_human: current memory used by Redis data
# used_memory_peak_human: peak memory ever used
# used_memory_rss_human: memory as seen by OS (includes fragmentation)
# mem_fragmentation_ratio: rss/used_memory - above 1.5 is high fragmentation
# maxmemory_human: configured memory limit (0 means no limit)
# maxmemory_policy: eviction policy (noeviction = no automatic eviction)

# Check total key count
redis-cli dbsize

# Check key count by database
redis-cli info keyspace

# Check memory for a specific key
redis-cli memory usage "your:key:here"
redis-cli memory usage "your:key:here" SAMPLES 0  # exact (slower)

Step 2: Find Keys with No TTL

Keys with TTL of -1 will never expire. On a large keyspace, scanning all keys to find persistent ones requires using the SCAN command with a cursor rather than KEYS * (which blocks Redis). The following approach identifies persistent keys by prefix pattern, which reveals which application code paths are writing non-expiring keys.

# Scan all keys and find those with no TTL
# WARNING: Do not use KEYS * on production - use SCAN instead
redis-cli --scan --pattern "*" | while read key; do
  ttl=$(redis-cli ttl "$key")
  if [ "$ttl" = "-1" ]; then
    size=$(redis-cli memory usage "$key" 2>/dev/null || echo 0)
    echo "$size $key"
  fi
done | sort -rn | head -50

# Count keys by prefix to find which application area has the most persistent keys
redis-cli --scan --pattern "*" | sed 's/:.*//' | sort | uniq -c | sort -rn | head -20

# More efficient: sample a subset
redis-cli --scan --pattern "*" --count 1000 | shuf | head -200 | while read key; do
  ttl=$(redis-cli ttl "$key")
  type=$(redis-cli type "$key")
  echo "$ttl $type $key"
done | sort -n | head -30

Step 3: Identify the Largest Keys

A small number of very large keys often account for the majority of Redis memory usage. Finding them quickly requires using redis-cli with the --bigkeys flag, which scans the keyspace and reports the largest key by type. For more precise analysis, use the MEMORY USAGE command on specific keys.

# Find largest keys by type (uses SCAN internally, safe for production)
# Run during low-traffic periods as it can be CPU-intensive
redis-cli --bigkeys

# Find the memory usage of specific key patterns
redis-cli --scan --pattern "cache:api:*" | while read key; do
  size=$(redis-cli memory usage "$key")
  echo "$size $key"
done | sort -rn | head -20

# For a hash, check its field count and total size
redis-cli hlen "your:hash:key"
redis-cli memory usage "your:hash:key"

# For a sorted set (common in leaderboards and queues)
redis-cli zcard "your:zset:key"
redis-cli memory usage "your:zset:key"

# Debug object to see encoding and serialized length
redis-cli debug object "your:key:here"

Step 4: Configure maxmemory and Eviction Policy

Every Redis instance used for caching must have a maxmemory limit and a sensible eviction policy. Without these, Redis will use all available system RAM and then either crash or cause the OS to swap, which makes Redis effectively unusable. The correct eviction policy depends on your use case: allkeys-lru for a pure cache, volatile-lru if some keys must never be evicted (set those without TTL), and noeviction only for Redis instances used for durable data (sessions, queues).

# Check current maxmemory configuration
redis-cli config get maxmemory
redis-cli config get maxmemory-policy

# Set maxmemory to 80% of available RAM (set in redis.conf for persistence)
# For a 4GB instance, set to 3.2GB
redis-cli config set maxmemory 3435973836  # 3.2GB in bytes

# Set eviction policy
# allkeys-lru: evict least recently used keys (best for pure caches)
# volatile-lru: evict LRU keys among those with TTL set
# allkeys-lfu: evict least frequently used (better for Zipf distributions)
# noeviction: return errors on write when full (use for queues/sessions)
redis-cli config set maxmemory-policy allkeys-lru

# Persist to redis.conf:
# maxmemory 3435973836
# maxmemory-policy allkeys-lru

# Make config changes survive restart (on AWS ElastiCache, use parameter groups)

Step 5: Audit Application Code for Missing TTLs and Oversized Values

Trace all Redis SET, HSET, ZADD, and LPUSH calls in application code and verify each has an appropriate TTL. Any cache write without a TTL is a potential memory leak. Also check the size of values being cached serialised ORM objects often include circular references or unnecessary fields that bloat the cached value by 10-50x compared to a minimal cache representation.

// BAD: Caching without TTL - key will persist forever
await redis.set(`user:${userId}`, JSON.stringify(user));

// BAD: Caching the entire database object with all fields
const user = await db.users.findOne({ where: { id: userId } });
await redis.setEx(`user:${userId}`, 3600, JSON.stringify(user)); // includes ORM metadata

// GOOD: Cache only the fields you need with explicit TTL
const user = await db.users.findOne({ where: { id: userId } });
const cacheValue = {
  id: user.id,
  name: user.name,
  email: user.email,
  plan: user.plan,
  avatarUrl: user.avatarUrl
};
await redis.setEx(`user:${userId}`, 3600, JSON.stringify(cacheValue));

// GOOD: Cache API response as minimal projection
const apiResponse = await fetchFromThirdParty(params);
const minimal = {
  id: apiResponse.id,
  status: apiResponse.status,
  result: apiResponse.result
  // omit: raw headers, debug info, full response metadata
};
await redis.setEx(`api:${cacheKey}`, 300, JSON.stringify(minimal));

// Helper to enforce TTL on all cache writes
async function cacheSet(redis, key, value, ttlSeconds) {
  if (!ttlSeconds || ttlSeconds <= 0) {
    throw new Error(`TTL required for cache key: ${key}`);
  }
  return redis.setEx(key, ttlSeconds, JSON.stringify(value));
}

Production Failure Scenarios

Scenario 1: Session Keys Without TTL Growing to Millions

A session management library stores session data in Redis without expiry because the application manages session lifecycle in PostgreSQL. Sessions that are explicitly invalidated are deleted from Redis, but sessions that simply expire through inactivity are never cleaned up. Over 18 months, Redis accumulates 4 million abandoned session keys consuming 12GB. Fix: always set TTL on session keys equal to the session timeout period, and implement a secondary cleanup job that removes orphaned sessions.

Scenario 2: Full API Response Cached Including Pagination Metadata

An endpoint that returns a paginated list of products caches the entire API response JSON, including 200 product objects, pagination links, and debug metadata, as a single Redis string. Each cached response is 800KB. With 2,000 unique query parameter combinations being cached, Redis holds 1.6GB of product listing caches. Fix: cache only the product IDs list (a few KB), cache individual product objects separately, and reassemble the response from the ID list and individual object cache on each request.

Scenario 3: Redis Used as Primary Database for Event Log

An analytics feature logs every user action to a Redis sorted set (ZADD user_events:{id} timestamp event). The sorted set grows indefinitely because the intent was to query it like a database. With 10,000 active users each generating 200 events per day, Redis is receiving 2 million ZADD commands daily with no ZREMRANGEBYSCORE cleanup. Fix: use PostgreSQL as the primary store for event data and use Redis only as a cache or for real-time aggregations with bounded time windows.

Scenario 4: Keyspace Notifications Creating Subscriber Backlog

Redis keyspace notifications are enabled (notify-keyspace-events = KEA) to watch for key expiry events. The subscriber application was restarted but the notification channel continued buffering events in a Redis list. The backlog grew to 50 million entries consuming 4GB. Fix: disable keyspace notifications if not actively used (notify-keyspace-events = ''), implement subscriber health checks that alert when the backlog exceeds a threshold, and use XADD/XREAD (Redis Streams) instead of pub/sub for durable message queuing.

Scenario 5: Memory Fragmentation After Mass Key Deletion

A cache warming job runs at midnight and deletes 500,000 keys before rebuilding the cache. The deletion returns virtual memory to Redis's allocator but not to the OS. used_memory drops by 4GB but used_memory_rss stays high the OS still sees Redis using the same physical RAM. The fragmentation ratio climbs to 2.5. Fix: run MEMORY PURGE (Redis 4+) after mass deletions, or schedule a rolling Redis restart during low-traffic periods to fully return fragmented memory to the OS.

Prevention Checklist

• Set maxmemory to 75-80% of available instance RAM and configure an appropriate maxmemory-policy
• Enforce TTL on every Redis write through a wrapper function that throws if TTL is not provided
• Use redis-cli --bigkeys monthly to find keys that have grown disproportionately large
• Monitor used_memory_human and FreeableMemory in CloudWatch with alerts at 70% and 85% capacity
• Audit all Redis key patterns quarterly and verify each has appropriate TTL and value size
• Never cache entire ORM objects or full API responses cache minimal projections
• Use Redis Streams instead of pub/sub for durable messaging to avoid subscriber backlog
• Set notify-keyspace-events to empty string unless actively using keyspace notifications
• Implement SCAN-based key auditing (never KEYS *) as a scheduled maintenance job
• Document Redis key naming conventions and TTL requirements in your engineering wiki

Resolution Summary

Start with redis-cli info memory to understand current usage and fragmentation. Run redis-cli --bigkeys to find the largest keys. Scan a sample of keys to estimate the proportion with no TTL. Set maxmemory and maxmemory-policy if not configured. Then audit application code for missing TTLs and oversized values, starting with the key patterns identified by --bigkeys. Add TTLs to existing persistent keys with EXPIRE, refactor value serialisation to store minimal projections, and implement a wrapper function that enforces TTL on all future cache writes.

Lessons Learned

• A Redis instance without maxmemory configured is a memory leak waiting to happen set it at provisioning
• The most common cause of Redis memory growth is not a bug but missing TTLs on keys that developers assumed would be short-lived
• redis-cli --bigkeys is the fastest way to find the highest-impact memory consumers and should be run regularly
• Caching entire ORM objects or API responses is an anti-pattern always define a minimal cache schema
• Using Redis as a primary database without expiry policy removes the main safety mechanism that makes Redis viable as in-memory storage
• Memory fragmentation after mass deletions is normal and expected plan for MEMORY PURGE or rolling restarts after large-scale key cleanup

Conclusion

Redis memory growth is almost always a policy problem rather than a capacity problem. The solution is rarely 'get a bigger Redis instance' it is 'enforce TTLs, cache smaller objects, and configure eviction'. These are one-time fixes that eliminate the growth permanently. After implementing TTLs and eviction policy, add monitoring on used_memory_human with alerts at 70% and 85% of maxmemory, and review key patterns quarterly to catch new accumulation patterns before they become production incidents.