Docker Container Keeps Restarting in Production

Problem Summary

A Docker container in a restart loop is one of the most time-pressured production incidents because the container is continuously crashing and the application is unavailable or intermittently available. The restart behavior (driven by restart: always or --restart=on-failure) is designed to recover from transient failures but becomes a crash loop when the underlying cause is persistent. The key is reading the exit code and the last few log lines from the crashed container these almost always reveal the cause within minutes.

Symptoms

• docker ps shows the container status flipping between 'Up X seconds' and 'Restarting'
• docker logs shows the application starting up and then abruptly stopping with no error message, or with an OOM error
• The container runs for a predictable amount of time (e.g., always crashes after ~30 seconds) suggesting a health check timeout
• The container starts fine in docker-compose up but crashes when run in production with docker run
• Error message: 'cannot connect to database' or 'ECONNREFUSED' on startup
• docker inspect shows exit code 137 (OOM kill) or exit code 1 (application error)
• The application works in the container for a few minutes then crashes under load
• Health check status shows 'unhealthy' repeatedly before each restart

Business Impact

A restarting container means your application is continuously unavailable or serving requests only during the brief up-time between restarts. Depending on restart backoff settings, the container might be up for only seconds every few minutes. If the container hosts an API, every in-flight request during a restart is lost. For stateful applications, repeated unclean shutdowns can corrupt in-memory state or leave database transactions uncommitted.

Root Cause

Container restart loops have five primary causes: (1) OOM (out of memory) kills where the container hits the memory limit and the kernel terminates it with SIGKILL, (2) application crash on startup due to missing environment variables or inability to connect to a database that is not yet ready, (3) health check misconfiguration where the health check path, port, or start period is wrong causing the container to be considered unhealthy and killed, (4) depends_on in Docker Compose which only waits for the container to start, not for the service inside it to be ready, and (5) PID 1 signal handling where the application does not handle SIGTERM gracefully and Docker kills it after a timeout.

Diagnosis

Step 1 Read the Exit Code and Last Logs

The exit code and the final log lines from the crashed container tell you most of what you need. Use docker inspect to get the exit code and docker logs to see what the container printed before dying.

Step 2 Diagnose OOM Kills

If OOMKilled is true in docker inspect, the container is hitting its memory limit. Check the container's memory usage over time and compare to its limit.

services:
  app:
    image: my-app:latest
    deploy:
      resources:
        limits:
          memory: 512M   # Increase if OOM killing
          cpus: "1.0"
        reservations:
          memory: 256M
    # For older docker-compose syntax:
    mem_limit: 512m
    memswap_limit: 512m   # Set equal to mem_limit to disable swap

Step 3 Identify Application Startup Crashes

If the container exits with code 1 immediately after startup, look for missing environment variables or failed database connections. Run the container without restart policy to see the full output before it disappears.

#!/bin/bash
# Use this as your container entrypoint to wait for database readiness
set -e

host="${DB_HOST:-postgres}"
port="${DB_PORT:-5432}"
max_attempts=30
attempt=0

echo "Waiting for $host:$port to be ready..."

until pg_isready -h "$host" -p "$port" -U "${DB_USER}" 2>/dev/null; do
  attempt=$((attempt + 1))
  if [ $attempt -ge $max_attempts ]; then
    echo "Database not ready after $max_attempts attempts. Exiting."
    exit 1
  fi
  echo "Attempt $attempt/$max_attempts  sleeping 2 seconds..."
  sleep 2
done

echo "Database is ready. Starting application..."
exec "$@"

Step 4 Fix Health Check Configuration

A misconfigured health check will kill a container that is actually running correctly. The most common mistakes: wrong port, wrong path, start_period too short (application has not finished initializing yet), and interval too short (health checks run before the previous one finishes).

services:
  app:
    image: my-app:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s        # How often to run the check
      timeout: 10s         # How long to wait for a response
      retries: 3           # Failures before marking unhealthy
      start_period: 60s    # Grace period after startup before checks count
    ports:
      - "3000:3000"

Step 5 Fix depends_on Race Conditions and PID 1 Signal Handling

Docker Compose's depends_on only waits for the dependent container to start, not for the service inside it to be ready. Use the condition: service_healthy form combined with health checks on the dependency.

services:
  postgres:
    image: postgres:16
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
      interval: 5s
      timeout: 5s
      retries: 10
      start_period: 10s

  app:
    image: my-app:latest
    depends_on:
      postgres:
        condition: service_healthy   # Waits for postgres healthcheck to pass
    # Use tini or dumb-init as PID 1 for proper signal handling
    init: true   # Docker's built-in init (tini)
    # OR in Dockerfile: ENTRYPOINT ["tini", "--", "node", "server.js"]

Production Failure Scenarios

Scenario 1 Node.js Heap Grows Slowly Until OOM Kill

A memory leak in the application (uncleaned event listeners, growing cache without eviction) causes the Node.js heap to grow gradually until hitting the container memory limit. The container runs fine for hours then gets killed. Symptom: crash happens on a schedule, not at startup. Fix: add --max-old-space-size, implement memory leak profiling with clinic.js, and add a /health endpoint that returns 503 if memory usage exceeds 90% of the limit.

Scenario 2 Health Check Hitting Non-Existent Endpoint

The Dockerfile defines a HEALTHCHECK to curl /healthz but the application only has a /health endpoint. Every health check returns a 404, Docker marks the container unhealthy after 3 retries, and kills it. The application itself is running fine the health check is wrong. Fix: always test your health check command manually inside the container with docker exec <container> curl http://localhost:3000/health

Scenario 3 Container Exits with Code 0 Immediately

The CMD or ENTRYPOINT in your Dockerfile runs a script that exits cleanly (exit 0) after completing setup, before the main process starts. Example: CMD ["sh", "-c", "npm run migrate && npm start"] if npm run migrate runs as a background process or the shell exits before npm start is ready. Fix: always exec the final process: CMD ["sh", "-c", "npm run migrate && exec npm start"]

Scenario 4 Missing Secret Causes Crash After Rotation

An API key or database password was rotated in the secrets manager but the Docker container's environment variable was not updated. The application starts, tries to authenticate with the old key, fails, and crashes. Fix: use Docker secrets or a secrets management sidecar that refreshes credentials without restart, and implement retry logic for authentication failures at startup.

Prevention Checklist

• Always check exit codes with docker inspect 137 means OOM, 1 means app error, 0 means CMD exited (likely wrong entrypoint)
• Set --max-old-space-size for Node.js to container memory limit minus 100 MB overhead
• Add start_period to health checks that is longer than your slowest measured startup time plus 30 seconds buffer
• Test health check commands manually with docker exec before deploying
• Use depends_on with condition: service_healthy instead of bare depends_on for any service with a startup time
• Add init: true to all Docker Compose services or use tini as ENTRYPOINT in Dockerfile for correct PID 1 signal handling
• Implement a wait-for-it.sh or pg_isready loop in your entrypoint for database dependencies
• Monitor container memory usage with docker stats and set alerts before the OOM threshold
• Set up container restart backoff (restart: on-failure:5) to prevent infinite restart loops from burning through resources
• Use a structured logging format (JSON) so container logs are parseable and searchable in your log aggregation system

Resolution Summary

Read the exit code first: 137 = OOM (increase memory or fix memory leak), 1 = app error (check logs for missing env vars or connection failures), 0 = wrong CMD (fix entrypoint). Then check health check configuration, dependency readiness with depends_on conditions, and PID 1 signal handling. In most cases, one of these five causes is responsible and the fix is a configuration change, not a code change.

Lessons Learned

• The exit code is the most information-dense signal available read it before reading logs
• Docker's depends_on is container-ready, not service-ready; the health check condition is mandatory for reliable startup ordering
• PID 1 signal handling is a container-specific concern that does not exist when running processes outside containers
• Health check start_period being too short is responsible for a large fraction of false-positive restart loops
• Memory limits without language-level heap size configuration (Node.js --max-old-space-size) lead to OOM kills even when memory usage looks reasonable
• Secret rotation events are a common trigger for startup crashes credential refresh without restart is worth the engineering investment

Conclusion

Docker container restart loops are stressful in production but they follow predictable patterns. Exit codes and logs give you the diagnosis within minutes if you know what to look for. Build your container infrastructure with health checks that have generous start periods, depends_on with service_healthy conditions, init: true for signal handling, and explicit memory limits with corresponding language-level heap settings. A container that restarts is a container telling you something is wrong the restart loop is the symptom, not the problem.