OOMKilled

This document explains the OOMKilled error in Kubernetes — how to identify, confirm, troubleshoot, and prevent it in production environments.

---

1. What Is OOMKilled?

OOMKilled occurs when a container uses more memory than its configured limit and is forcibly terminated by the Linux kernel.

• Kubernetes does not kill the container
• The kernel kills it to protect the node
• Kubernetes only reports what happened

---

2. What OOMKilled Does NOT Mean

OOMKilled does not mean:

• Kubernetes crashed
• The application exited normally
• There was a code exception

It strictly means: Memory limit was exceeded

---

3. Where OOMKilled Happens (Flow)

1. Pod starts normally
2. Memory usage increases
3. Memory limit is crossed
4. Linux kernel kills the process
5. Container exits with code 137
6. Pod shows OOMKilled

---

4. Common Production Causes

Cause	Description
Memory leak	Application never releases memory
Low memory limits	Limits set lower than real usage
Incorrect requests	Requests much lower than actual usage
JVM / Node / Python	No heap or memory tuning
Traffic spikes	Sudden increase in load

---

5. How to Confirm OOMKilled (Always Verify)

Step 1: Describe the Pod

kubectl describe pod <pod-name>

Look for:

Reason: OOMKilled
Exit Code: 137

This confirms OOMKilled.

Step 2: Check Pod Status

kubectl get pod <pod-name>

Status may show: CrashLoopBackOff

---

6. Understand Requests vs Limits (Critical)

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

• Requests → used for scheduling
• Limits → enforced by the kernel

Exceed the limit → container is killed

---

7. Reproduce OOMKilled

Step 1: Create a Pod with Low Memory Limit

apiVersion: apps/v1
kind: Deployment
metadata:
  name: oom-example
  labels:
    app: oom
spec:
  replicas: 1
  selector:
    matchLabels:
      app: outofmemory
  template:
    metadata:
      labels:
        app: outofmemory
    spec:
      containers:
      - name: outofmemory
        image: praveen22233/app:v1
        ports:
        - containerPort: 8000
        resources:
          limits:
            cpu: "25m"
            memory: "25Mi"
          requests:
            cpu: "10m"
            memory: "10Mi"

Apply:

kubectl apply -f oom-test.yaml

Step 2: Observe Pod Behavior

kubectl get pods

Pod will restart repeatedly.

Step 3: Confirm OOMKilled

kubectl describe pod <pod-name>

You will see:

Reason: OOMKilled
Exit Code: 137

This error occurs because the CPU and memory limits are set too low — insufficient for the pod to run the application.

---

8. How to Troubleshoot

Step 1: Check Real Memory Usage

kubectl top pod <pod-name>

Compare:

• Actual usage
• Memory request
• Memory limit

Step 2: Identify the Offending Pod

• Look for pods near memory limit
• Check restart count
• Review application logs

Step 3: Fix the Root Cause

Scenario	Fix
Memory leak	Fix application
Low limit	Increase memory limit
Bursty usage	Increase limit + headroom
JVM apps	Set heap size
High load	Scale replicas

---

9. What NOT to Do

• Disable memory limits
• Ignore OOMKilled events
• Set very low requests
• Remove node pressure protections

These lead to node crashes.

Golden Rules

1. Memory requests must reflect real application usage, not assumptions or best-case estimates. Incorrect requests lead to poor scheduling and instability.

2. Memory limits must always include headroom to handle traffic spikes, GC cycles, and temporary memory bursts. Limits set too tight will inevitably cause OOMKilled.

3. Memory-intensive applications must be explicitly tuned (Java, Node.js, Python). Default runtime settings are not container-aware and often exceed Kubernetes memory limits.

---

10. Prevention Best Practices

• Always set memory requests & limits
• Monitor pod memory usage
• Alert on OOMKilled events
• Test workloads under load
• Tune language runtimes

---

11. Final Statement

OOMKilled is not a Kubernetes bug. It is Kubernetes and Linux protecting node stability.

If memory limits are ignored, the kernel enforces them.