# ImagePullBackOff

## What Is ImagePullBackOff

`ImagePullBackOff` means:

- Kubernetes tried to pull the container image
- It failed
- It will retry again, but with increasing delay

This is **not a registry outage** in most cases. Kubernetes is working fine — the image **cannot be pulled**.

### What Happens Internally

1. Pod gets scheduled to a node
2. kubelet tries to pull the image
3. Image pull fails
4. Kubernetes retries with increasing delay
5. Pod shows `ImagePullBackOff`

> Kubernetes is healthy. Image access is the problem.

---

## 1. First Thing to Check (Always)

```bash
kubectl get pods
```

You will see:

```
STATUS: ImagePullBackOff
```

or

```
STATUS: ErrImagePull
```

For all namespaces:

```bash
kubectl get pods -A | grep ImagePullBackOff
```

If **many pods fail together**, it's usually **configuration**, not Docker Hub.

---

## 2. Read Pod Events (Most Important Step)

```bash
kubectl describe pod <pod-name>
```

Look only at **Events**. You will see exact reasons, for example:

- `pull access denied`
- `unauthorized`
- `manifest unknown`
- `no such host`
- `repository does not exist`

Read the error literally. Kubernetes is not guessing.

---

## 3. The Most Common Root Causes

### 3.1 Wrong Image Name or Tag (Very Common)

**Example (Broken)**

```yaml
image: myapp:v1.0.0
```

But `v1.0.0` does not exist.

**How to confirm:**

```bash
docker pull myapp:v1.0.0
```

If it fails locally → it will fail in Kubernetes.

---

### 3.2 Private Registry Without Credentials

Registry is private, but Kubernetes has **no login details**.

**Event message:**

```
pull access denied
unauthorized
```

**Fix: Create an image pull secret**

```bash
kubectl create secret docker-registry regcred \
  --docker-server=<registry-url> \
  --docker-username=<username> \
  --docker-password=<password>
```

---

### 3.3 Secret Exists but NOT Attached

This is very common. The secret exists:

```bash
kubectl get secret regcred
```

But the pod does not reference it.

**Broken Deployment:**

```yaml
containers:
- name: app
  image: private-registry/myapp:latest
```

**Fixed Deployment:**

```yaml
containers:
- name: app
  image: private-registry/myapp:latest
imagePullSecrets:
- name: regcred
```

No reference = no authentication.

---

### 3.4 Wrong Registry URL

Small mistakes break everything.

Examples:

- `docker.io/myapp` vs `myapp`
- Missing cloud region
- Wrong hostname

Kubernetes will **not auto-correct** registry URLs.

---

### 3.5 Node Has No Internet / Registry Access

This is infrastructure related.

**Common causes:**

- No outbound NAT
- Firewall blocking registry
- Proxy misconfiguration

**To test from inside the cluster, run a temporary debug pod:**

```bash
kubectl run net-test \
  --image=curlimages/curl:latest \
  --restart=Never \
  --command -- sleep 3600
```

Exec into it:

```bash
kubectl exec -it net-test -- sh
```

Test registry access:

```bash
curl -v https://registry-1.docker.io
```

**Expected success:**

- HTTP response (401 / 200 is OK)
- DNS resolves
- TLS handshake succeeds

---

## Hands-On Demo

The following examples **deliberately introduce failures** to show how ImagePullBackOff occurs.

### Example 1: Wrong Image Tag

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: imagepull-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: imagepull-demo
  template:
    metadata:
      labels:
        app: imagepull-demo
    spec:
      containers:
      - name: app
        image: nginx:app
```

**Result:** `ImagePullBackOff` — the tag `app` does not exist for the nginx image.

---

### Example 2: Private Image Without Secret

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: private-image-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: private-demo
  template:
    metadata:
      labels:
        app: private-demo
    spec:
      containers:
      - name: app
        image: private-registry/myapp:latest
        ports:
        - containerPort: 3000
```

**Result:** `ImagePullBackOff` — `pull access denied`

**Fix:** Add `imagePullSecrets` to the deployment:

```yaml
spec:
  containers:
  - name: app
    image: private-registry/myapp:latest
    ports:
    - containerPort: 3000
  imagePullSecrets:
  - name: regcred
```

---

## About imagePullPolicy

```yaml
imagePullPolicy: Always
```

This forces Kubernetes to pull the image **every restart**.

In unstable setups:

- Makes failures noisy
- Slows recovery

Use carefully.

---

## Quick Debug Checklist

Ask these **in sequence**:

1. Does the image exist?
2. Can I pull it locally?
3. Is the registry private?
4. Is `imagePullSecret` created?
5. Is `imagePullSecret` attached?
6. Can the node reach the registry?

> **90% of ImagePullBackOff issues are solved by this checklist.**

---

## How to Recover

After fixing the issue:

```bash
kubectl rollout restart deployment/<name>
```

Verify:

```bash
kubectl rollout status deployment/<name>
```

Status should move: `ImagePullBackOff` → `Running`

---

## Important Clarification

| Status | Meaning |
|---|---|
| ErrImagePull | Pull failed (first attempt) |
| ImagePullBackOff | Pull failed repeatedly |

They are the **same problem**, different stages.

`ImagePullBackOff` means Kubernetes cannot fetch your image — not that the registry is down.

---

## Final Truth

- Not a Kubernetes bug
- Usually not a registry outage
- Almost always:
  - Wrong image
  - Missing secret
  - Wrong registry
  - No node internet

Kubernetes is strict — and that's a good thing.


---