--- title: "Troubleshooting Kubernetes" description: "Fix E2E Kubernetes problems with kubectl and certificates, clusters stuck on Creating, LoadBalancer IPs, nodes, pools, volumes, and billing." content_type: "troubleshooting" product: "kubernetes" platform: "MyAccount" url: "/myaccount/kubernetes/troubleshooting/kubernetes/" relationships: {"troubleshooting-for":["/myaccount/kubernetes/how-to/create-a-cluster/","/myaccount/kubernetes/how-to/connect-with-kubectl/","/myaccount/kubernetes/how-to/manage-node-pools/","/myaccount/kubernetes/how-to/configure-autoscaling/","/myaccount/kubernetes/how-to/expose-a-service-with-a-load-balancer/","/myaccount/kubernetes/how-to/use-persistent-volumes/","/myaccount/kubernetes/how-to/manage-security-groups/","/myaccount/kubernetes/how-to/renew-the-cluster-certificate/","/myaccount/kubernetes/how-to/upgrade-the-master-plan/","/myaccount/kubernetes/how-to/install-marketplace-apps/","/myaccount/kubernetes/how-to/back-up-a-cluster-with-velero/","/myaccount/kubernetes/how-to/monitor-a-cluster/","/myaccount/kubernetes/how-to/use-committed-billing/","/myaccount/kubernetes/how-to/delete-a-cluster/"]} --- # Troubleshooting Kubernetes A Kubernetes problem shows up either in kubectl on your computer or in MyAccount on the Kubernetes page. For a problem inside the cluster, look first at the cluster's status in the list and at `kubectl get nodes`. For questions about limits, regions, and billing, see [Kubernetes FAQ](/myaccount/kubernetes/faq/). ## kubectl says x509: certificate has expired or is not yet valid kubectl stops with this message, and later commands may say `You must be logged in to the server`: ```text x509: certificate has expired or is not yet valid ``` The cluster's API certificate is due for renewal one year after the cluster is created, and one year after each renewal. E2E renews it on its renewal date while the cluster reads **Running**. Check, in order: 1. Download the kubeconfig again from **Cluster Details**; the file you download after a renewal has the new certificate. Point kubectl at the new copy; see [Connect to a cluster with kubectl](/myaccount/kubernetes/how-to/connect-with-kubectl/#download-the-kubeconfig). 2. Your computer's clock is right. A clock that is far off makes a valid certificate look expired or not yet valid. 3. The cluster reads **Running**. E2E renews the certificate only while it does; see [Renew the cluster certificate](/myaccount/kubernetes/how-to/renew-the-cluster-certificate/). If kubectl still fails, [contact support](#contact-support) with the cluster's name and the full message. ## kubectl times out on port 6443 kubectl stops with a message like this one: ```text Unable to connect to the server: dial tcp :6443: i/o timeout ``` The kubeconfig points kubectl at the master node's public IP on TCP port 6443, and the connection did not get through. Check, in order: 1. The cluster reads **Running** in the Kubernetes list. 2. Your network reaches the master node's public IP on TCP port 6443. A company firewall or VPN may block outgoing connections to that port; try from another network to find out. 3. The cluster's security groups allow inbound TCP port 6443. On the cluster's **Security Groups** tab, check the attached groups' rules; see [Add or edit rules](/myaccount/security-group/how-to/add-or-edit-rules/). A rule counts when its protocol is TCP or All, and its ports are All, 6443, or a range that includes it. Then run `kubectl get nodes` again. ## kubectl connects to the wrong cluster or to localhost:8080 kubectl uses the kubeconfig you point it at. If it finds none, it says `The connection to the server localhost:8080 was refused`. If it reaches another cluster, it is using another kubeconfig. Point kubectl at your cluster's kubeconfig with `--kubeconfig`, or set `KUBECONFIG` to the file's full path; see [Connect to a cluster with kubectl](/myaccount/kubernetes/how-to/connect-with-kubectl/#point-kubectl-at-the-kubeconfig). To see which server kubectl uses now, run: ```bash kubectl config view --minify ``` The `server` line names the address kubectl connects to. ## The cluster stays on Creating After **Create Cluster**, allow 5 to 10 minutes for the cluster to be set up. A cluster can be deleted once it reads **Running**. If it still reads **Creating** after 80 minutes, [contact support](#contact-support) with the cluster's name, its region, and the time you created it. See [Create a cluster](/myaccount/kubernetes/how-to/create-a-cluster/#create-the-cluster). ## You do not have sufficient credits to launch the kubernetes On a prepaid account, creation is refused with this message: ```text You do not have sufficient credits to launch the kubernetes. Additional Balance required: Available: ``` Creating a cluster needs infra credits for 24 hours of the master node and of each hourly pool's starting nodes, plus the full price of every committed node. Adding a pool needs the same for its nodes, and a master plan upgrade needs 24 hours of the new plan. Add infra credits of at least the **Additional Balance required** amount, and try again. See [Create a cluster](/myaccount/kubernetes/how-to/create-a-cluster/#create-the-cluster). ## Resource creation is currently restricted. Please try from a different location. In Delhi, an account created on or after May 15, 2026 cannot create a cluster unless E2E has exempted it, and the request is refused with this message. In Delhi, the same accounts' CPU plans show **Request** instead of being selectable. Create the cluster in Chennai instead, or [contact support](#contact-support) to create one in Delhi. See [Create a cluster](/myaccount/kubernetes/how-to/create-a-cluster/). ## Kubernetes requires TCP port 6443 and UDP port 8472 to be open in the security group The security group you chose on the create form does not allow both ports inbound, and creation is refused. The message may name only the missing port. Add inbound rules for TCP port 6443 and UDP port 8472 to the group, or choose a group that has them, and click **Create Cluster** again. A rule counts when its protocol is the one needed or All, and its ports are All, the exact port, or a range that includes it. See [Add or edit rules](/myaccount/security-group/how-to/add-or-edit-rules/). ## My VPC is not listed on the create form **Network Settings** lists only VPCs with an E2E provided range, with the note **Custom VPCs are not supported for Kubernetes.** Choose one of the listed VPCs, or click **Create New** to make one without leaving the form. See [Create a cluster](/myaccount/kubernetes/how-to/create-a-cluster/#choose-the-network). ## The plan is temporarily not available The form says **The plan is temporarily not available. You can try after some time or click on the 'Request' button to place a request.** The plan is out of stock. Click **Request** to ask for it, and E2E contacts you; **We have accepted your request, will contact you shortly.** confirms it. For a worker plan, if the request fails, email cloud-platform@e2enetworks.com. Or choose another plan. See [Create a cluster](/myaccount/kubernetes/how-to/create-a-cluster/#choose-how-the-master-node-is-billed). ## A LoadBalancer service stays on pending `kubectl get service` shows `` under `EXTERNAL-IP`. A LoadBalancer service gets its external IP from the cluster's **LB IP Pool**, one IP per service, and it waits while no IP in the pool is free. 1. To see why the service waits, run `kubectl describe service `, where `` is the name `kubectl get service` lists, and read its events. When the pool has no free IP, the events show: ```text Failed to allocate IP for "/": no available IPs ``` 2. Open the cluster's **LB IP Pool** tab. If it says **Currently no Service IP attached**, attach one. If every IP on the tab is already used by another LoadBalancer service, attach one more. See [Expose a service with a load balancer](/myaccount/kubernetes/how-to/expose-a-service-with-a-load-balancer/#attach-a-public-service-ip-to-the-pool). 3. If the tab says **Currently no reserved Service IP is available**, click **Click here** to reserve one first. A public service IP counts toward your account's reserved-IP limit and needs enough infra credits. The service picks up the address once the pool has one for it; you do not need to create the service again. ## Failed to allocate IP: is not allowed in config A LoadBalancer service stays on ``, and its events show a message like this one: ```text Failed to allocate IP for "/": [""] is not allowed in config ``` The service asks for an IP that is not in the cluster's **LB IP Pool**. Attach that IP to the pool, or change the service's MetalLB annotation to an IP the pool holds, and apply it again. See [Give a service a private LB IP](/myaccount/kubernetes/how-to/expose-a-service-with-a-load-balancer/#optional-give-a-service-a-private-lb-ip). ## A pod stays Pending `kubectl get pods` shows the pod as `Pending`: Kubernetes has not found a node to run it on. Run `kubectl describe pod `, where `` is the name `kubectl get pods` lists, and read the events at the end: - `Insufficient cpu` or `Insufficient memory`: the nodes have no room for the pod's requests. Add nodes to a pool, or add a pool with a bigger plan; see [Manage node pools](/myaccount/kubernetes/how-to/manage-node-pools/#resize-a-pool). - `didn't match Pod's node affinity/selector` or `untolerated taint`: the pod's node selector or the nodes' taints keep it off every node. Check the labels and taints; see [Set labels and taints with kubectl](/myaccount/kubernetes/how-to/manage-node-pools/#set-labels-and-taints-with-kubectl). - A claim that is not bound: see [A volume claim stays Pending](#a-volume-claim-stays-pending). ## A pod reads ImagePullBackOff, CrashLoopBackOff, or OOMKilled These come from the pod itself, and Kubernetes' own guide covers them; see [Debug Pods](https://kubernetes.io/docs/tasks/debug/debug-application/debug-pods/) on the Kubernetes site. For images in E2E's Container Registry, see [Pull images from a Kubernetes pod](/myaccount/container-registry/how-to/pull-images-from-a-kubernetes-pod/). ## Pods cannot resolve service or external names A pod cannot reach another service by name, or an address on the internet by host name. Name lookups inside a cluster go through CoreDNS. Check, in order: 1. The CoreDNS pods read `Running`: ```bash kubectl get pods -n kube-system -l k8s-app=kube-dns ``` 2. A test pod can resolve the cluster's own API name: ```bash kubectl run dnstest --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default ``` The command prints the name's address, then deletes the test pod. 3. If lookups still fail, restart CoreDNS, and try again: ```bash kubectl rollout restart deployment coredns -n kube-system ``` See [Connect to a cluster with kubectl](/myaccount/kubernetes/how-to/connect-with-kubectl/). ## A node reads NotReady `kubectl get nodes` shows a node as NotReady, so Kubernetes stops placing pods on it. To see why, run: ```bash kubectl describe node ``` `` is the name in the NAME column of `kubectl get nodes`. Under `Conditions`, a condition that reads `True` other than `Ready` names the cause: - DiskPressure: the node's disk is nearly full. Remove what fills it, such as large logs or files written inside containers, or move that data to a persistent volume. - MemoryPressure: the pods on the node use nearly all its memory. Set memory requests and limits on your pods, or add nodes so the pods spread out. To restart a node, open the cluster's **Node Pool** tab, and in the node's **Actions**, click **Reboot**. The **Monitoring** tab shows the node's RAM, storage, and CPU graphs; see [Monitor a cluster](/myaccount/kubernetes/how-to/monitor-a-cluster/). If the node stays NotReady, [contact support](#contact-support) with the cluster's name, the node's name, and the output of `kubectl describe node`. ## A node I deleted with kubectl still shows on the Node Pool tab `kubectl delete node` removes a node from Kubernetes only. Its machine keeps running and is billed, and it still reads **Running** on the cluster's **Node Pool** tab. Delete it from the tab: in the node's **Actions**, click **Delete**, and confirm. You see **Node deletion process initiated successfully. It takes 3-5 minutes for the pool to get updated.** See [Manage node pools](/myaccount/kubernetes/how-to/manage-node-pools/#delete-a-worker-node). ## The node was low on resource: ephemeral-storage A pod is evicted with this message: ```text The node was low on resource: ephemeral-storage ``` The pods on the node wrote more to the node's own disk than it has room for, and Kubernetes evicted some to free space. Set ephemeral-storage requests and limits on your pods, so they are placed on nodes with room, and keep data that grows, such as uploads or database files, on a persistent volume. See [Use persistent volumes](/myaccount/kubernetes/how-to/use-persistent-volumes/). ## MountVolume.MountDevice failed with DeadlineExceeded A pod that uses a claim stays in ContainerCreating, and `kubectl describe pod ` shows an event like this one: ```text MountVolume.MountDevice failed ... rpc error: code = DeadlineExceeded ``` The node could not reach the volume's storage in time. Check, in order: 1. The cluster is in Delhi, where persistent volumes are available. 2. The storage driver's provisioning log shows no errors for the claim: ```bash kubectl logs --namespace default deploy/csi-rbdplugin-provisioner -c csi-provisioner ``` If the pod still does not start, [contact support](#contact-support) with the claim's name, the pod's name, and the event text. See [Use persistent volumes](/myaccount/kubernetes/how-to/use-persistent-volumes/). ## Cannot delete the node! The minimum node count for the pool is 2. A worker node cannot be deleted when its pool is at its minimum: 2 nodes on a CPU pool, and 1 on any other pool, such as **GPU** or **GDC 3rd generation**, where the message names 1. To go below the minimum, delete the whole pool. See [Manage node pools](/myaccount/kubernetes/how-to/manage-node-pools/#delete-a-worker-node). ## Cannot delete node service as minimum one node pool is required to run kubernetes cluster. A cluster keeps at least one worker node pool, so its last pool cannot be deleted. To replace it, add the new pool first, then delete the old one. To remove every pool, delete the cluster. See [Manage node pools](/myaccount/kubernetes/how-to/manage-node-pools/#delete-a-pool). ## Currently inventory is not available Growing a GPU pool is refused with this message when E2E has no GPU capacity. Try again later, or [contact support](#contact-support) with the plan and the number of nodes you need. See [Manage node pools](/myaccount/kubernetes/how-to/manage-node-pools/#resize-a-pool). ## Resize Pool or Edit Node Pool is not available Wait until the pool reads **Running**, and the cluster too: the **Node Pool** tab shows its pools while the cluster reads **Running**. To add capacity to a committed pool, add a pool. **Edit Node Pool** is available on autoscaling pools. See [Manage node pools](/myaccount/kubernetes/how-to/manage-node-pools/). ## The pool drawer refuses the autoscaling values - **The maximum worker nodes must be greater than the desired worker nodes.**: set **Max Nodes** to at least **Min Nodes**. - **Recurrences should be different.**: the **Upscale Recurrence** and the **Downscale Recurrence** must be different cron expressions. **Min Nodes** can be 2 to 20, or 1 to 20 on the **GPU** family, and **Max Nodes** up to 25. See [Configure autoscaling for a node pool](/myaccount/kubernetes/how-to/configure-autoscaling/). ## An autoscaling pool adds nodes with little load An **Elastic** pool on **CPU** follows the worker nodes' CPU use as E2E measures it on the machines, as a share of their vCPUs. It does not follow the CPU that pods request. System and background processes on the nodes count too, so a pool can scale up while your own pods are quiet. Check the pool's policy on the **Node Pool** tab: a low **Scale Up (CPU %)**, or a short **Watch Period**, adds nodes sooner. To change them, click **Edit Node Pool**. See [Configure autoscaling for a node pool](/myaccount/kubernetes/how-to/configure-autoscaling/#scale-on-cpu). ## The Persistent Volume tab is missing Persistent volumes are available on clusters in Delhi. In Chennai, the cluster has no **Persistent Volume** tab, and a request to create one is refused with **Persistent Volume creation is currently not supported in the Chennai region.** For a volume, run the cluster in Delhi. For shared file storage your pods can mount, see [Use an SFS from Kubernetes](/myaccount/sfs/how-to/use-an-sfs-from-kubernetes/). ## A volume claim stays Pending `kubectl get pvc` shows the claim as `Pending`. To see why, run `kubectl describe pvc `, where `` is the name `kubectl get pvc` lists, and read its events. A claim creates its own volume when it names the storage class `csi-rbd-sc`. The cluster gets that storage class when you create its first persistent volume on the **Persistent Volume** tab, in Delhi. Run `kubectl get storageclass` to check that the cluster has it, and name `csi-rbd-sc` as the claim's storage class. The storage driver that creates the volumes runs in the `default` namespace as the Deployment `csi-rbdplugin-provisioner` and the DaemonSet `csi-rbdplugin`. To read why a volume was not created, read its provisioning log: ```bash kubectl logs --namespace default deploy/csi-rbdplugin-provisioner -c csi-provisioner ``` See [Use persistent volumes](/myaccount/kubernetes/how-to/use-persistent-volumes/#create-a-volume-from-a-claim). ## Pod attach to PVC. Please delete pod first. Deleting a persistent volume is refused with this message while a pod still uses it. Find and delete the pod that mounts the claim, then delete the volume again. To list the pods and the claims they use, run: ```bash kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,POD:.metadata.name,CLAIM:.spec.volumes[*].persistentVolumeClaim.claimName' ``` See [Use persistent volumes](/myaccount/kubernetes/how-to/use-persistent-volumes/#delete-a-volume). ## MountVolume.MountDevice failed with RBD image not found A pod that uses a claim stays in ContainerCreating, and `kubectl describe pod ` shows an event like this one: ```text MountVolume.MountDevice failed … (image not found: RBD image not found) ``` The claim's volume was deleted from the **Persistent Volume** tab. Its storage is gone, but its PersistentVolume and claim stay in the cluster reading `Bound`. Delete the pod, then clear the claim and the volume, where `` is the claim's name and `` its volume, as `kubectl get pvc --namespace default` lists them: ```bash kubectl delete pvc --namespace default kubectl delete pv ``` The data that was on the volume is gone; to keep data, restore it from your own backup into a new volume. See [Use persistent volumes](/myaccount/kubernetes/how-to/use-persistent-volumes/#delete-a-volume). ## Converting says the node is already committed Converting is refused with a message like this one: ```text Kubernetes node is Already committed. ``` The master node or the pool is already on a committed plan, so it cannot be converted again. To choose what happens when its term ends, use **Committed Cluster Settings** or **Committed Node Pool Settings**. Converting also needs the cluster or pool to read **Running** or **Warning**. See [Use committed billing for a cluster](/myaccount/kubernetes/how-to/use-committed-billing/). ## Auto-scaling node pool cannot be launched as a committed node pool The message names the pool: ```text Auto-scaling node pool cannot be launched as a committed node pool. ``` An autoscaling pool is always billed by the minute. Choose **On-Demand** for it, or make it a static pool to commit it. See [Use committed billing for a cluster](/myaccount/kubernetes/how-to/use-committed-billing/). ## A tab asks you to wait while the cluster is Creating or Upgrading Master A tab shows **Cluster is currently in state, please wait.**, where `` is the cluster's state. While a cluster reads **Creating** or **Upgrading Master**, only its **Cluster Details** tab works. Wait until the cluster reads **Running**. If it still reads **Creating** after 80 minutes, see [The cluster stays on Creating](#the-cluster-stays-on-creating). For the upgrade, see [Upgrade the master plan](/myaccount/kubernetes/how-to/upgrade-the-master-plan/). ## Cluster is in Terminating state A cluster that reads **Terminating** is being deleted and cannot be opened. See [Delete a cluster](/myaccount/kubernetes/how-to/delete-a-cluster/). ## SSL Renewal Not Allowed!!! You can renew the certificate yourself only in the five days before its renewal date. At other times, E2E renews it on the date while the cluster reads **Running**. See [Renew the cluster certificate](/myaccount/kubernetes/how-to/renew-the-cluster-certificate/). ## There are no plans higher than your current Kubernetes plan **Upgrade Kubernetes Master Plan** moves the master node to a bigger plan, with more vCPU and more RAM, and keeps the cluster's Kubernetes version. No such plan is available for this master. To ask for a bigger plan, contact cloud-platform@e2enetworks.com. See [Upgrade the master plan](/myaccount/kubernetes/how-to/upgrade-the-master-plan/). ## Atleast one security group must be present A cluster keeps at least one security group attached. Attach another group first, then detach this one. If you use **Allow All Traffic** and the group is already attached, you see **Security Group Already Attached.** See [Manage a cluster's security groups](/myaccount/kubernetes/how-to/manage-security-groups/). ## Your Resource does not belong to the project you have provided The cluster is in a different MyAccount project from the one you are working in. Switch to the project that holds the cluster, and try again. See [Kubernetes](/myaccount/kubernetes/). ## Create, Delete, or another action is not available to a team member Creating, changing, and deleting a cluster need the matching Kubernetes permission in your project. To get the permission you need, ask whoever manages your account's roles. See [Create a cluster](/myaccount/kubernetes/how-to/create-a-cluster/#before-you-begin). ## Contact support Write to support with the cluster's name, its region, the exact error text, and the time it happened. For a pod problem, add the events `kubectl describe` printed.