--- title: "Troubleshooting node performance and resources" description: "What to do when a node runs out of disk space and blocks SSH, or its CPU is stuck at 100 percent." content_type: "troubleshooting" product: "node" platform: "MyAccount" url: "/myaccount/node/troubleshooting/performance-and-resources/" relationships: {"troubleshooting-for":["/myaccount/node/how-to/increase-root-storage/","/myaccount/node/how-to/upgrade-node-plan/","/myaccount/node/how-to/enable-recovery-mode/","/myaccount/node/how-to/connect-to-a-linux-node/","/myaccount/node/how-to/connect-to-a-windows-node/"]} --- # Troubleshooting node performance and resources For a node that has run out of disk space, or whose CPU is stuck at 100 percent, start with the storage and memory charts on the node's own details page: they are the one view you get before you log in to the node. To find out exactly what is using the disk or the CPU, log into the node and use its own command-line tools. For the disk, see [Find what is filling the disk](#find-what-is-filling-the-disk). For the CPU, see [Check what is using the CPU](#check-what-is-using-the-cpu). ## Disk full, SSH connection fails A disk that is completely full can stop SSH from working, because the system can no longer write the files a new session needs. Confirm it first from the node's own details page: the storage chart there shows how full the disk is, without needing to log in at all. If the node is still **Running**, try **Access Console** first: it opens the node's screen directly in the browser without needing SSH, letting you delete files straight from that session. See [Open a node's console](/myaccount/node/how-to/open-the-node-console/). If the node is not **Running**, or Access Console does not give you a working session, turn on **Enable Recovery Mode** from the **Actions** menu instead. It needs the node **Powered off** first. Recovery mode boots the node from a separate rescue disk, so **Access Console** reaches it even with the original disk full. See [Enable recovery mode](/myaccount/node/how-to/enable-recovery-mode/) for those steps. From there, the full disk is mounted at `/mnt/sysroot`, not at `/`. Run the commands in [Find what is filling the disk](#find-what-is-filling-the-disk) against that path to find what to delete. On an E1-series node, you can [increase the root storage](/myaccount/node/how-to/increase-root-storage/) instead. Once there is room again, turn recovery mode back off to boot from the normal disk. ## Storage chart shows almost full Catch this before it locks you out. A disk that fills up completely can stop SSH from working, so acting while there is still room avoids reaching that point. Check the storage chart on the node's details page from time to time, or set a **Free Disk Space** alert so you are notified before it fills; see [Set up alerts on a node](/myaccount/node/how-to/set-up-alerts-on-a-node/). For an E1-series node, increase root storage from the **Actions** menu instead of waiting for it to run out; see [Increase a node's root storage](/myaccount/node/how-to/increase-root-storage/). Other series bundle disk size into the plan itself: attach a new volume from the **Volumes** tab for extra space, or move to a larger plan; see [Upgrade a node's plan](/myaccount/node/how-to/upgrade-node-plan/). Upgrading a plan needs at least 500 MB of free disk space, so free some room first if the disk is that full. ## Find what is filling the disk Once you have a working session on the node, use its own tools to find what is using the space. Start with `df -h` to see how full each filesystem is: ```bash frame="none" df -h ``` `df -h` lists every mounted filesystem, how much space it is using, and how much is free. The `h` makes it show sizes people can read, like `1.5G`, instead of raw blocks. Watch the `Use%` column: the filesystem at or near 100% is the one you need to clear. `df` tells you how full each filesystem is, but not which files fill it. To find that, walk the directory tree with `du`: ```bash frame="none" du -sh * ``` Run this inside the directory you want to inspect, such as `/var` or your home directory. `du -sh *` prints a size for every entry in the current directory. The `s` keeps it to one summary line per entry and the `h` uses readable sizes. The largest entry is where the space went; run the command again inside that directory to keep narrowing it down. For a quicker interactive view, use `ncdu`: ```bash frame="none" ncdu / ``` `ncdu` is an interactive version of `du`: it scans the path you give it and shows the biggest directories in a list. Move through the list with the arrow keys, delete a selected file with `d`, and quit with `q`. If it is not installed, install it with your package manager first: for example `yum install ncdu` on RHEL or CentOS, or `sudo apt-get install ncdu` on Ubuntu or Debian. Most full disks come down to one of three things: - **Log files in `/var/log`.** Logs grow steadily, faster when an application is noisy or failing. Set up `logrotate` to keep them to a known size. To empty a log you do not need without stopping the program writing to it, run `truncate -s 0 /var/log/`. - **Backups kept on the node itself.** A local backup that is never deleted grows with every run. Delete old ones on a schedule, or move them off the node, for example to [E2E Object Storage](/myaccount/object-storage/). - **Temporary files in `/var/tmp`.** Programs leave files here and forget them, and unlike `/tmp` it is not cleared at reboot. Check what is there and delete what is no longer needed. Once you know what is taking the space, delete or move files to free it up. If the disk keeps filling because your workload genuinely outgrew the node, the longer-term fix is in the portal: attach a volume, move to a bigger plan, or, for an E1-series node, increase root storage. ## Disk looks full but `du` disagrees Sometimes `df` reports the disk is full but `du` cannot account for all the space. That happens when a running process still has a file open that was already deleted from the filesystem. The file no longer shows up in `du`, but its space stays held until the process closes the file or the node restarts. Find those files with: ```bash frame="none" lsof | grep deleted ``` `lsof` lists every open file on the system. Filtering for `deleted` shows the ones that were removed but are still held open, and each line names the process holding the file. Restart that process, or reboot the node, and the kernel releases the space the file was using. ## CPU stuck at 100% The node's details page shows uptime, a memory chart, and a storage chart, but no live CPU chart. E2E never restarts a node or changes its plan because of its usage, and an alert only notifies you, so a node stuck at 100% stays there until you act on it. To check what is using the CPU right now, log into the node and use its own tools: Task Manager on Windows, or the commands in [Check what is using the CPU](#check-what-is-using-the-cpu) on Linux. To watch CPU load going forward without logging in each time, set a **Processor Load** alert; see [Set up alerts on a node](/myaccount/node/how-to/set-up-alerts-on-a-node/). If the graphs are not showing data yet, see [Troubleshooting node monitoring and encryption](/myaccount/node/troubleshooting/monitoring-and-encryption/). ## Check what is using the CPU Log in and run `uptime` first for a quick overview of the load: ```bash frame="none" uptime ``` `uptime` shows how long the node has been running and the load average over the last 1, 5, and 15 minutes. Read the three numbers left to right. If they are consistently higher than the number of CPUs the node has, too many processes are waiting for CPU time. A 1-minute value higher than the 15-minute value means load is rising; a lower one means it is recovering. To see which process is using the CPU, run `top`: ```bash frame="none" top ``` `top` lists the running processes sorted by CPU usage, so the ones at the top are the ones driving the load. Press `P` to sort by CPU, or `M` to sort by memory, and `q` to quit. To snapshot one specific process, use `ps` with the service name: ```bash frame="none" ps -ef | grep mysql ``` Replace `mysql` with the service you suspect. The output lists every matching process and its PID, which you need if you want to stop a runaway worker without touching the parent service. If you find a process that is misbehaving, stop or restart it. Try a graceful stop first: ```bash frame="none" pkill mysql ``` Replace `mysql` with the actual process name. `pkill` sends a termination signal that gives the process a chance to clean up. If it does not respond, force it by its PID: ```bash frame="none" kill -9 pid ``` Replace `pid` with the process's PID from `ps` or `top`. `kill -9` is a last resort: the process cannot refuse it, but it may leave temporary files behind. If a genuinely busy workload, rather than a misbehaving process, is driving the load, moving to a bigger plan is the better fix. If the node stays slow or full, give support its name, IP address, and CRN, what `top` and `df -h` showed, and when it started.