Troubleshooting node performance and resources

For a node that has run out of disk space, or whose CPU is stuck at 100 percent, start with the storage and memory charts on the node’s own details page: they are the one view you get before you log in to the node. To find out exactly what is using the disk or the CPU, log into the node and use its own command-line tools. For the disk, see Find what is filling the disk. For the CPU, see Check what is using the CPU.

Disk full, SSH connection fails

A disk that is completely full can stop SSH from working, because the system can no longer write the files a new session needs.

Confirm it first from the node’s own details page: the storage chart there shows how full the disk is, without needing to log in at all.

If the node is still Running, try Access Console first: it opens the node’s screen directly in the browser without needing SSH, letting you delete files straight from that session. See Open a node’s console.

If the node is not Running, or Access Console does not give you a working session, turn on Enable Recovery Mode from the Actions menu instead. It needs the node Powered off first. Recovery mode boots the node from a separate rescue disk, so Access Console reaches it even with the original disk full. See Enable recovery mode for those steps.

From there, the full disk is mounted at /mnt/sysroot, not at /. Run the commands in Find what is filling the disk against that path to find what to delete. On an E1-series node, you can increase the root storage instead. Once there is room again, turn recovery mode back off to boot from the normal disk.

Storage chart shows almost full

Catch this before it locks you out. A disk that fills up completely can stop SSH from working, so acting while there is still room avoids reaching that point. Check the storage chart on the node’s details page from time to time, or set a Free Disk Space alert so you are notified before it fills; see Set up alerts on a node.

For an E1-series node, increase root storage from the Actions menu instead of waiting for it to run out; see Increase a node’s root storage. Other series bundle disk size into the plan itself: attach a new volume from the Volumes tab for extra space, or move to a larger plan; see Upgrade a node’s plan. Upgrading a plan needs at least 500 MB of free disk space, so free some room first if the disk is that full.

Find what is filling the disk

Once you have a working session on the node, use its own tools to find what is using the space. Start with df -h to see how full each filesystem is:

df -h

df -h lists every mounted filesystem, how much space it is using, and how much is free. The h makes it show sizes people can read, like 1.5G, instead of raw blocks. Watch the Use% column: the filesystem at or near 100% is the one you need to clear.

df tells you how full each filesystem is, but not which files fill it. To find that, walk the directory tree with du:

du -sh *

Run this inside the directory you want to inspect, such as /var or your home directory. du -sh * prints a size for every entry in the current directory. The s keeps it to one summary line per entry and the h uses readable sizes. The largest entry is where the space went; run the command again inside that directory to keep narrowing it down.

For a quicker interactive view, use ncdu:

ncdu /

ncdu is an interactive version of du: it scans the path you give it and shows the biggest directories in a list. Move through the list with the arrow keys, delete a selected file with d, and quit with q. If it is not installed, install it with your package manager first: for example yum install ncdu on RHEL or CentOS, or sudo apt-get install ncdu on Ubuntu or Debian.

Most full disks come down to one of three things:

• Log files in /var/log. Logs grow steadily, faster when an application is noisy or failing. Set up logrotate to keep them to a known size. To empty a log you do not need without stopping the program writing to it, run truncate -s 0 /var/log/<log-file>.
• Backups kept on the node itself. A local backup that is never deleted grows with every run. Delete old ones on a schedule, or move them off the node, for example to E2E Object Storage.
• Temporary files in /var/tmp. Programs leave files here and forget them, and unlike /tmp it is not cleared at reboot. Check what is there and delete what is no longer needed.

Once you know what is taking the space, delete or move files to free it up. If the disk keeps filling because your workload genuinely outgrew the node, the longer-term fix is in the portal: attach a volume, move to a bigger plan, or, for an E1-series node, increase root storage.

Disk looks full but du disagrees

Sometimes df reports the disk is full but du cannot account for all the space. That happens when a running process still has a file open that was already deleted from the filesystem. The file no longer shows up in du, but its space stays held until the process closes the file or the node restarts.

Find those files with:

lsof | grep deleted

lsof lists every open file on the system. Filtering for deleted shows the ones that were removed but are still held open, and each line names the process holding the file. Restart that process, or reboot the node, and the kernel releases the space the file was using.

CPU stuck at 100%

The node’s details page shows uptime, a memory chart, and a storage chart, but no live CPU chart. E2E never restarts a node or changes its plan because of its usage, and an alert only notifies you, so a node stuck at 100% stays there until you act on it. To check what is using the CPU right now, log into the node and use its own tools: Task Manager on Windows, or the commands in Check what is using the CPU on Linux.

To watch CPU load going forward without logging in each time, set a Processor Load alert; see Set up alerts on a node. If the graphs are not showing data yet, see Troubleshooting node monitoring and encryption.

Check what is using the CPU

Log in and run uptime first for a quick overview of the load:

uptime

uptime shows how long the node has been running and the load average over the last 1, 5, and 15 minutes. Read the three numbers left to right. If they are consistently higher than the number of CPUs the node has, too many processes are waiting for CPU time. A 1-minute value higher than the 15-minute value means load is rising; a lower one means it is recovering.

To see which process is using the CPU, run top:

top

top lists the running processes sorted by CPU usage, so the ones at the top are the ones driving the load. Press P to sort by CPU, or M to sort by memory, and q to quit.

To snapshot one specific process, use ps with the service name:

ps -ef | grep mysql

Replace mysql with the service you suspect. The output lists every matching process and its PID, which you need if you want to stop a runaway worker without touching the parent service.

If you find a process that is misbehaving, stop or restart it. Try a graceful stop first:

pkill mysql

Replace mysql with the actual process name. pkill sends a termination signal that gives the process a chance to clean up. If it does not respond, force it by its PID:

kill -9 pid

Replace pid with the process’s PID from ps or top. kill -9 is a last resort: the process cannot refuse it, but it may leave temporary files behind. If a genuinely busy workload, rather than a misbehaving process, is driving the load, moving to a bigger plan is the better fix.

If the node stays slow or full, give support its name, IP address, and CRN, what top and df -h showed, and when it started.