Troubleshooting when your site is down

A site stops responding for a reason somewhere in its stack: the node itself, its resources, the web server, the config, the database, the ports, or the DNS pointing at the site. A Webcheck on your site’s URL can notify you when it stops responding, so you know roughly when it went down. A Webcheck is a check that visits a URL on a schedule and alerts you if it is down or slow.

Work through the sections in order, from the node itself up to DNS. Stop when one tells you where the fault is. Each check either clears that layer or names the next place to look.

The node is reachable

Rule out the node itself first. From your own machine, ping the node’s public IP:

ping 203.0.113.10

A reachable node answers each packet:

64 bytes from 203.0.113.10: icmp_seq=1 ttl=57 time=18.0 ms

Then try SSH, because ping can lie: many systems drop ICMP even when they are up, so a ping failure on its own is not proof the node is down. SSH is the conclusive check.

ssh <username>@203.0.113.10

Replace <username> with the username from your node’s creation email. If SSH connects, the node is up and the fault is higher in the stack. If neither ping nor SSH works, the node is likely down or hung; see Troubleshooting node networking and connectivity for what to check next.

The node has free resources

A node can be running and still not serve pages if its memory, CPU, or disk is exhausted. The node’s details page shows uptime, a memory chart, and a storage chart before you log in, so check those for the period of the outage first. A disk that is completely full can even stop SSH, so rule it out early; see Troubleshooting node performance and resources when a resource is at its limit.

Log into the node and watch its own tools to see what is using it right now:

top

The logs point at the cause

The web server log is the fastest way to the root cause, because the server usually writes the error there before it fails. Logs live under /var/log/. Apache on Ubuntu or Debian:

/var/log/apache2/error.log
/var/log/apache2/access.log

Apache on RHEL or CentOS, where the service is called httpd:

/var/log/httpd/error_log
/var/log/httpd/access_log

Nginx:

/var/log/nginx/access.log
/var/log/nginx/error.log

Tail the newest entries and watch them arrive:

tail -f /var/log/nginx/error.log

Or page through the full file when you want the history around the outage:

less /var/log/nginx/access.log

If you set custom log paths, look there instead. Otherwise, the default paths in this section are where the logs are.

The web server is running

A server can run while one of its services is stopped. Ask the service manager for its status, then start it if it is not running.

Apache on Ubuntu or Debian:

service apache2 status
service apache2 start

Apache on RHEL or CentOS:

service httpd status
service httpd start

Nginx:

service nginx status
service nginx start

Each of these answers with running or stopped. If it is stopped, start it, then re-check the site.

The web server config is valid

If the web server will not start after an edit, the config almost certainly has a syntax error, and the service manager will not have let it run at all. Test the config before you restart, so a bad file cannot take the site down a second time.

Apache:

apache2ctl -t
httpd -t

Run the one that matches your system: apache2ctl on Debian and Ubuntu, httpd on RHEL and CentOS. Either prints Syntax OK when the config is valid.

Nginx:

nginx -t

A valid config prints test is successful. A test failed message names the file and line with the bad directive, so fix that, re-run the check, and only then restart the service.

The database is up

If the site depends on a database, confirm that service is running too. Which one to ask depends on what the site uses:

service mysql status
service mysqld status
service mongod status

The port is a second, independent check: a database that is listening can be reached even when its service status is confusing.

netstat -ntlp | grep 3306

A healthy MySQL listener looks like this, with the process listening on port 3306:

tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 3356/mysqld

The web server can reach the database

The web server and the database can both be running and the site still fail, if the application cannot authenticate or connect. For a WordPress site, the connection settings live in the wp-config.php file: check the database name, the database user, and the database password it holds. Test those credentials against the database directly:

mysql -h <host> -u <user> -p

Replace <host> with the database hostname and <user> with the database user. Enter the password when prompted, rather than on the command line, so it does not appear in your shell history. A successful login means the database accepts the credentials and is reachable. If it fails, the message tells you whether the problem is the credentials, the hostname, or the network.

The ports are open

The web server listens on port 80 for HTTP and 443 for HTTPS; the database listens on its own port, 3306 for MySQL by default. From your own machine, test that traffic can reach the node’s web ports:

telnet 203.0.113.10 80
telnet 203.0.113.10 443

An open port connects and shows an escape character prompt. If the application server and the database run on different nodes, test the database port from the application server:

telnet 203.0.113.20 3306

If a port is unreachable, check the node’s security group and its own firewall, and open the port that is closed. The database port in particular often needs to allow the web server before the site can load.

DNS resolves to the node

If the site works when you visit the public IP directly but not by its domain name, the DNS record is wrong or stale. Ask the resolver for the address:

dig A example.com +short

Expected output is the IP of your node. If it returns a different IP, or nothing, fix the DNS record. If it returns the right IP and the site still will not load, check the web server’s virtual host (Apache) or server block (Nginx) for the domain to confirm it is configured to answer for that hostname.

If you have worked through every layer and the site is still down, contact support with the node’s name, IP address, and your customer reference number (CRN), the exact error text, and when it happened.