Troubleshooting Steps when your Site is down.

If you have configured web check alert for your site, You might receive the alert for the below scenario’s during which the web-check alert will be triggered

  • When the site is DOWN

  • When the site loads very slow (i.e. when the site takes more than 60 seconds to load) -> This depends on the Timeout setting.

Below are steps to Troubleshoot the issue when your site is Down

Sometimes we may run into trouble when one of our web-server or site is down, We should be able to identify this problem which is likely causing the issues, and fix the problem.

Step 1 : Check the server status

Before proceeding with the mentioned steps,First we need to check if you are able to access the server itself. Sometimes the Server itself may have went down.You can verify this with ping and ssh command

ping your_server_ip

If you able ping your server and its UP,You will get the output as below

e2e@compaqlaptop:~$ ping 8.8.8.8

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

64 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=18.0 ms

64 bytes from 8.8.8.8: icmp_seq=2 ttl=57 time=20.1 ms

64 bytes from 8.8.8.8: icmp_seq=3 ttl=57 time=21.3 ms

^C

--- 8.8.8.8 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2002ms

rtt min/avg/max/mdev = 18.042/19.837/21.342/1.362 ms

e2e@compaqlaptop:~$

If your server is down,in that case you will get not get any ping.You can Check your server console for any error and restart your server via Myaccount

Note

You may sometimes not be able to ping the server even when your server is UP and this may be due to ping being disabled on your server

You should also be able to SSH and login to the server to verify if its up and running fine, You can access your server via ssh with below command

ssh username@xx.xx.xx.xx

if you are not able to login to your server, Chances are that your server is down or in a hanged state and Check your server console for any error and restart your server via Myaccount

Step 2 : Monitoring your server

A high processor load or memory consumption can make your server unresponsive. You can refer to this article on how to Monitor E2E server via Zabbix

Once you have identified the anomalies in your server resources utilization,You can take necessary action accordingly to prevent the issue.

Step 3 : Check the Logs

Before tracking down any of the problems, We need to check the Logs first of our web server and related components. If we found any error logs or any suspicious logs on access logs, We will be able to identify the problems Precisely.

The logs will be usually present at directory /var/log. if you have an Apache server running on an Ubuntu server, by default the logs will be kept in /var/log/apache2 . Check the files in this directory to see what kind of error messages are being generated.If you’re using a distribution that refers to Apache as httpd by default the logs will be kept at

/var/log/httpd.log

If you are using Nginx as your web server, The logs are usually located at the below location

/var/log/nginx/access.log
/var/log/nginx/error.log
/var/log/nginx/nginx_error.log
/var/log/nginx/access_error.log

Check the files in this directory to see what kind of error messages are being generated. Alternately you can also check the web-server configuration files if you have configured a manual path for logs.

To check the latest logs generated from the server, You can run the below tail command as shown in the below example

tail -f /var/log/nginx/access.log

tail -f /var/log/httpd.log

You can also view with less command to view full logs as below,Make sure to change the log file path as per yours in below command.

less /var/log/nginx/access.log

once you have identified the issue over the logs, You can easily be able to detect the issue and fix the problem accordingly.

Step 4: Make sure your web server is running

You can verify if your services are up and running fine, One method to check if the services are running is distribution’s preferred method is to check with below-shared command.

Use the status command to check if the service is up or not. If the service is not running you can start it with the below start command

If you’re using a distribution that refers to Apache as Apache2, commands to use apache2 functionality is as below

service apache2 start

service apache2 stop

service apache2 restart

service apache2 reload

service apache2 status

If you’re using a distribution that refers to Apache as httpd, then the commands are as follows:

service httpd start

service httpd stop

service httpd restart

service httpd reload

service httpd status

The commands to use Nginx functionality is as below

service nginx start

service nginx stop

service nginx restart

service nginx reload

service nginx status

Step 5 : Verifying the Syntax of Web server

If your web server is failing to start,this means that there can be an issue with your configuration file.

Both Apache and Nginx need correct directive syntax in order for the files to be read.The configuration files are located as below

Default Configuration Directories for Apache is

Debian and Ubuntu distributions : /etc/apache2/ Fedora, CentOS distributions : /etc/httpd/

Configuration Directories for Nginx is

/etc/nginx/

Each of these web servers also provide you with the ability to check the configuration syntax of your files.To check the syntax of your Apache configuration files without needing to restart the server, you can run the following command on Debian and Ubuntu systems

apache2ctl -t

httpd -t

To check configuration Syntax on Nginx,Use below command

nginx -t

Once you run the above command you will get message like Syntax OK or Test is successful,This means that your there is no error at your web server configuration.if you get message like “test failed”,There is an invalid argument which are provided in the configuration file and that needs to be edited.

Step 6 : Is your Database back-end running Fine

If you have configured your site to connect with database back-end like MySQL, PostreSQL, MongoDB, etc. you need to make sure that it is up and running.You can do this as you checked for Web server.

Run the below command to verify if your database MySQL/Mongod whichever you are running.

service mysql status

service mysqld status

service mongod status

Alternately you can also verify with below netstat command

netstat -ntlp | grep mysql

You will receive output as below if your MySQL is up and running.

tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 3356/mysqld

Step 7 : Verify if your Web/App server is able to connect to Database backend

Even though if your web server and Database server is Up and running fine,Sometime we may face issue with its connectivity as your Application will not be able to connect to Database successfully.

For instance, In a WordPress site, the database connection settings can be tested with the configuration located in wp-config.php file. You need to check that the DB_NAME, DB_USER, and DB_PASSWORD are correct in order for your site to connect to the database.

You can test whether the file has the correct information by trying to connect to the database manually by changing appropriate Value in below command

mysql -hDB_Host -uDB_USER -pDB_PASSWORD

Step 8 : Make sure the Ports are open

Even if the all the configuration and connectivity are right,Some times you not be able to access the site since the configured ports must be accessible.Web servers run on port 80 for normal web traffic and use port 443 for traffic encrypted with SSL.

You can check whether if the configured ports are open with telnet command

telnet your_server_ip 80

telnet your_server_ip 443

You will get output like below if your ports are open

e2e@compaqlaptop:~$ telnet xx.xx.xx.xx 80
Trying xx.xx.xx.xx…
Connected to xx.xx.xx.xx.
Escape character is ‘^]’.

You can also verify if your App/Web server is able to connect to your database port on back-end.MySQL server by default run on port 3306

telnet your_database_server_ip 3306

If your web ports or Database ports is not accessible, you should look at your firewall configuration. You may need to open up port 80,port 443 or Port 3306 respectively.

Step 9 : Verifying the DNS Setting

You also need to make sure that your domain is correctly pointed to Server IP.if you can access the site with IP and not with your domain name,you may need to take a look at your DNS settings.

You can verify if your site is pointed to the right IP with below dig command

dig A example.com +short

You will get the output as below

e2e@compaqlaptop:~$ dig A example.com +short
xx.xx.xx.xx
e2e@compaqlaptop:~$

Apart from that,check your Apache virtual host files or the Nginx server block files to make sure they are configured to respond to requests for your domain.

Conclusion

We hope that the above mentioned troubleshooting tips will have helped you track down and fix some of the common issues that administrators face when trying to get their sites up and running.If you are encountering any issue with the above mentioned steps,feel free to discuss them in the comments.