Reboot Survival – Best Practices

Introduction

Cloud infrastructure offers resilience but all servers depends on its physical hardware, which in the long run need maintenance. Round the clock servers depends on uptime which is optimum for its usage but unavoidable downtime do happen. Apart from hardware maintenance, Update and Patches requires rebooting your server which makes your system secure and up-to-date. The guide below covers best practices to make sure your system handles unexpected reboot which will lead to downtime.

Test your Stack by Manual Reboot

During the development phase, When you make important changes on your stack, make sure you reboot your server once and ensure that your stack is fully operational without any issue. By doing this Reboot trail often you will be prepared for unexpected reboot that happens during maintenance or upgrades.

Backup your Crucial Data

Backing up you most important and critical data is one of the most important steps, Losing the data by unexpected downtime or by any other accidental deletion will have an adverse impact on your stack.E2E Networks CDP Backups works as an exact replica of your file system which is backed up automatically at periodic intervals. Each backup created contains all the data which was backed up to restore from the provisioning till the most recent block update. This allows point-in-time to restore from any of the recovery points present in the archive.You can refer here for more details.

You can also Backup your databases from MySQL or MariaDB and restore them by using mysqldump Command. For those who aren’t familiar with mysqldump you can refer here

Enable Services on Boot

You also need to make sure all your critical and important services and application that are configured in your server are able to start at boot,So that during future reboots,all the services are up and running without any manual intereption

In Debian and Ubuntu,Use update-rc.d to add or remove services from startup

sudo update-rc.d service_name defaults

To remove service from boot startup,Use below command

sudo update-rc.d -f service_name  remove

In Centos7,Use systemctl command

sudo systemctl enable service_name

sudo systemctl disable service_name

Ensure Firewall rules are saved

If you have manually configured the rules in firewall,You should make sure its already be saved, and gets loaded on boot automatically.

You can save the firewalls permanently by running the below command.

sudo iptables-save > /etc/iptables.firewall.rules

Using Load balancer

If your system totally cannot afford downtime, You can use a Load Balancer Appliance which helps you to dynamically distribute user traffic to multiple backend nodes — for high availability, scalability, fault-tolerance, and smooth user experience. It also helps you to apply changes to the backend without impacting the end-user experience. You can refer this article on how to launch a Load balancer in E2E .