Skip to main content

How To Use Logrotate and S3cmd to Archive Logs to Object Storage

Introduction​

Log files generated on your application and services are sometimes very useful for debugging your application, investigating for security purposes, or general data insights. Storing logs on your server can be a little bit painful. Often, it’s not possible to retain long-term data in these systems due to storage constraints or other server resource issues. A common solution for these long-term storage needs is archiving logs with an object storage service.

In this article, we will show you exactly how to use Logrotate and S3cmd to archive logs to Object Storage.

Prerequisites​

  • An Ubuntu or CentOS server with sudo access.
  • Familiarity with Logrotate and its default configuration and setup.
  • Bucket in E2E Object Store. If you have not created a bucket yet, please refer to Getting Started section.
  • Setting up s3cmd on your server. If you haven't set it up, please refer to the tutorial: Setting up s3cmd.

Configuring Logrotate​

Once you have set up a bucket in Object Storage and installed S3cmd on your server, you can easily archive logs to the Object Storage bucket by making a slight modification to your Logrotate config.

In this example, we are configuring Logrotate for message logs that are compressed and synced to the EOS bucket when the log file size exceeds 1GB.

  • Create a configuration file in the /etc/logrotate.d/ directory. We are naming it message.conf.
vim /etc/logrotate.d/message.conf
  • Assuming the messages logs are stored in the /var/log/message file, Copy paste the below Configuration on the File and Save it
/var/log/messages {
     rotate 3
     size 1G
     missingok
     notifempty
     compress
     copytruncate
     dateext
     dateformat -%Y-%m-%d-%s
     postrotate
     s3cmd sync  /var/log/messages*.gz "s3://bucketname/"
     endscript
}
Note

Replace the message Logs path,Bucket Name and other config as per your requirment in the above configuration

Below is the explanation of the configuration file we just used:

  • rotate 3: Keep three old log files.
  • size 1G: Rotate log files if the size is greater than 1GB.
  • missingok: Don’t write an error message if the log file is missing.
  • notifempty: Don’t rotate the log file if it is empty.
  • compress: Compress the rotated files. This uses gzip by default and results in files ending in .gz.
  • copytruncate: Truncate the original log file in place after creating a copy.
  • dateext: Archive old versions of log files by adding a daily extension like YYYYMMDD instead of simply adding a number.
  • dateformat -%Y-%m-%d-%s: Specify the extension format for dateext using the given notation.
  • The lines between postrotate and endscript (both of which must appear on lines by themselves) are executed (using /bin/sh) after the log file is rotated.

Here, the script we have used is to sync the messages log to the Object Storage bucket.

s3cmd sync  /var/log/messages*.gz "s3://yourbucketname/"

You can also include the Logrotate to Run based on timing like Monthly or weekly if you want, which we have not included in the above example.

Test and Executing Logrotate Config File​

After customizing the config to fit your needs and saving it in /etc/logrotate.d, you can test it by doing a dry run, You can test the Logorate config to check if its working as expected. The below command will show you just the Output of the Task and Not execute it. If all looks well, you’re done

logrotate -d /etc/logrotate.d/message.conf

You can Execute the Lograte Forceful by below command, Which otherwise would not run until the condition we provided in config is met. This is useful when testing the scripts etc.. which we added in Logrotate config file.

logrotate -f /etc/logrotate.d/message.conf

Conclusion​

In this tutorial, we have covered the basic configuration of Logrotate for setting up and archiving logs to EOS buckets. You can also use many other configurations available in Logrotate (refer to all the options by running the command man logrotate in your terminal). If you face any issues while configuring the same and need assistance, you can contact cloud-Platform@e2enetworks.com, where our team can assist you.