Tutorial : How To Use Logrotate and S3cmd to Archive Logs to Object Storage

Introduction

Log files generated on your Application and services are sometimes very useful for Debugging your Application, Investigating for Security Purposes or General Data Insights. Storing Logs on Your server can be a little bit painful, Often it’s not possible to retain long-term data in these systems due to storage constraints or other server resource issues. A common solution for these long-term storage needs is archiving logs with an object storage service.

In this article, We will show you exactly How To Use Logrotate and S3cmd to Archive Logs to Object Storage

Prerequisites

  • An Ubuntu or Centos Server with a Sudo access

  • Familiarity with Logrotate and its Default Configuration and Set-up

  • Bucket in E2E Object Store with Necessary Permissions. If you have not created a bucket yet, please refer Getting Started Guide

  • Setting up s3cmd on your server. If you haven’t setup please refer Tutorial: Setting up s3cmd

Configuring Logrotate

Once You have Setup Bucket in Object Storage and Have Installed S3cmd on Your Server, You can easily Archive Logs to Object Storage Bucket by a Slight Modification on Your Logrotate Config.

In this Example, We are Configuring Logrotate for Messages Logs which are compressed and Synced to EOS Bucket when the Logs file gets greater than 1GB

  • Create a Configuration File in the path /etc/logrotate.d Directory, We are naming it as message.conf

vim /etc/logrotate.d/message.conf

  • Assuming the messages logs are stored in the /var/log/message file, Copy paste the below Configuration on the File and Save it

/var/log/messages {
     rotate 3
     size 1G
     missingok
     notifempty
     compress
     copytruncate
     dateext
     dateformat -%Y-%m-%d-%s
     postrotate
     s3cmd sync  /var/log/messages*.gz "s3://bucketname/"
     endscript
}

Note

Replace the message Logs path,Bucket Name and other config as per your requirment in the above configuration

Below are Explanation of Config File, We Just used.

  • rotate 3 - Keep three old Logs File

  • Size 1G - Rotate Log Files if the size is greater than 1G

  • missingok: don’t write an error message if the log file is missing.

  • notifempty: don’t rotate the log file if it is empty.

  • compress: compress the rotated files. this uses gzip by default and results in files ending in .gz

  • copytruncate: Truncate the original log file in place after creating a copy,

  • dateext: Archive old versions of log files adding a daily extension like YYYYMMDD instead of simply adding a number

  • dateformat -%Y-%m-%d-%s: Specify the extension for dateext using the notation

  • The lines between postrotate and endscript (both of which must appear on lines by themselves) are executed (using /bin/sh) after the log file is rotated.

Here, The script we have Used is to Sync the messages log to Object Storage Bucket

s3cmd sync  /var/log/messages*.gz "s3://yourbucketname/"

You can also include the Logrotate to Run based on timing like Monthly or weekly if you want, which we have not included in the above example.

Test and Executing Logrotate Config File

After customizing the config to fit your needs and saving it in /etc/logrotate.d, you can test it by doing a dry run, You can test the Logorate config to check if its working as expected. The below command will show you just the Output of the Task and Not execute it. If all looks well, you’re done

logrotate -d /etc/logrotate.d/message.conf

You can Execute the Lograte Forceful by below command, Which otherwise would not run until the condition we provided in config is met. This is useful when testing the scripts etc.. which we added in Logrotate config file.

logrotate -f /etc/logrotate.d/message.conf

Conclusion

In this tutorial we have covered the basic configuration of Logrotate for setting up and Archiving the logs to EOS Buckets. You can also use many other configuration which are available Logrotate (Refer all the Option by running the command man logrotate in your terminal). If you face any issue while configuring the same and need some assistance, You can contact cloud-Platorm@e2enetworks.com where our team can assist you on it.