Archive logs to E2E Object Storage with logrotate and s3cmd

Log files grow without bound unless something rotates them, and keeping them on the server forever eats its disk. To keep it in check, rotate a log file with logrotate and copy the rotated, compressed archives to an E2E Object Storage bucket with s3cmd, so the server keeps only a few recent archives and the long-term copy lives in a bucket.

This works from any Linux server that can reach the internet, including a node. You need a bucket and s3cmd configured first; see Connect with s3cmd.

Configure logrotate

Logrotate reads its jobs from /etc/logrotate.d/. Create a configuration file there for the log you want to archive; this example rotates /var/log/messages, keeping three archives, rotating once the file reaches 1 GB, and syncing the compressed archives to your bucket:

/var/log/messages {
     rotate 3
     size 1G
     missingok
     notifempty
     compress
     copytruncate
     dateext
     dateformat -%Y-%m-%d-%s
     postrotate
     s3cmd sync /var/log/messages*.gz "s3://<bucket-name>/"
     endscript
}

Replace the log path and bucket name with your own. The lines between postrotate and endscript run after the log is rotated; here they sync the archives to the bucket.

What each option does:

Option	What it does
rotate 3	Keep three old log files on the server.
size 1G	Rotate once the log file grows past 1 GB.
missingok	Do not warn if the log file is missing.
notifempty	Do not rotate an empty log file.
compress	gzip the rotated archives, giving them a .gz ending.
copytruncate	Copy the log and truncate the original in place, so the writer keeps going.
dateext	Name archives by date instead of by number.
dateformat -%Y-%m-%d-%s	The date format used for the dateext names.

Test the configuration

Logrotate will not run a job until its condition (the size, or the schedule) is met, so test it before relying on it. A dry run shows what would happen without doing it:

logrotate -d /etc/logrotate.d/<your-config>.conf

Force it to run now, even if the condition is not met, to exercise the whole flow including the sync to the bucket:

logrotate -f /etc/logrotate.d/<your-config>.conf

A successful forced run leaves a compressed archive in the bucket. The postrotate script runs through /bin/sh, so test the exact s3cmd sync command on its own first if the sync does not look right.

Schedule the log archive

Logrotate itself runs daily from the system scheduler, so a config in /etc/logrotate.d/ normally needs no extra schedule of its own. If you want a specific weekly or monthly rotation instead, set the schedule in the config (for example a weekly or monthly directive) and let the system’s daily logrotate run pick it up.

The bucket keeps every archive until you delete it. To have old ones removed after a set number of days, see Set a lifecycle rule.