We want to keep archive logs for a while, but they're big.
Solution: store them in containers (it works for shipping companies!)
This charm provides a way to archive log files from a given directory to
container-based storage (currently supports Amazon S3, OpenStack Swift and
Google Compute Engine (GCE)).
It is a subordinate charm, requiring either a "log-archive-relation-changed"
hook in the parent to provide the directory to be archived and when to archive
it or have those same values passed as charm options. Note: If both methods
are used, all the directories will be archived, but the timing from the relation
is preferred over a charm option.
e.g. relation-set logdirs="base64_dir1 base64_dir2"
Any files older than "archive_after" days in the specified directories will be
archived to container storage. Deletion of local log files is assumed to be
handled by the parent charm unless the "delete_after_archive" config variable is
set to true (in which case the charm will remove the log file immediately
after it is successfully archived)
The logdir relation variable is a space-separated list of names. To avoid any
issues with unusual filenames, each name must be base64-encoded. This is not
true when passed as a charm option.
e.g. To archive files in both /var/log/logdir1 and /var/log/logdir2:
Logdirs can be either a directory name, where everything in it will be archived
or a glob pattern, where just matches will be archived. If you want just one
file archived, use a glob pattern that only matches that file
Note: Using "delete_after_archive" on a log file still being written to may
lead to filesystem weirdness (why would you archive an active log file anyway?)
Also using this option when multiple processes are parsing archived logs can lead
to unpredictable behavior. You have been warned.
The container_credentials variable should be a base64-encoded JSON string with the
authorisation details needed to access the container. For Swift it should look
For GCE use the standard JSON file Google provides:
It is also possible to add pre-processors that are run against the log files
before they are updated to swift. The available pre-processors include gzip,
gunzip and a web log anonymizer. To specify pre-processors just specify a space
separated list in the pre_processors variable and they will run in order.
preprocessors="gunzip anonymize_web_log_ips gzip"
Some preprocessors take additional options for example:
anonymize_web_log_options="--skip-private -s 254.254.254.0/24"
If you would like to send raw logs to one location and pre-processed logs to
another the charm can be installed multiple times with different Juju
application names. In this case be careful with the delete_after_archive
option, it is best avoided.