container log archive #4


We want to keep archive logs for a while, but they're big.
Solution: store them in containers (it works for shipping companies!)

This charm provides a way to archive log files from a given directory to
container-based storage (currently supports Amazon S3, OpenStack Swift and
Google Compute Engine (GCE)).

Common Configuration

It is a subordinate charm, requiring either a "log-archive-relation-changed"
hook in the parent to provide the directory to be archived and when to archive
it or have those same values passed as charm options. Note: If both methods
are used, all the directories will be archived, but the timing from the relation
is preferred over a charm option.

e.g. relation-set logdirs="base64_dir1 base64_dir2"
relation-set archive_after=1

Any files older than "archive_after" days in the specified directories will be
archived to container storage. Deletion of local log files is assumed to be
handled by the parent charm unless the "delete_after_archive" config variable is
set to true (in which case the charm will remove the log file immediately
after it is successfully archived)

The logdir relation variable is a space-separated list of names. To avoid any
issues with unusual filenames, each name must be base64-encoded. This is not
true when passed as a charm option.

e.g. To archive files in both /var/log/logdir1 and /var/log/logdir2:
logdirs="L3Zhci9sb2cvbG9nZGlyMQo= L3Zhci9sb2cvbG9nZGlyMgo="

Logdirs can be either a directory name, where everything in it will be archived
or a glob pattern, where just matches will be archived. If you want just one
file archived, use a glob pattern that only matches that file
(e.g. /var/log/sys[l]og)

Note: Using "delete_after_archive" on a log file still being written to may
lead to filesystem weirdness (why would you archive an active log file anyway?)
You have been warned.

Provider Configuration

The container_credentials variable should be a base64-encoded JSON string with the
authorisation details needed to access the container. For Swift it should look
like this:

"os_username": "myuser",
"os_tenant_name": "myuser_project",
"os_password": "secret",
"os_auth_url": "http://mykeystoneurl/",
"os_region_name": "openstack-region-name"

For S3:

"AWS_ACCESS_KEY_ID": "my-amazon-id",
"AWS_REGION": "aws-region-name"

For GCE use the standard JSON file Google provides:

"private_key_id": "mykey",
"private_key": "my-cge-key",
"client_email": "email-address",
"client_id": "gce-client-id",
"type": "service_account"

For Azure:

"private_key": "my-private-key"


It is also possible to add pre-processors that are run against the log files
before they are updated to swift. The available pre-processors include gzip,
gunzip and a web log anonymizer. To specify pre-processors just specify a space
separated list in the pre_processors variable and they will run in order.

preprocessors="gunzip anonymize_web_log_ips gzip"

Some preprocessors take additional options for example:

anonymize_web_log_options="--skip-private -s"


(string) The user managing the archiving process
(boolean) If true, logs will be deleted from the local host after successful archiving
(string) When to run the log-archiver (crontab(5) format timespec). Use <<rnd>> for a random value
<<rnd>> <<rnd>> * * *
(string) MANDATORY: Apt sources.list line for a repository containing necessary packages
(string) Space separated list of directories or glob patterns to archive (in addition to any specified by relations)
(string) Default archive_after value in case it isn't set in any relation data
(string) The project name (used by Azure and GCE)
(string) If set, this is the Nagios servicegroup for alerts. If unset, an appropriate one will be chosen
(string) GPG key for apt_repository
(string) Install location for any related scripts
(string) MANDATORY: Which container format to use ('azure', 'gce', 's3' or 'swift')
(string) Used to determine how the server is identified in nagios. Choices are 'unit' (ubuntu-mirror-1 format) or 'host' ( format)
(string) MANDATORY: The name of the container/bucket to archive files into Note: For some providers, this has to be a globally unique name
(string) Operating name of the charm
(string) A space separated list of preprocessors to run, ie gunzip, gzip or anonymize_web_log_ips. They will be run in order.
(string) Options to pass to the anonymize web logs script if it is running as a preprocessor. In addition to anonymizing web logs it can skip ranges or private addresses if specified.
(string) Used for nagios monitoring. See nrpe-external-master charm for details
(string) MANDATORY: Login credentials needed for container access (base64 encoded)