container log archive #4

Description

We want to keep archive logs for a while, but they're big.
Solution: store them in containers (it works for shipping companies!)


This charm provides a way to archive log files from a given directory to
container-based storage (currently supports Amazon S3, OpenStack Swift and
Google Compute Engine (GCE)).

Common Configuration

It is a subordinate charm, requiring either a "log-archive-relation-changed"
hook in the parent to provide the directory to be archived and when to archive
it or have those same values passed as charm options. Note: If both methods
are used, all the directories will be archived, but the timing from the relation
is preferred over a charm option.

e.g. relation-set logdirs="base64_dir1 base64_dir2"
relation-set archive_after=1

Any files older than "archive_after" days in the specified directories will be
archived to container storage. Deletion of local log files is assumed to be
handled by the parent charm unless the "delete_after_archive" config variable is
set to true (in which case the charm will remove the log file immediately
after it is successfully archived)

The logdir relation variable is a space-separated list of names. To avoid any
issues with unusual filenames, each name must be base64-encoded. This is not
true when passed as a charm option.

e.g. To archive files in both /var/log/logdir1 and /var/log/logdir2:
logdirs="L3Zhci9sb2cvbG9nZGlyMQo= L3Zhci9sb2cvbG9nZGlyMgo="

Logdirs can be either a directory name, where everything in it will be archived
or a glob pattern, where just matches will be archived. If you want just one
file archived, use a glob pattern that only matches that file
(e.g. /var/log/sys[l]og)

Note: Using "delete_after_archive" on a log file still being written to may
lead to filesystem weirdness (why would you archive an active log file anyway?)
You have been warned.

Provider Configuration

The container_credentials variable should be a base64-encoded JSON string with the
authorisation details needed to access the container. For Swift it should look
like this:

{
"os_username": "myuser",
"os_tenant_name": "myuser_project",
"os_password": "secret",
"os_auth_url": "http://mykeystoneurl/",
"os_region_name": "openstack-region-name"
}

For S3:

{
"AWS_ACCESS_KEY_ID": "my-amazon-id",
"AWS_SECRET_ACCESS_KEY": "secret",
"AWS_REGION": "aws-region-name"
}

For GCE use the standard JSON file Google provides:

{
"private_key_id": "mykey",
"private_key": "my-cge-key",
"client_email": "email-address",
"client_id": "gce-client-id",
"type": "service_account"
}

For Azure:

{
"private_key": "my-private-key"
}

Pre-processors

It is also possible to add pre-processors that are run against the log files
before they are updated to swift. The available pre-processors include gzip,
gunzip and a web log anonymizer. To specify pre-processors just specify a space
separated list in the pre_processors variable and they will run in order.

preprocessors="gunzip anonymize_web_log_ips gzip"

Some preprocessors take additional options for example:

anonymize_web_log_options="--skip-private -s 254.254.254.0/24"

Configuration

username
(string) The user managing the archiving process
log-archive
delete_after_archive
(boolean) If true, logs will be deleted from the local host after successful archiving
cron_time
(string) When to run the log-archiver (crontab(5) format timespec). Use <<rnd>> for a random value
<<rnd>> <<rnd>> * * *
apt_repository
(string) MANDATORY: Apt sources.list line for a repository containing necessary packages
logdirs
(string) Space separated list of directories or glob patterns to archive (in addition to any specified by relations)
archive_after_default
(string) Default archive_after value in case it isn't set in any relation data
7
project
(string) The project name (used by Azure and GCE)
nagios_servicegroup
(string) If set, this is the Nagios servicegroup for alerts. If unset, an appropriate one will be chosen
apt_repository_key
(string) GPG key for apt_repository
script_dir
(string) Install location for any related scripts
/srv/container-log-archive/bin
container_type
(string) MANDATORY: Which container format to use ('azure', 'gce', 's3' or 'swift')
nagios_hostname_type
(string) Used to determine how the server is identified in nagios. Choices are 'unit' (ubuntu-mirror-1 format) or 'host' (myhost.mydomain.com format)
unit
container_name
(string) MANDATORY: The name of the container/bucket to archive files into Note: For some providers, this has to be a globally unique name
application_name
(string) Operating name of the charm
container-log-archive
preprocessors
(string) A space separated list of preprocessors to run, ie gunzip, gzip or anonymize_web_log_ips. They will be run in order.
anonymize_web_log_options
(string) Options to pass to the anonymize web logs script if it is running as a preprocessor. In addition to anonymizing web logs it can skip ranges or private addresses if specified.
nagios_host_context
(string) Used for nagios monitoring. See nrpe-external-master charm for details
juju
container_credentials
(string) MANDATORY: Login credentials needed for container access (base64 encoded)