container log archive

Description

We want to keep archive logs for a while, but they're big.
Solution: store them in containers (it works for shipping companies!)


This charm provides a way to archive log files from a given directory to
container-based storage (currently supports Amazon S3, OpenStack Swift and
Google Compute Engine (GCE)).

Common Configuration

It is a subordinate charm, requiring either a "log-archive-relation-changed"
hook in the parent to provide the directory to be archived and when to archive
it or have those same values passed as charm options. Note: If both methods
are used, all the directories will be archived, but the timing from the relation
is preferred over a charm option.

e.g. relation-set logdirs="base64_dir1 base64_dir2"
relation-set archive_after=1

Any files older than "archive_after" days in the specified directories will be
archived to container storage. Deletion of local log files is assumed to be
handled by the parent charm unless the "delete_after_archive" config variable is
set to true (in which case the charm will remove the log file immediately
after it is successfully archived)

The logdir relation variable is a space-separated list of names. To avoid any
issues with unusual filenames, each name must be base64-encoded. This is not
true when passed as a charm option.

e.g. To archive files in both /var/log/logdir1 and /var/log/logdir2:
logdirs="L3Zhci9sb2cvbG9nZGlyMQo= L3Zhci9sb2cvbG9nZGlyMgo="

Logdirs can be either a directory name, where everything in it will be archived
or a glob pattern, where just matches will be archived. If you want just one
file archived, use a glob pattern that only matches that file
(e.g. /var/log/sys[l]og)

Note: Using "delete_after_archive" on a log file still being written to may
lead to filesystem weirdness (why would you archive an active log file anyway?)
Also using this option when multiple processes are parsing archived logs can lead
to unpredictable behavior. You have been warned.

Provider Configuration

The container_credentials variable should be a base64-encoded JSON string with the
authorisation details needed to access the container. For Swift it should look
like this:

{
"os_username": "myuser",
"os_tenant_name": "myuser_project",
"os_password": "secret",
"os_auth_url": "http://mykeystoneurl/",
"os_region_name": "openstack-region-name"
}

For S3:

{
"AWS_ACCESS_KEY_ID": "my-amazon-id",
"AWS_SECRET_ACCESS_KEY": "secret",
"AWS_REGION": "aws-region-name"
}

For GCE use the standard JSON file Google provides:

{
"private_key_id": "mykey",
"private_key": "my-cge-key",
"client_email": "email-address",
"client_id": "gce-client-id",
"type": "service_account"
}

For Azure:

{
"private_key": "my-private-key"
}

Pre-processors

It is also possible to add pre-processors that are run against the log files
before they are updated to swift. The available pre-processors include gzip,
gunzip and a web log anonymizer. To specify pre-processors just specify a space
separated list in the pre_processors variable and they will run in order.

preprocessors="gunzip anonymize_web_log_ips gzip"

Some preprocessors take additional options for example:

anonymize_web_log_options="--skip-private -s 254.254.254.0/24"

If you would like to send raw logs to one location and pre-processed logs to
another the charm can be installed multiple times with different Juju
application names. In this case be careful with the delete_after_archive
option, it is best avoided.

Configuration

username
(string) The user managing the archiving process
log-archive
delete_after_archive
(boolean) If true, logs will be deleted from the local host after successful archiving
cron_time
(string) When to run the log-archiver (crontab(5) format timespec). Use <<rnd>> for a random value
<<rnd>> <<rnd>> * * *
apt_repository
(string) Apt sources.list line for a repository containing necessary packages
logdirs
(string) Space separated list of directories or glob patterns to archive (in addition to any specified by relations)
archive_after_default
(string) Default archive_after value in case it isn't set in any relation data
7
project
(string) The project name (used by Azure and GCE)
nagios_servicegroup
(string) If set, this is the Nagios servicegroup for alerts. If unset, an appropriate one will be chosen
apt_repository_key
(string) GPG key for apt_repository
container_type
(string) MANDATORY: Which container format to use ('azure', 'gce', 's3' or 'swift')
nagios_hostname_type
(string) Used to determine how the server is identified in nagios. Choices are 'unit' (ubuntu-mirror-1 format) or 'host' (myhost.mydomain.com format)
unit
container_name
(string) MANDATORY: The name of the container/bucket to archive files into Note: For some providers, this has to be a globally unique name
preprocessors
(string) A space separated list of preprocessors to run, ie gunzip, gzip or anonymize_web_log_ips. They will be run in order.
anonymize_web_log_options
(string) Options to pass to the anonymize web logs script if it is running as a preprocessor. In addition to anonymizing web logs it can skip ranges or private addresses if specified.
nagios_host_context
(string) Used for nagios monitoring. See nrpe-external-master charm for details
juju
container_credentials
(string) MANDATORY: Login credentials needed for container access (base64 encoded)