Ubuntu logo

Charm Testing

Intro

DRAFT

Juju has been designed from the start to foster a large collection of "charms". Charms are expected to number in the thousands, and be self contained, with well defined interfaces for defining their relationships to one another.

Because this is a large complex system, not unlike a Linux software distribution, there is a need to test the charms and how they interact with one another. This specification defines a plan for implementing a simple framework to help this happen.

Static tests have already been implemented in the charm proof command as part of charm-tools. Any static testing of charms is beyond the scope of this specification.

Phase 1 - Generic tests

All charms share some of the same characteristics. They all have a yaml file called metadata.yaml, and when deployed, juju will always attempt to progress the state of the service from install to config to started. Because of this, all charms can be tested using the following algorithm:

deploy charm
while state != started
  if timeout is reached, FAIL
  if state == install_error, config_error, or start_error, FAIL
  if state == started, PASS

Other generic tests may be identified, so a collection of generic tests should be the focus of an implementation.

Note that this requirement is already satisfied by Mark Mims' jenkins tester: http://charmtests.markmims.com/

Phase 2 - Charm Specific tests

Charm authors will have the best insight into whether or not a charm is working properly.

A simple structure will be utilized to attach tests to charms. Under the charm root directory, a sub-directory named 'tests' will be scanned by a test runner for executable files matching the glob *.test. These will be run in lexical order by the test runner, with a predictible environment. The tests can make the following assumptions:

  • A minimal install of the release of Ubuntu which the charm is targetted at will be available.
  • A version of juju is installed and available in the system path.
  • A juju environment with no services deployed inside it is already bootstrapped, and will be the default for command line usage.
  • The CWD is the tests directory off the charm root.
  • Full network access to deployed nodes will be allowed.
  • the bare name of any charm in arguments to juju will be resolved to a charm url and/or repository arguments of the test runner's choice. This means that if you need mysql, you do not do juju deploy cs:mysql or juju deploy --repository ~/charms local:mysql, but just juju deploy mysql. A wrapper will resolve this to the latest version of the given charm from the list of official charms.

The following restrictions may be enforced:

  • Internet access will be restricted from the testing host.

If present, tests/tests.yaml will be read to determine packages that need to be installed on the host running tests in order to facilitate the tests. The packages can only be installed from the official, default Ubuntu archive for the release which the charm is intended for, from any of the repositories enabled by default in said release. The format of tests.yaml is as such:

packages: [ package1, package2, package3 ]

If a tool is needed to perform a test and is not available in the Ubuntu archive, it can also be included in the tests/ directory, as long as the file which contains it does not end in .test. Note that build tools cannot be assumed to be available on the testing system.

Purpose of tests

The purpose of these tests is to assert that the charm works well on the intended platform and performs the expected configuration steps. Examples of things to test in each charm beyond install/start is:

  • After install, expose, and adding of required relations, the service is listening on the intended ports and is functional.
  • Adding, removing, and re-adding a relation should work without error.
  • Setting config values should result in the config value reflected in the service's configuraion.
  • Adding multiple units to a web app charm and relating to a load balancer results in the same HTML on both units directly and the load balancer.

Exit Codes

Upon exit, the test's exit code will be evaluated to mean the following:

  • 0: Test passed
  • 1: Failed test
  • 100: Test is skipped because of incomplete environment

Output

There is a general convention which output should follow, though it will not be interpreted by machine. On stdout, a message indicating the reason for the exit code should be printed, with a prefix string corresponding to the exit codes defined above. The correlation is:

  • PASS - 0
  • FAIL - 1
  • SKIP - 100

Anything else intentional should be prefixed with the word 'INFO'. If the contents of files are to be logged, the contents should be preceeded by INFO: BEGIN filename, where filename is a logical name unique to this run of the test, and then the file ended with INFO: END filename.

Example Tests

Deploy requirements and Poll

The test below [*] deploys mediawiki with mysql and memcached related to it, and then tests to make sure it returns a page via http with "<title>" somewhere in the content.:

#!/bin/sh

set -e

teardown() {
    if [ -n "$datadir" ] ; then
        if [ -f $datadir/passed ]; then
            rm -r $datadir
        else
            echo INFO: $datadir preserved
            if [ -f $datadir/wget.log ] ; then
                echo INFO: BEGIN wget.log
                cat $datadir/wget.log
                echo INFO: END wget.log
            fi
        fi
    fi
}
trap teardown EXIT


juju deploy mediawiki
juju deploy mysql
juju deploy memcached
juju add-relation mediawiki:db mysql:db
juju add-relation memcached mediawiki
juju expose mediawiki

for try in `seq 1 600` ; do
    host=`juju status | tests/get-unit-info mediawiki public-address`
    if [ -z "$host" ] ; then
        sleep 1
    else
        break
    fi
done

if [ -z "$host" ] ; then
    echo FAIL: status timed out
    exit 1
fi

datadir=`mktemp -d ${TMPDIR:-/tmp}/wget.test.XXXXXXX`
echo INFO: datadir=$datadir

wget --tries=100 --timeout=6 http://$host/ -O - -a $datadir/wget.log | grep -q '<title>'

if [ $try -eq 600 ] ; then
    echo FAIL: Timed out waiting.
    exit 1
fi

touch $datadir/passed

trap - EXIT
teardown

echo PASS
exit 0

Test config settings

The following example tests checks to see if the default_port change the admin asks for is actually respected post-deploy:

#!/bin/sh

if [ -z "`which nc`" ] ; then
    echo "SKIP: cannot run tests without netcat"
    exit 100
fi

set -e

juju deploy mongodb
juju expose mongodb

for try in `seq 1 600` ; do
    host=`juju status | tests/get-unit-info mongodb public-address`
    if [ -z "$host" ] ; then
        sleep 1
    else
        break
    fi
done

if [ -z "$host" ] ; then
    echo FAIL: status timed out
    exit 1
fi

assert_is_listening() {
    local port=$1
    listening=""
    for try in `seq 1 10` ; do
        if ! nc $host $port < /dev/null ; then
            continue
        fi
        listening="$port"
        break
    done

    if [ -z "$listening" ] ; then
       echo "FAIL: not listening on port $port after 10 retries"
       return 1
    else
       echo "PASS: listening on port $listening"
       return 0
    fi
}

assert_is_listening 27017

juju set mongodb default_port=55555

assert_is_listening 55555
echo PASS: config change tests passed.
exit 0
[*]get-unit-info The example tests script uses a tool that is not widely available yet, get-unit-info. In the future enhancements should be made to juju core to allow such things to be made into plugins. Until then, it can be included in each test dir that uses it, or we can build a package of tools that are common to tests.

Test Runner

A test runner will periodically poll the collection of charms for changes since the last test run. If there have been changes, the entire set of changes will be tested as one delta. This delta will be recorded in the test results in such a way where a developer can repeat the exact set of changes for debugging purposes.

All of the charms will be scanned for tests in lexical order by series, charm name, branch name. Non official charms which have not been reviewed by charmers will not have their tests run until the test runner's restrictions have been vetted for security, since we will be running potentially malicious code. It is left to the implementor to determine what mix of juju, client platform, and environment settings are appropriate, as all of these are variables that will affect the running charms, and so may affect the outcome.

If tests exit with services still in the environment, the test runner may clean them up, whether by destroying the environment or destroying the services explicitly, and the machines may be terminated as well. Any artifacts needed from the test machines should be retrieved and displayed before the test exits.