storm #5

Description

Installs the Twitter Storm Master/ Worker Server. The Master server runs Nimbus and the UI. The Worker server runs Supervisor, Workers and DRPC.


Overview

This charm provides access to Storm, the distributed realtime computation system. The Charm allows you
to setup a master and one or more workers. The supervisor contains Nimbus and the UI.
The workers containi a supervisor, a DRPC server and one or more workers when a topology is deployed.
A relation with zookeeper is required.

Usage

To setup a test environment::

juju bootstrap
juju deploy zookeeper
juju deploy storm stormmaster
juju deploy storm stormworker
juju add-relation stormmaster zookeeper
juju add-relation stormworker zookeeper
juju add-relation stormmaster:master stormworker:worker
juju expose stormmaster
juju expose stormworker

The UI is running on port 8080 of the stormmaster.

Optionally you can use a configuration file, e.g. config.saml.

An example could be:
stormworker:
nimbusmemory: 512
uimemory: 384
supervisormemory: 128
workermemory: 128
nimbusport: 6627
uiport: 8080
zookeeperport: 2181
drpcport: 3772
numberofworkers: 5
startingworkerport: 6700

The memory requirements of several components can be specified in
megabytes: nimbus, ui, supervisor and worker.

Also the default ports can be changed for nimbus, ui, zookeeper and drpc.
These processes run as a non-priviledged user hence the port needs to be
higher than 1024.

Finally each worker node can have multiple workers running on it.
The startingworkerport defines the port of the first worker, e.g. 6700.
The numberofworkers defines how many workers are started up on each worker
node. The ports will be a combination of these two configuration options.
If the starting port is 6700 and the number of workers is 3, then the ports
that are used will be: 6700 6701 6702.

You can use add-unit to add additional workers. However for the moment the
master can not be scaled to more than one. In order for the master to be scaled,
a distributed file system would have to synchronize /mnt/storm between different
peers and if the master died, a zookeeper master selection process would have to
select a new master and all workers would have to change their nimbus server
reference to point to the new master and restart.

Configuration

workermemory
(int) The maximum heap size in MB to allocate for Workers spun by the Supervisor daemons processes within the service units managed by this charm.
768
zookeeperport
(int) The port zookeeper is listening on.
2181
supervisormemory
(int) The maximum heap size in MB to allocate for Supervisor daemons processes within the service units managed by this charm.
256
drpcport
(int) The port DRPC is listening on. Must be higher than 1024.
3772
nimbusport
(int) The port nimbus is listening on. Must be higher than 1024.
6627
startingworkerport
(int) The port a worker is listening on. Must be higher than 1024. Each worker on a supervisor takes a separate port so if there are 4 workers then the ports 6700, 6701, 6702, 6703 are used.
6700
numberofworkers
(int) The number of workers that a supervisor starts.
4
uimemory
(int) The maximum heap size in MB to allocate for UI daemons processes within the service units managed by this charm.
768
uiport
(int) The port storm ui is listening on. Must be higher than 1024.
8080
nimbusmemory
(int) The maximum heap size in MB to allocate for Nimbus daemons processes within the service units managed by this charm.
1024