hadoop spark #42

  • 5 machines, 9 units


This is a five unit big data cluster that includes Hadoop 2.7.3 and Spark 2.1 from Apache Bigtop. Use it to analyse batch data with MapReduce or streaming data with Spark. It will run on 5 machines in your cloud.


The Apache Hadoop software library is a framework that allows for the
distributed processing of large data sets across clusters of computers
using a simple programming model.

Hadoop is designed to scale from a few servers to thousands of machines,
each offering local computation and storage. Rather than rely on hardware
to deliver high-availability, Hadoop can detect and handle failures at the
application layer. This provides a highly-available service on top of a
cluster of machines, each of which may be prone to failure.

Apache Spark is a fast and general engine for large-scale data processing.
Learn more at spark.apache.org.

This bundle provides a complete deployment of Hadoop and Spark components
from Apache Bigtop that performs distributed data processing at scale.
Ganglia and rsyslog applications are also provided to monitor cluster health
and syslog activity.

Bundle Composition

The applications that comprise this bundle are spread across 5 units as

  • NameNode v2.7.3
  • ResourceManager v2.7.3
    • Colocated on the NameNode unit
  • Slave (DataNode and NodeManager) v2.7.3
    • 3 separate units
  • Spark (Driver in yarn-client mode) v2.1.0
  • Client (Hadoop endpoint)
    • Colocated on the Spark unit
  • Plugin (Facilitates communication with the Hadoop cluster)
    • Colocated on the Spark/Client unit
  • Ganglia (Web interface for monitoring cluster metrics)
    • Colocated on the Spark/Client unit
  • Rsyslog (Aggregate cluster syslog events in a single location)
    • Colocated on the Spark/Client unit

Deploying this bundle results in a fully configured Apache Bigtop
cluster on any supported cloud, which can be scaled to meet workload


This charm requires Juju 2.0 or greater. If Juju is not yet set up, please
follow the getting-started instructions prior to deploying this bundle.

Note: This bundle requires hardware resources that may exceed limits
of Free-tier or Trial accounts on some clouds. To deploy to these
environments, modify a local copy of bundle.yaml to set
services: 'X': num_units: 1 and machines: 'X': constraints: mem=3G as
needed to satisfy account limits.

Deploy this bundle from the Juju charm store with the juju deploy command:

juju deploy hadoop-spark

Alternatively, deploy a locally modified bundle.yaml with:

juju deploy /path/to/bundle.yaml

The charms in this bundle can also be built from their source layers in the
Bigtop charm repository. See the Bigtop charm README for instructions
on building and deploying these charms locally.

Network-Restricted Environments

Charms can be deployed in environments with limited network access. To deploy
in this environment, configure a Juju model with appropriate proxy and/or
mirror options. See Configuring Models for more information.



The applications that make up this bundle provide status messages to indicate
when they are ready:

juju status

This is particularly useful when combined with watch to track the on-going
progress of the deployment:

watch -n 2 juju status

The message for each unit will provide information about that unit's state.
Once they all indicate that they are ready, perform application smoke tests
to verify that the bundle is working as expected.

Smoke Test

The charms for each core component (namenode, resourcemanager, slave, and
spark) provide a smoke-test action that can be used to verify the
application is functioning as expected. Note that the 'slave' component runs
extensive tests provided by Apache Bigtop and may take up to 30 minutes to
complete. Run the smoke-test actions as follows:

juju run-action namenode/0 smoke-test
juju run-action resourcemanager/0 smoke-test
juju run-action slave/0 smoke-test
juju run-action spark/0 smoke-test

Watch the progress of the smoke test actions with:

watch -n 2 juju show-action-status

Eventually, all of the actions should settle to status: completed. If
any report status: failed, that application is not working as expected. Get
more information about a specific smoke test with:

juju show-action-output <action-id>


Applications in this bundle include command line and web utilities that
can be used to verify information about the cluster.

From the command line, show the HDFS dfsadmin report and view the current list
of YARN NodeManager units with the following:

juju run --application namenode "su hdfs -c 'hdfs dfsadmin -report'"
juju run --application resourcemanager "su yarn -c 'yarn node -list'"

To access the HDFS web console, find the PUBLIC-ADDRESS of the namenode
application and expose it:

juju status namenode
juju expose namenode

The web interface will be available at the following URL:


Similarly, to access the Resource Manager web consoles, find the
PUBLIC-ADDRESS of the resourcemanager application and expose it:

juju status resourcemanager
juju expose resourcemanager

The YARN and Job History web interfaces will be available at the following URLs:


Finally, to access the Spark web console, find the PUBLIC-ADDRESS of the
spark application and expose it:

juju status spark
juju expose spark

The web interface will be available at the following URL:



This bundle includes Ganglia for system-level monitoring of the namenode,
resourcemanager, slave, and spark units. Metrics are sent to a
centralized ganglia unit for easy viewing in a browser. To view the ganglia web
interface, find the PUBLIC-ADDRESS of the Ganglia application and expose it:

juju status ganglia
juju expose ganglia

The web interface will be available at:



This bundle includes rsyslog to collect syslog data from the namenode,
resourcemanager, slave, and spark units. These logs are sent to a
centralized rsyslog unit for easy syslog analysis. One method of viewing this
log data is to simply cat syslog from the rsyslog unit:

juju run --unit rsyslog/0 'sudo cat /var/log/syslog'

Logs may also be forwarded to an external rsyslog processing service. See
the Forwarding logs to a system outside of the Juju environment section of
the rsyslog README for more information.


The resourcemanager charm in this bundle provide several benchmarks to gauge
the performance of the Hadoop cluster. Each benchmark is an action that can be
run with juju run-action:

$ juju actions resourcemanager
mrbench     Mapreduce benchmark for small jobs
nnbench     Load test the NameNode hardware and configuration
smoke-test  Run an Apache Bigtop smoke test.
teragen     Generate data with teragen
terasort    Runs teragen to generate sample data, and then runs terasort to sort that data
testdfsio   DFS IO Testing

$ juju run-action resourcemanager/0 nnbench
Action queued with id: 55887b40-116c-4020-8b35-1e28a54cc622

$ juju show-action-output 55887b40-116c-4020-8b35-1e28a54cc622
      direction: asc
      units: secs
      value: "128"
    start: 2016-02-04T14:55:39Z
    stop: 2016-02-04T14:57:47Z
    raw: '{"BAD_ID": "0", "FILE: Number of read operations": "0", "Reduce input groups":
      "8", "Reduce input records": "95", "Map output bytes": "1823", "Map input records":
      "12", "Combine input records": "0", "HDFS: Number of bytes read": "18635", "FILE:
      Number of bytes written": "32999982", "HDFS: Number of write operations": "330",
      "Combine output records": "0", "Total committed heap usage (bytes)": "3144749056",
      "Bytes Written": "164", "WRONG_LENGTH": "0", "Failed Shuffles": "0", "FILE:
      Number of bytes read": "27879457", "WRONG_MAP": "0", "Spilled Records": "190",
      "Merged Map outputs": "72", "HDFS: Number of large read operations": "0", "Reduce
      shuffle bytes": "2445", "FILE: Number of large read operations": "0", "Map output
      materialized bytes": "2445", "IO_ERROR": "0", "CONNECTION": "0", "HDFS: Number
      of read operations": "567", "Map output records": "95", "Reduce output records":
      "8", "WRONG_REDUCE": "0", "HDFS: Number of bytes written": "27412", "GC time
      elapsed (ms)": "603", "Input split bytes": "1610", "Shuffled Maps ": "72", "FILE:
      Number of write operations": "0", "Bytes Read": "1490"}'
status: completed
  completed: 2016-02-04 14:57:48 +0000 UTC
  enqueued: 2016-02-04 14:55:14 +0000 UTC
  started: 2016-02-04 14:55:27 +0000 UTC

The spark charm in this bundle provides benchmarks to gauge the performance
of the Spark/YARN cluster. Each benchmark is an action that can be run with
juju run-action:

$ juju actions spark
pagerank                          Calculate PageRank for a sample data set
sparkpi                           Calculate Pi

$ juju run-action spark/0 pagerank
Action queued with id: 339cec1f-e903-4ee7-85ca-876fb0c3d28e

$ juju show-action-output 339cec1f-e903-4ee7-85ca-876fb0c3d28e
      direction: asc
      units: secs
      value: "83"
    start: 2017-04-12T23:22:38Z
    stop: 2017-04-12T23:24:01Z
  output: '{''status'': ''completed''}'
status: completed
  completed: 2017-04-12 23:24:02 +0000 UTC
  enqueued: 2017-04-12 23:22:36 +0000 UTC
  started: 2017-04-12 23:22:37 +0000 UTC


By default, three Hadoop slave units are deployed. Scaling these is as simple
as adding more units. To add one unit:

juju add-unit slave

Multiple units may be added at once. For example, add four more slave units:

juju add-unit -n4 slave


Apache Bigtop tracks issues using JIRA (Apache account required). File an
issue for this bundle at:


Ensure Bigtop is selected as the project. Typically, bundle issues are filed
in the deployment component with the latest stable release selected as the
affected version. Any uncertain fields may be left blank.

Contact Information