Description
Hadoop is a software platform that lets one easily write and run applications that process vast amounts of data.
This charm provides a simplified connection point for client / workload services which require access to Apache Hadoop. This connection is established via the Apache Bigtop gateway.
Overview
The Apache Hadoop software library is a framework that allows for the
distributed processing of large data sets across clusters of computers
using a simple programming model.
This charm facilitates communication between Hadoop components of an
Apache Bigtop cluster and workload applications.
Deploying
A working Juju installation is assumed to be present. If Juju is not yet set
up, please follow the getting-started instructions prior to deploying this
charm.
This charm is intended to be deployed via one of the apache bigtop bundles.
For example:
juju deploy hadoop-processing
Note: The above assumes Juju 2.0 or greater. If using an earlier version
of Juju, use juju-quickstart with the following syntax:juju quickstart hadoop-processing
.
This will deploy an Apache Bigtop cluster with a client unit preconfigured to
work with the cluster. More information about this deployment can be found in the
bundle readme.
Network-Restricted Environments
Charms can be deployed in environments with limited network access. To deploy
in this environment, configure a Juju model with appropriate proxy and/or
mirror options. See Configuring Models for more information.
Verifying
Status
Apache Bigtop charms provide extended status reporting to indicate when they
are ready:
juju status
This is particularly useful when combined with watch
to track the on-going
progress of the deployment:
watch -n 2 juju status
The message column will provide information about a given unit's state.
This charm is ready for use once the status message indicates that it is
ready with hdfs and/or yarn.
Smoke Test
This charm provides a smoke-test
action that can be used to verify the
application is functioning as expected. Run the action as follows:
juju run-action plugin/0 smoke-test
Note: The above assumes Juju 2.0 or greater. If using an earlier version
of Juju, the syntax isjuju action do plugin/0 smoke-test
.
Watch the progress of the smoke test actions with:
watch -n 2 juju show-action-status
Note: The above assumes Juju 2.0 or greater. If using an earlier version
of Juju, the syntax isjuju action status
.
Eventually, the action should settle to status: completed
. If it
reports status: failed
, the application is not working as expected. Get
more information about a specific smoke test with:
juju show-action-output <action-id>
Note: The above assumes Juju 2.0 or greater. If using an earlier version
of Juju, the syntax isjuju action fetch <action-id>
.
Utilities
This charm includes Hadoop command line utilities that can be used
to verify information about the cluster.
Show the dfsadmin report on the command line with the following:
juju run --application plugin "su hdfs -c 'hdfs dfsadmin -report'"
Contact Information
Resources
