Bigdata Dev Spark

Learn about configurations >

• driver_memory | string

  Default: 1g

  Specify gigabytes (e.g. 1g) or megabytes (e.g. 1024m). If running in 'local' or 'standalone' mode, you may also specify a percentage of total system memory (e.g. 50%).

• executor_memory | string

  Default: 1g

  Specify gigabytes (e.g. 1g) or megabytes (e.g. 1024m). If running in 'local' or 'standalone' mode, you may also specify a percentage of total system memory (e.g. 50%).

• spark_bench_enabled | boolean

  Default: True

  When set to 'true' (the default), this charm will download and install the SparkBench benchmark suite from the configured URLs. When set to 'false', SparkBench will be removed from the unit, though any data stored in hdfs:///user/ubuntu/spark-bench will be preserved.

• spark_bench_ppc64le | string

  Default: https://s3.amazonaws.com/jujubigdata/ibm/noarch/SparkBench-2.0-20170403.tgz#sha256=709caec6667dd82e42de25eb8bcd5763ca894e99e5c83c97bdfcf62cb1aa00c8

  URL (including hash) of a ppc64le tarball of SparkBench. By default, this points to a pre-built SparkBench binary based on sources in the upstream repository. This option is only valid when 'spark_bench_enabled' is 'true'.

• spark_bench_x86_64 | string

  Default: https://s3.amazonaws.com/jujubigdata/ibm/noarch/SparkBench-2.0-20170403.tgz#sha256=709caec6667dd82e42de25eb8bcd5763ca894e99e5c83c97bdfcf62cb1aa00c8

  URL (including hash) of an x86_64 tarball of SparkBench. By default, this points to a pre-built SparkBench binary based on sources in the upstream repository. This option is only valid when 'spark_bench_enabled' is 'true'.

• spark_execution_mode | string

  Default: standalone

  Options are "local", "standalone", "yarn-client", and "yarn-cluster". Consult the readme for details on these options.