kafka

  • By vpalos
  • Latest version (#0)
  • trusty
  • Stable
  • Edge

Description

Fast
A single Kafka broker can handle hundreds of megabytes of reads and writes per
second from thousands of clients.

Currently deploys Apache Kafka 2.9.2-0.8.2.1

Scalable
Kafka is designed to allow a single cluster to serve as the central data
backbone for a large organization. It can be elastically and transparently
expanded without downtime. Data streams are partitioned and spread over a
cluster of machines to allow data streams larger than the capability of any
single machine and to allow clusters of co-ordinated consumers

Durable
Messages are persisted on disk and replicated within the cluster to prevent
data loss. Each broker can handle terabytes of messages without performance
impact.

Distributed by Design
Kafka has a modern cluster-centric design that offers strong durability and
fault-tolerance guarantees.


Kafka

Scalable distributed publish/subscribe system built ontop of a distributed log.
This charm deploys Apache Kafka 2.9.2-0.8.2.1.

Install

$ juju deploy zookeeper
$ juju deploy kafka
$ juju add-relation kafka zookeeper

Usage

First we need to get a hold of the zookeeper servers that are kafka brokers
are connected to:

$ juju run --unit=kafka/0 ./hooks/utils.py
10.0.3.221:2181

We can list all the cli commands available to us via

$ juju run --unit=kafka/0 "ls bin"

kafka-console-consumer.sh
kafka-console-producer.sh
kafka-consumer-perf-test.sh
kafka-preferred-replica-election.sh
kafka-producer-perf-test.sh
kafka-reassign-partitions.sh
kafka-replay-log-producer.sh
kafka-simple-consumer-perf-test.sh
kafka-simple-consumer-shell.sh
kafka-topics.sh

We can create a topic with

$ juju run --unit=kafka/0 "./bin/kafka-topics.sh --zookeeper 10.0.3.221:2181 --create --topic my_topic_name --partitions 5 --replication-factor 1"

We can list topics with

$ juju run --unit=kafka/0 "./bin/kafka-topics.sh --zookeeper 10.0.3.221:2181 --list