A single Kafka broker can handle hundreds of megabytes of reads and writes per
second from thousands of clients.
Kafka is designed to allow a single cluster to serve as the central data
backbone for a large organization. It can be elastically and transparently
expanded without downtime. Data streams are partitioned and spread over a
cluster of machines to allow data streams larger than the capability of any
single machine and to allow clusters of co-ordinated consumers
Messages are persisted on disk and replicated within the cluster to prevent
data loss. Each broker can handle terabytes of messages without performance
Distributed by Design
Kafka has a modern cluster-centric design that offers strong durability and
Scalable distributed publish/subscribe system built ontop of a distributed log.
$ juju deploy zookeeper
$ juju deploy kafka
$ juju add-relation kafka zookeeper
First we need to get a hold of the zookeeper servers that are kafka brokers
are connected to:
$ juju run --unit=kafka/0 ./hooks/utils.py
We can list all the cli commands available to us via
$ juju run --unit=kafka/0 "ls bin"
We can create a topic with
$ juju run --unit=kafka/0 "./bin/kafka-topics.sh --zookeeper 10.0.3.221:2181 --create --topic my_topic_name --partitions 5 --replication-factor 1"
We can list topics with
$ juju run --unit=kafka/0 "./bin/kafka-topics.sh --zookeeper 10.0.3.221:2181 --list