Search Icon

Ryan Harrison My blog, portfolio and technology related ramblings

Kafka Command Cheat Sheet

Environment Variables

Set path to binaries

export JAVA_HOME=/path/to/jdk

export PATH=$PATH:/path/to/kafka/common/bin

Add Kerberos / SASL config options (if required)

export KAFKA_OPTS=" -Dzookeeper.sasl.client.username=myuser"

export KCONFIG=/var/tmp/

Where properties contains:


Set common broker lists

export KBROKER=host:9092

export ZBROKER=host:2181


Create a new topic --zookeeper $ZBROKER --create --replication-factor 3 --partitions 4 --topic topic1

Describe an existing topic --zookeeper $ZBROKER --describe --topic topic1

List all topics in the cluster --zookeeper $ZBROKER --list

Delete a topic --zookeeper $ZBROKER --delete --topic topic1.*

Alter topic configuration --zookeeper $ZBROKER --alter --entity-type topics --entity-name topic1 --add-config


Run a console consumer starting from the beginning of a topic --bootstrap-server $KBROKER --topic topic1 --from-beginning --consumer-config $KCONFIG

Consumer which prints key and value for each message --bootstrap-server $KBROKER --topic topic1 --from-beginning --property print.key=true --property key.separator="|" --consumer-config $KCONFIG

Create a consumer inside a specific group --bootstrap-server $KBROKER --topic topic1 --group group1 --consumer-config $KCONFIG


Run a console producer pushing to a specific topic --broker-list $KBROKER --topic topic1 --producer-config $KCONFIG

Include a specific key in each published message --broker-list $KBROKER --topic topic1 --property "parse.key=true" --property "key.separator=|" --producer-config $KCONFIG

Consumer Groups

List all consumer groups in the cluster --bootstrap-server $KBROKER --list --command-config $KCONFIG

Describe offsets and status of a consumer group --bootstrap-server $KBROKER --group group1 --describe --command-config $KCONFIG

Delete a consumer group --bootstrap-server $KBROKER --group group1 --delete --command-config $KCONFIG

Reset offsets to the beginning for all topics --bootstrap-server $KBROKER --group group1 --reset-offsets --to-earliest --all-topics --execute --command-config $KCONFIG

Reset offsets to beginning for a specific topic --bootstrap-server $KBROKER --group group1 --reset-offsets --to-earliest --topic topic1 --execute --command-config $KCONFIG

Reset offsets to latest --bootstrap-server $KBROKER --group group1 --reset-offsets --to-latest --all-topics --execute --command-config $KCONFIG

Reset offsets to specific value --bootstrap-server $KBROKER --group group1 --reset-offsets --to-offset 10 --topic topic1:10 --execute --command-config $KCONFIG

Reset offsets by a specific shift amount --bootstrap-server $KBROKER --group group1 --reset-offsets --shift-by -10 --topic topic1 --execute --command-config $KCONFIG


Grant superuser access to all topics and groups --authorizer-properties zookeeper.connect=$ZBROKER --add --allow-principal User:myuser --operation ALL --topic '*' --group '*' --cluster


Connect to the Zookeeper cluster $ZHOST:$ZPORT

Set Zookeeper ACL for superuser

ls /
setAcl / sasl:myuser:cdrqa,world:anyone:r
getAcl /