Kafka on AWS EC2 w/ SSL and External Visibility

I’m truly shocked by how difficult this information is to gather up in 1 place. Maybe because AWS has their own version of Kafka functionality.

At any rate, after much reading and irritation I have it working. There is still some work to do securing Zookeeper and adding ACL’s to Kafka, but we’ll get there later.

Tip 1: Use an Elastic IP per kafka broker and give it a DNS entry. We’re using Route53 so that’s pretty easy.
Tip 2: Put a complete list of your Kafka brokers and their INTERNAL IP addresses in /etc/hosts on each broker to match their DNS Hostname
Tip 3: Edit the network settings on your brokers so the hostname is accurate w/ the DNS Entry
Tip 4: Bounce box after this to ensure it works properly
Tip 5: Do NOT use underscores in your hostname.

All of this is to ensure Zookeeper can figure out WTF is going on. It won’t let you tell it directly… 🙁

We have a 3 node cluster running across 3 AZ’s in our VPC. Single node kafka is a lot easier. The following settings for server.properties are what makes this work for us:

Make sure broker.id is set to a unique # for each broker
auto.leader.rebalance.enable=true — enables better shutdown/restart experiences
controlled.shutdown.enable=true — helps with this as well
listeners=SSL://__HOST__:9093 — This is for your specific broker. We disabled PLAINTEXT as an option.
advertised.listeners=SSL://__HOST__:9093 — One of these is deprecated.

We wanted to keep everything forever, so you have to set a really high value like log.retention.hours=2147483647 to make that work. For the SSL stuff:



There is a whole lot to learn about generating sign CA Keys and keystores for Java. I don’t have the energy, so I’ll give you the link:

Kafka has “Rack Awareness” built in, so this is kinda cool. I use a bash script to start my brokers, which allows me to use a template and fill in the rack location using stuff that EC2 instances know about themselves.

In the template set broker.rack=’__AZ__’
and in your bash script do something like this:
AZ=$(curl -s
cat /root/kafka_current/config/server.properties.tmp | sed -e “s/__ID__/${ID}/g;s/__HOST__/${HOST}/g;s/__AZ__/${AZ}/g” > /root/kafka_current/config/server.properties

I embed the Broker ID in the hostname so I can do stuff like:
ID=$(hostname | cut -f 1 -d. | cut -f 2 -d “b”)

So my broker names are kb1.mydomain.com, kb2.mydomain.com, etc.

Most of the Kafka Monitoring and Management tools completely fail w/ SSL. 🙁

Grease Monkey ~~ GM

About Grease Monkey

Computer nerd since the 80's. Data nerd since the 90's. Generic nerd for a lifetime.
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *