Kafka is a distributed event streaming platform originally developed at LinkedIn and now part of the Apache project. It excels at high-throughput, fault-tolerant streaming of record/event data. Historically Kafka required ZooKeeper to coordinate cluster metadata, but more recent versions support KRaft mode (Kafka Raft) to externalize that dependency and simplify operations.
Running Kafka yourself (versus using managed services) gives you full control over configuration, versioning, networking, and cost—but you also absorb operational burden: upgrades, scaling, monitoring, security. DigitalOcean now even offers a Managed Kafka product to relieve that burden (starting ~$147/month). But if you prefer to self-host (for flexibility, experimentation, or cost control), this guide is for you.
We’ll use Ubuntu Linux on DigitalOcean Droplets as the host OS. If you prefer another distro, adapt package and service commands accordingly.
This guide has these main parts:
- Droplet provisioning (system setup)
- Single-node Kafka in KRaft mode
- Multi-node Kafka cluster via KRaft
- Securing Kafka (network, TLS, authentication)
- Monitoring, maintenance, backups
- Caveats, scaling tips, next steps
Let’s get to work.
Provisioning your servers on DigitalOcean
For a Kafka node, start with at least:
- 4 GB RAM (8 GB recommended for production)
- 2 vCPUs
- SSD disk (50–100 GB or more, depending on log retention)
- Ubuntu 22.04 LTS (or latest stable release)
You can later scale vertically or horizontally.
Create droplets
- In DigitalOcean control panel, go to Droplets → Create Droplet.
- Select Ubuntu 22.04 (or later).
- Choose a plan (e.g. 4 GB / 2 vCPU).
- Pick a datacenter region (ideally same region for nodes to minimize latency).
- Add SSH key (preferred) or password access.
- Optionally enable backups, monitoring.
- Create.
Once droplets are active, note their public/private IPs (you may prefer to use private networking if within the same VPC/region for inter-node traffic).
System setup & prerequisites
On each droplet (ssh in as root or sudo user):
sudo apt update
sudo apt upgrade -y
sudo apt install openjdk-17-headless curl wget vim net-tools -yKafka requires Java (OpenJDK 17 is a reasonable choice). Confirm:
java --versionCreate a dedicated kafka user (non-root) to run services:
sudo adduser --system --no-create-home --group kafkaCreate a directory to host Kafka:
sudo mkdir /opt/kafka
sudo chown kafka:kafka /opt/kafkaYou may also create a log or data directory, e.g.:
sudo mkdir /var/lib/kafka-logs
sudo chown kafka:kafka /var/lib/kafka-logsWe’ll use these in configuration.
Deploy a single-node Kafka broker using KRaft mode
Kafka versions ≥ 3.3 support KRaft (built-in metadata management, no ZooKeeper). This is simpler and future-forward.
On the droplet:
cd /opt/kafka
sudo -u kafka wget https://dlcdn.apache.org/kafka/4.1.0/kafka_2.13-4.1.0.tgz
sudo -u kafka tar -xzf kafka_2.13-4.1.0.tgz --strip-components=1(Adjust version numbers if a newer Kafka is available.)
Configure KRaft mode
Open config/kraft/server.properties as the kafka user (or via sudo):
nano /opt/kafka/config/kraft/server.propertiesModify or add:
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
inter.broker.listener.name=PLAINTEXT
advertised.listeners=PLAINTEXT://<PUBLIC_IP>:9092
# log directory
log.dirs=/var/lib/kafka-logsYou may want to adjust defaults like num.partitions, retention, segment sizes, etc.
Format storage & start
Generate a cluster ID and format the log directory:
cd /opt/kafka
sudo -u kafka bin/kafka-storage.sh random-uuid > cluster-id.txt
CLUSTER_ID=$(cat cluster-id.txt)
sudo -u kafka bin/kafka-storage.sh format -t $CLUSTER_ID -c config/kraft/server.propertiesThen start Kafka:
sudo -u kafka bin/kafka-server-start.sh config/kraft/server.propertiesYou should see logs indicating it’s waiting on port 9092, etc.
Run as a systemd service
Create a systemd unit file at /etc/systemd/system/kafka.service:
[Unit]
Description=Kafka broker (KRaft mode)
Requires=network.target
After=network.target
[Service]
User=kafka
Group=kafka
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.targetEnable and start:
sudo systemctl daemon-reload
sudo systemctl enable kafka
sudo systemctl start kafkaCheck status:
sudo systemctl status kafkaNow you have a single Kafka broker self-hosted. Next, verify with topic creation, producing/consuming.
Testing (produce and consume)
/opt/kafka/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
echo "hello kafka" | /opt/kafka/bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092
/opt/kafka/bin/kafka-console-consumer.sh --topic test-topic --bootstrap-server localhost:9092 --from-beginningYou should see “hello kafka” echoed on the consumer side.
Multi-node Kafka cluster (KRaft mode)
If you plan to run more than one broker (for redundancy, high availability), set up a 3-node cluster.
Topology & DNS / IP planning
- Use three droplets in same region (ideally).
- Each node has private VPC IP and public IP.
- Use DNS names or static IPs for inter-node communication (e.g.
kafka1.dropletdrift.com,kafka2.dropletdrift.com,kafka3.dropletdrift.com). - All nodes must see each other on the controller port (default 9093) and broker port (9092).
Configuration changes per node
On each node’s /opt/kafka/config/kraft/server.properties, set:
process.roles=broker,controller- Unique
node.id(1, 2, 3) controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093listeners=PLAINTEXT://:9092,CONTROLLER://:9093advertised.listeners=PLAINTEXT://<node’s-public-or-private-host>:9092inter.broker.listener.name=PLAINTEXT
Do not format each node independently against the same cluster ID. Instead, pick one node to format, then copy its cluster-id metadata to the other nodes (or use the kafka-storage.sh tool’s import mechanism) so that all nodes share the same metadata group. The DigitalOcean tutorial for multi-node covers this in more detail.
After configuration, start all brokers (via systemd). Use kafka-metadata-quorum.sh to check cluster health and membership.
Topic creation & data redundancy
Use replication factor ≥ 3 (or at least 2) and partitions spanning nodes to get fault tolerance. Kafka will distribute partitions and leaders among nodes.
Securing & hardening your Kafka cluster
Self-hosting always demands extra security controls. Consider:
Network & firewall
- Use DigitalOcean’s VPC or private networking for inter-node communication so you don’t expose internal traffic to the internet.
- Close port 9092/9093 to public unless necessary; only allow specific IPs (your app servers) via firewall.
- Use
ufworiptablesto permit only needed ports (SSH, Kafka listener, maybe monitoring).
SSL / TLS encryption
Encrypt client <-> broker traffic (and inter-broker traffic) via TLS. Steps:
- Generate CA, broker certificates, client certificates.
- Configure
ssl.keystore.location,ssl.truststore.location,ssl.keystore.password,ssl.truststore.passwordinserver.properties. - Set
listeners=SSL://:9094,CONTROLLER://:9093(or dual listeners) andadvertised.listeners=SSL://host:9094 - In clients, configure SSL truststore, key, and proper endpoints.
Enable SASL (e.g. SCRAM) or OAuth to require clients to authenticate. Then set ACLs (via authorizer) to limit which users can produce, consume, create topics. This adds another layer of protection.
System hardening & permissions
- Run Kafka as non-root (we already use
kafkauser). - Limit open file descriptors (
LimitNOFILE) in systemd. - Use regular OS updates and security patches.
- Backup configuration and data directories.
Monitoring, backups, and maintenance
The same logic that applies to security is also relevant for maintenance and backups. Here’s the modern standard that you should be following to ensure smooth operations.
Monitoring & logging
- Enable JMX metrics (Kafka exports a variety of metrics via JMX).
- Use a tool like Prometheus + Grafana or Datadog to scrape and visualize.
- Collect Kafka logs (server log, controller log) to ELK, Loki, or another centralized logging system.
- Track disk usage, CPU, memory, network I/O, GC pauses.
Backups & snapshots
- Snapshot your data disk (DigitalOcean offers volume snapshots).
- Periodically export topic data (for critical topics) via
kafka-exporteror a custom consumer. - Keep metadata backups (server.properties, cluster-id files).
- Test restores.
Upgrades & rolling restarts
- Upgrade Kafka one node at a time; ensure cluster stays healthy.
- Use controlled rolling restarts: decommission a broker, bring new version up, rejoin, repeat.
- Monitor for compatibility changes across Kafka versions (especially around metadata or inter-broker protocol changes).
Caveats, scaling, and next-steps
When working with Kafka, it is important to distinguish between single-node setups and production clusters. A single node can be useful for development and testing, but it does not provide the resilience needed in production. Because Kafka is I/O intensive, fast SSDs or provisioned IOPS are recommended to handle disk throughput and retention efficiently. Partition counts also require careful planning, since they cannot easily be reduced once created. In larger clusters running in KRaft mode, separating controller and broker roles can improve scalability and stability.
Finally, when the operational overhead of running Kafka in-house becomes significant, using a managed service such as DigitalOcean Managed Kafka can simplify deployment and maintenance.
