I’m running a Kafka cluster within Docker and exploring different configurations for optimal performance. I have four scenarios in mind:
Scenario 1: Two Kafka servers with two brokers each (ksrv1:9092,9093 & ksrv2:9092,9093), served by a single ZooKeeper.
Scenario 2: Same as Scenario 1, but with two or three ZooKeepers for increased fault tolerance.
Scenario 3: Each Kafka server with a single broker and a dedicated ZooKeeper.
Scenario 4: Similar to Scenario 3, but multiple ZooKeepers for high fault tolerance.
which scenario has better performance?
>Solution :
Kafka doesn’t require Zookeeper anymore. It especially should not be "one / multiple per broker". Kafka clusters can have tens of brokers, but only need up to 7 Zookeeper, max. If you use Kraft mode, then you need zero.
Multiple brokers (or Zookeeper) on a single host is unlikely to improve performance. Your single disk, cpu, and RAM will be the limiting factors, especially if data starts replicating within the same host disk