General Information

• How Kafka Stores Data (blog post)
• Integrating NiFi w/Kafka (blog post)
• Q&A on Max Size for Message; https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
• Writing Custom Partitioner examples
  • https://www.learningjournal.guru/courses/kafka/kafka-foundation-training/custom-partitioner/
  • https://www.javaworld.com/article/3066873/big-data-messaging-with-kafka-part-2.html
• Confluent Blogs
  • Kafka Streams; https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/
  • KSQL; https://www.confluent.io/blog/ksql-streaming-sql-for-apache-kafka/
• topology supervision features of streaming frameworks (or lack thereof)

Sizing & Best Practices

• Kafka Best Practices (HCC article) plus some additional HWX Consulting experience of
  
  • Use newer API to put less load on Zookeeper if same zookeeper is shared with other services like Namenode HA, HBase etc.
  • Use multiple and dedicated disks for Kafka for better throughput.
  • Kafka parallelism is tightly coupled with number of Partitions so make sure number of partitions are correct. General recommendation is to use one partition per physical disk and one consumer per partition. The maximum number of consumers for a topic is equal to the number of partitions. Consumers in the same consumer group split the partitions among them.
  • As far as sizing is concern, it depends on message size, Daily volume, retention period and I/O requirement.
• Unofficial Storm and Kafka Best Practices Guide (HCC article)
•