Intro to Apache Kafka

We finally start talking about Apache Kafka! Also, Allen is getting acquainted with Aesop, Outlaw is killing clusters, and Joe is paying attention in drama class.

The full show notes are available on the website at https://www.codingblocks.net/episode235

News

• Atlanta Dev Con is coming up, on September 7th, 2024 (www.atldevcon.com)

Intro to Apache Kafka

What is it?

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Core capabilities

• High throughput - Deliver messages at network-limited throughput using a cluster of machines with latencies as low as 2ms.
• Scalable - Scale production clusters up to a thousand brokers, trillions of messages per day, petabytes of data, and hundreds of thousands of partitions. Elastically expand and contract storage and processing
• Permanent storage - Store streams of data safely in a distributed, durable, fault-tolerant cluster.
• High availability - Stretch clusters efficiently over availability zones or connect separate clusters across geographic regions.

Ecosystem

• Built-in stream processing - Process streams of events with joins, aggregations, filters, transformations, and more, using event-time and exactly-once processing.
• Connect to almost anything - Kafka's out-of-the-box Connect interface integrates with hundreds of event sources and event sinks including Postgres, JMS, Elasticsearch, AWS S3, and more.
• Client libraries - Read, write, and process streams of events in a vast array of programming languages
• Large ecosystem of open source tools - Large ecosystem of open source tools: Leverage a vast array of community-driven tooling.

Trust and Ease of Use

• Mission critical - Support mission-critical use cases with guaranteed ordering, zero message loss, and efficient exactly-once processing.
• Trusted by thousands of organizations - Thousands of organizations use Kafka, from internet giants to car manufacturers to stock exchanges. More than 5 million unique lifetime downloads.
• Vast user community - Kafka is one of the five most active projects of the Apache Software Foundation, with hundreds of meetups around the world.

What is it?

• Getting data in real-time from event sources like databases, sensors, mobile devices, cloud services, applications, etc. in the form of streams of events. Those events are stored "durably" (in Kafka) for processing, either in real-time or retrospectively, and then routed to various destinations depending on your needs. It's this continuous flow and processing of data that is known as "streaming data"
  How can it be used? (some examples)
• Processing payments and financial transactions in real-time
• Tracking automobiles and shipments in real time for logistical purposes
• Capture and analyze sensor data from IoT devices or other equipment
• To connect and share data from different divisions in a co