Topics, Partitions, and APIs oh my! This episode we're getting further into how Apache Kafka works and its use cases. Also, Allen is staying dry, Joe goes for broke, and Michael (eventually) gets on the right page.
The full show notes are available on the website at https://www.codingblocks.net/episode236
News
- Thanks for the reviews! angingjellies and Nick Brooker
- Please leave us a review! (/review)
- Atlanta Dev Con is coming up, on September 7th, 2024 (www.atldevcon.com)
Kafka Topics
- They are partitioned - this means they are distributed (or can be) across multiple Kafka brokers into "buckets"
- New events written to Kafka are appended to partitions
- The distribution of data across brokers is what allows Kafka to scale so well as data can be written to and read from many brokers simultaneously
- Events with the same key are written to the same partition as the original event
- Kafka guarantees reads of events within a partition are always read in the order that they were written
- For fault tolerance and high availability, topics can be replicated…even across regions and data centers
- NOTE: If you're using a cloud provider, know that this can be very costly as you pay for inbound and outbound traffic across regions and availability zones
- Typical replication configurations for production setups are 3 replicas
Kafka APIS
- Admin API - used for managing and inspecting topics, brokers, and other Kafka objects
- Producer API - used to write events to Kafka topics
- Consumer API - used to read data from Kafka topics
- Kafka Streams API - the ability to implement stream processing applications/microservices. Some of the key functionality includes functions for transformations, stateful operations like aggregations, joins, windowing, and more
- In the Kafka streams world, these transformations and aggregations are typically written to other topics (in from one topic, out to one or more other topics)
- Kafka Connect API - allows for the use of reusable import and export connectors that usually connect external systems. These connectors allow you to gather data from an external system (like a database using CDC) and write that data to Kafka. Then you could have another connector that could push that data to another system OR it could be used for transforming data in your streams applic
Popular Podcasts
Las Culturistas with Matt Rogers and Bowen Yang
Ding dong! Join your culture consultants, Matt Rogers and Bowen Yang, on an unforgettable journey into the beating heart of CULTURE. Alongside sizzling special guests, they GET INTO the hottest pop-culture moments of the day and the formative cultural experiences that turned them into Culturistas. Produced by the Big Money Players Network and iHeartRadio.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
Model Wars
On the surface, the modeling industry in the 80s was about parties and pretty clothes, but underneath the flash, there were drugs, sex, and violence. And Paul Fisher, a former straight-A student from the San Fernando Valley, rose to prominence amidst the turmoil. After a chance encounter with an alleged mobster, Paul suddenly found himself with a million dollars in funding and the chance to start a modeling agency in New York City. The city was then embroiled in a “model war” between industry titans Eileen Ford and John Casablancas, and Paul tried to make his mark. But even as Paul thrived in the party-heavy culture of the time, burning cash like a furnace, he came face to face with the darkest forces in the industry. As his shady backers put the screws to him, Paul faced down deadly consequences, trying to make a star while saving his soul.