Topics, Partitions, and APIs oh my! This episode we're getting further into how Apache Kafka works and its use cases. Also, Allen is staying dry, Joe goes for broke, and Michael (eventually) gets on the right page.
The full show notes are available on the website at https://www.codingblocks.net/episode236
News
- Thanks for the reviews! angingjellies and Nick Brooker
- Please leave us a review! (/review)
- Atlanta Dev Con is coming up, on September 7th, 2024 (www.atldevcon.com)
Kafka Topics
- They are partitioned - this means they are distributed (or can be) across multiple Kafka brokers into "buckets"
- New events written to Kafka are appended to partitions
- The distribution of data across brokers is what allows Kafka to scale so well as data can be written to and read from many brokers simultaneously
- Events with the same key are written to the same partition as the original event
- Kafka guarantees reads of events within a partition are always read in the order that they were written
- For fault tolerance and high availability, topics can be replicated…even across regions and data centers
- NOTE: If you're using a cloud provider, know that this can be very costly as you pay for inbound and outbound traffic across regions and availability zones
- Typical replication configurations for production setups are 3 replicas
Kafka APIS
- Admin API - used for managing and inspecting topics, brokers, and other Kafka objects
- Producer API - used to write events to Kafka topics
- Consumer API - used to read data from Kafka topics
- Kafka Streams API - the ability to implement stream processing applications/microservices. Some of the key functionality includes functions for transformations, stateful operations like aggregations, joins, windowing, and more
- In the Kafka streams world, these transformations and aggregations are typically written to other topics (in from one topic, out to one or more other topics)
- Kafka Connect API - allows for the use of reusable import and export connectors that usually connect external systems. These connectors allow you to gather data from an external system (like a database using CDC) and write that data to Kafka. Then you could have another connector that could push that data to another system OR it could be used for transforming data in your streams application
- These connectors are referred to as Sources and Sinks in the connector portfolio (confluent.io)
- Source - gets data from an external system and writes it to a Kafka topic
- Sink - pushes data to an external system from a Kafka topic
Use Cases
- Message queue - usually talking about replacing something like ActiveMQ or RabbitMQ
- Message brokers are often used for responsive types of processing, decoupling systems, etc. - Kafka is usually a great
Popular Podcasts
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!
Ruthie's Table 4
For more than 30 years The River Cafe in London, has been the home-from-home of artists, architects, designers, actors, collectors, writers, activists, and politicians. Michael Caine, Glenn Close, JJ Abrams, Steve McQueen, Victoria and David Beckham, and Lily Allen, are just some of the people who love to call The River Cafe home. On River Cafe Table 4, Rogers sits down with her customers—who have become friends—to talk about food memories. Table 4 explores how food impacts every aspect of our lives. “Foods is politics, food is cultural, food is how you express love, food is about your heritage, it defines who you and who you want to be,” says Rogers. Each week, Rogers invites her guest to reminisce about family suppers and first dates, what they cook, how they eat when performing, the restaurants they choose, and what food they seek when they need comfort. And to punctuate each episode of Table 4, guests such as Ralph Fiennes, Emily Blunt, and Alfonso Cuarón, read their favourite recipe from one of the best-selling River Cafe cookbooks. Table 4 itself, is situated near The River Cafe’s open kitchen, close to the bright pink wood-fired oven and next to the glossy yellow pass, where Ruthie oversees the restaurant. You are invited to take a seat at this intimate table and join the conversation. For more information, recipes, and ingredients, go to https://shoptherivercafe.co.uk/ Web: https://rivercafe.co.uk/ Instagram: www.instagram.com/therivercafelondon/ Facebook: https://en-gb.facebook.com/therivercafelondon/ For more podcasts from iHeartRadio, visit the iheartradio app, apple podcasts, or wherever you listen to your favorite shows. Learn more about your ad-choices at https://www.iheartpodcastnetwork.com
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.