I recently heard that Apache Kafka does not store data on disks. Yeah, I’m not kidding, my mate told me, that Kafka manages most of the operations in memory and doesn’t rely heavily on disks. That seemed odd to me, and today I will try to investigate Kafka’s persistence process...
Continue reading...Kafka
Increasing reliability/durability of Apache Kafka
What comes to your mind when you hear the term “Distributed Messaging”? Well, not even that, let’s paraphrase: What are your expectations concerning the messaging in distributed systems? I suppose that “reliability” will occupy one of the top positions in the list of answers. What sense do we put into...
Continue reading...Beware of Message Ordering In Apache Kafka Topic Partitions
Hi! I’ve started a new series on my YouTube channel named “Alex Tried to Understand”. It will be short tech-related clips dedicated to some technical aspects. The first episode is dedicated to the ordering guarantees in Apache Kafka. Synopsis The official docs state that Kafka preserves message ordering inside a...
Continue reading...Add healthchecks for Apache Kafka in docker-compose
Let’s say we have an application inside docker image that connects to Apache Kafka on start and tries to publish a bunch of messages to topics. Now let’s imagine that you want to test your app locally and thus you’ve created a docker-compose.yml with Apache Kafka, Zookeeper, and your application...
Continue reading...Hands-on: Akka + Akka HTTP + Akka Streams + Alpakka Java Tutorial
Which technologies will be used Today Akka offers a great technology stack that is built on top of the Actor paradigm. It delivers a great Akka Stream API which together with a simplified concurrency model and great abilities for use and control backpressure empowers us with top-class tools to create...
Continue reading...Backup unhandled events to Kafka with shutdown hook using @PreDestroy
I was struggling with a task which I’ve mentally called “a distributed back-ups using Kafka”. So the idea in a few words was that we had a service used to invoke database functions. The service had a few RESTful endpoints and was accepting a predefined set of input objects along...
Continue reading...Visualizing Kafka Streams topology
Kafka Streams generates an ASCII Kafka Topology and prints it to the console during startup. These ASCII topologies can be huge and hard to understand. There is a handy tool which allows you to visualize your ASCII topology to a hand drawn diagram. Just copy the topology from the console...
Continue reading...