Debezium, Apache Kafka®, and an Acyclic Synchronization Algorithm

Join us for an Apache Kafka® VIRTUAL meetup on Wednesday, July 2nd from 6:30 pm CEST.

Have follow-up questions afterward? Log in to Slack or Join our Forum to ask any follow-up questions!

Speaker:

MD Sayem Ahmed / MTS 2, Software Engineer

Title:

Migrating Millions of Users with Debezium, Apache Kafka, and an Acyclic Synchronization Algorithm

Abstract:

In this talk we will look into the details of how Kleinanzeigen, a leader in classifieds business in Germany, built a data migration system using Apache Kafka and Debezium that migrated millions of users' data from a legacy to a new platform and allowed bi-directional data sync between them in real time. We will also discover how the system allowed user's data to be updated on both platforms (partially, under certain conditions) while keeping the entire system in sync. Finally, we will learn how the system leveraged a logical clock to implement a custom synchronization algorithm that helped avoid infinite update loops between the platforms.

Bio:

MD Sayem Ahmed works at Kleinanzeigen as a software engineer specializing in distributed systems. He works on initiatives that often involve both organisational and technical challenges that span across multiple teams. A major part of his work involves creating systems that can scale to meet both operational and business needs while ensuring resilience and fault tolerance.

Online Meetup Etiquette

• Please hold your questions until the end of the presentation or use the zoom chat during!
• Please arrive on time as zoom meetings can become locked for many reasons (though if you get locked out a recording will be available, but you may have to wait a little while for it!)
• Important note: If Zoom asks for a password to join please use 'kafka'

If you would like to speak or host our next event please let us know!