Meetup #110 - Life Beyond Distributed Transactions

At a certain scale, transactions become impractical. In fact, in the era of microservices, many developers run into the problem of having to work around not being able to use ACID transactions across multiple of these distributed (micro)services. The paper that we are going to discuss this week describes approaches to dealing with partitioned data in the context of transactions.

Paper to read:

Life beyond Distributed Transactions:an Apostate’s Opinion by Pat Helland

Meeting Lifeguard:

• Grzegorz Bańczak
  

Meeting location:

Medius, ul. Królewska 57, Biprostal Building 8th floor

Abstract:

Many decades of work have been invested in the area of distributed transactions including protocols such as 2PC, Paxos, and various approaches to quorum. These protocols provide the application programmer a façade of global serializability. Personally, I have invested a nontrivial portion of my career as a strong advocate for the implementation and use of platforms providing guarantees of global serializability.

My experience over the last decade has led me to liken these platforms to the Maginot Line. In general, application developers simply do notimplement large scalable applications assumingdistributed transactions. When they attempt touse distributed transactions, the projects founderbecause the performance costs and fragility makethem impractical. Natural selection kicks in.

Instead, applications are built using different techniques which do not provide the same transactional guarantees but still meet the needs of their businesses. This paper explores and names some of the practical approaches used in the implementations of large-scale mission-critical applications in a world which rejects distributed transactions. We discuss the management of fine-grained pieces of application data which may be repartitioned overtime as the application grows. We also discuss the design patterns used in sending messages between these repartitionable pieces of data. The reason for starting this discussion is to raise awareness of new patterns for two reasons. First, it is my belief that this awareness can ease the challenges of people hand-crafting very large scalable applications. Second, by observing the patterns, hopefully the industry can work towards the creation of platforms that make it easier to build these very large applications.