Distributed Locks, Sagas & Coordination with Azure Cosmos DB | Eric Boyd | Azure Cosmos DB Conf 2026

0:08 I want to talk about a practical coordination stack for cloud systems.

0:13 Let me start with the failure.

0:16 Two users try to buy the same seat

0:19 in a theater for a concert at nearly the same time.

0:25 I'll click into our demo app here, Cosmic Ticks,

0:28 and I'll browse the seats that I can buy.

0:31 Now I'm logged in as Alice, and you'll see that I have some options here.

0:39 I'm going to turn on this race simulator.

0:44 I'm going to simulate this race condition that we

0:48 just talked about two users trying to buy

0:50 the same ticket at the same time So when

0:53 I do that, you'll see that this ticket Floor

0:57 right row three seat seven both Alice and Bob

1:00 tried to buy this ticket at the same time

1:04 and They double booked this they both think if

1:07 I look at my tickets Alice has this ticket,

1:13 and if I look at Bob, Bob thinks he also has this ticket.

1:22 I'm going switch back to Alice,

1:24 and we'll come back to her and buy some more tickets in just a That last

1:31 write silently overwrote the other one and ah

1:35 one user walked away thinking they bought the ticket,

1:39 but they can't possibly both have those tickets.

1:43 There's one seat.

1:44 So this is not a weird edge case.

1:46 This is what distributed systems do when

1:49 concurrency is real and coordination is missing.

1:52 The first problem is the simplest one.

1:57 Two writers, the same record.

2:00 the last right wins.

2:01 Now, if you do nothing, silent data loss is the default behavior.

2:07 Now make it worse.

2:08 Two processes read the same state and both send an email or both

2:14 call a payment processor or both hit

2:17 some external system that you don't control.

2:20 At that point, rollback is mostly fiction.

2:25 Now make it a workflow.

2:27 You save something, you call another service, maybe send an email and step three

2:33 or four fails after earlier work already happened.

2:37 That's the beginning of saga territory.

2:40 And then the retry happens, but now the retry can duplicate these side effects.

2:47 And that is why this talk exists.

2:50 So the question is not just how to store data.

2:52 The question is how to coordinate work correctly under concurrency.

2:57 failure and retry.

3:00 Before I get too far ahead of myself, I'm Eric Boyd.

3:03 I have been in technology and software development for 30 years.

3:07 I'm a 15 time Azure MVP.

3:09 I'm also an AI or Microsoft Foundry MVP.

3:12 And I'm also a Microsoft Regional Director.

3:16 I run responsive X.

3:18 We help customers design, build and modernize cloud native applications,

3:22 data platforms and AI driven systems on Azure.

3:26 That work sits right at the intersection of user experience,

3:30 distributed systems, DevSecOps, and AI, which is exactly why this topic matters.

3:37 And before I name Cosmos DB specifically,

3:40 here's what the platform has to give us.

3:43 We need a way to stop overriding data we've not seen.

3:48 We need a way to react to changes.

3:51 We need a way to expire data automatically.

3:55 We need throughput scale and storage scale.

3:58 We need low latency and multi-region capability and failover.

4:03 That is the selection criteria.

4:05 Now we can talk about the building blocks.

4:08 Cosmos gives us the primitives we need.

4:13 ETAGs plus conditional rights give us optimistic concurrency.

4:17 TTL gives us lease expiration.

4:23 Change feed gives us durable notification of changes.

4:28 Partitions give us scalable key distribution and global replication,

4:34 routing and failover.

4:36 give us the operational footprint to run this in real systems.

4:41 Now, everything else in this talk is built on those primitives.

4:46 I'm going to reproduce that seat purchase race condition,

4:50 but this time the right that I'm going to do will include that ETAG.

4:57 So I'll go back to Cosmic Ticks.

5:01 And here I'm going to simulate.

5:03 I'm going to go back to actually buy tickets for Alice.

5:07 I'm going go back to that same concert or event.

5:13 And here's my race condition demo again.

5:15 You'll see Alice only has this one or thinks

5:19 she has this one seat at the moment.

5:22 I'm going to simulate that race again and now you'll see in the upper left.

5:28 We bought a ticket.

5:29 Alice bought the ticket.

5:30 She won or got the ticket.

5:32 Bob in this race condition failed.

5:35 It says it was taken by another buyer.

5:37 So only one buyer got the seat.

5:39 Now, previously Bob had one ticket.

5:42 If I go look at Bob again, and I look at my tickets for Bob,

5:47 you'll see Bob still only has one ticket.

5:50 That ticket up in the upper left-hand corner did not get attributed to Bob.

5:55 And so that's all using those e-tags.

5:58 So let's go back and talk about what just happened.

6:02 Now, both users retrieved a document before any updates and it

6:06 had an e-tag and we'll say this e-tag starts with value V1.

6:11 Buyer A writes and succeeds.

6:16 Buyer B writes with a stale e-tag.

6:19 So Cosmos rejected that right instead of just silently accepting it.

6:24 Now the race still happened.

6:26 That part didn't just disappear.

6:28 What changed is the platform detected that the document had

6:32 changed after the read and it refused that stale update.

6:36 So the e-tag doesn't eliminate the concurrency.

6:39 just turns that silent corruption into an explicit conflict.

6:44 The code pattern is pretty straightforward.

6:46 We'll read the document that contains the seat.

6:50 We'll capture the resource data.

6:53 We'll then capture the e-tag.

6:56 We'll modify the document.

6:59 We'll write it back, but we'll do it with this if match eTAG header.

7:05 Now if Cosmos returns a precondition failure,

7:08 we're not going to pretend it was successful.

7:11 Instead, we'll tell the user the seat

7:13 changed and they need to refresh the availability.

7:16 eTAG solves one document right,

7:19 but it does not solve these longer critical sections.

7:24 For example, If the operation is read availability,

7:29 place a hold on a seat, update related state, trigger follow-up work.

7:35 Well, two workers can still overlap in that unit of work.

7:39 Even with those per right e-tag protections, the section level race remains.

7:44 So you need exclusive access around that full critical section.

7:48 If we step this up to a distributed lock, maybe we can get that.

7:53 So let's go back to.

7:55 our race condition and let's try another scenario here.

8:00 I'm going to go into that same event or concert.

8:05 And this time I'm going to turn

8:06 on a different feature instead of the race simulator.

8:08 I'm going to turn on best available and what best available does for us is this.

8:14 We want to buy some number of tickets.

8:17 In this case, we're going to say two tickets.

8:19 We don't want to pick.

8:20 the exact seats, we want the system to choose the seats

8:24 that are best for us and just purchase them for us.

8:27 Now the challenge with this is the system

8:29 has to go out and see what's available.

8:31 It then has to get a hold of whatever number of seats

8:35 that we've asked for and buy those before somebody else does.

8:39 And so we need to have some way to kind of temporarily hold those so

8:43 that no one else can buy them while

8:45 we're sorting out from all the available seats,

8:47 which ones are actually the best.

8:49 And so I'm going to start by just

8:52 running this and I'm to simulate this race condition.

8:56 And what we should see again here is two people,

8:59 Alice and Bob, they both got the same seats.

9:03 They both bought what we think are the best seats in this venue,

9:08 floor center, uh, five and six.

9:10 So both of them have them.

9:11 Well, that can't happen again.

9:13 We can't sell the same seats to two different people.

9:17 So if I change this and I say, let's use a distributed lock here.

9:21 So behind the scenes,

9:22 I'm just toggling on a distributed lock feature and we go try this again.

9:27 And I say, simulate this race condition.

9:30 Well, now you'll see that Alice and Bob still successfully bought seats,

9:34 but they actually bought their own seats.

9:36 So Alice got seats uh in that center floor section, uh row one, three and four.

9:43 Bob got seats in that center floor section.

9:46 row two, five and six, that dead middle center uh area.

9:52 So in this case, that distributed lock worked.

9:54 It held on to tickets, bought them,

9:56 let them buy the tickets before letting somebody else get in there.

10:01 Let's go back and explain really what happened here.

10:04 Process A acquired the lock for the seat and it

10:08 entered into that critical section we talked about earlier.

10:11 Process B tried to acquire the same lock, but it couldn't.

10:14 I couldn't proceed with that resource with those specific seats.

10:17 Only one user was able to get to that sensitive

10:21 critical section at any given time and do the operation.

10:25 Now, when the lock is released or it expires,

10:29 then process B could get ahold of this.

10:32 this case, know, Alice and Bob fighting for the same tickets.

10:37 And code, it's pretty simple.

10:38 I'm going to build this in what kind of looks like an awkward way,

10:41 but the The first thing we're going to start with is what we want to do.

10:43 We want to do this protected work.

10:44 We want to reserve a seat for an individual buyer.

10:49 Well, to do that, we need to acquire a lock.

10:53 We need to enter that critical section using a weight,

10:56 wrapping that with a using block with a weight,

10:59 and then we will release it because of that await that disposable pattern.

11:03 We're going to release it automatically when that handle is disposed.

11:07 We'll handle any errors that occur and react appropriately.

11:11 And if the acquisition fails, we don't let the second worker continue.

11:16 Now, this is where teams get themselves into a little bit of trouble.

11:19 The first thing to talk about is fence tokens.

11:21 Now, if a worker pauses,

11:24 its lease can expire and the worker then has a token and it becomes a zombie.

11:31 What I mean by that is another worker can take

11:34 over and then that original worker, if it wakes up,

11:38 It's a zombie, it can still resume unless we

11:42 protect the resources by using some monotonically increasing token.

11:46 And this is where fence tokens come into play.

11:50 The second is renewal.

11:52 Leases expire on wall clock time.

11:55 So if the worker runs long, the lease, you have to renew it.

12:00 If the renewal fails, you have to stop.

12:03 You no longer own that lock exclusively.

12:08 And then third is retry.

12:10 Under contention, backoff and jitter are not optional.

12:15 Uniform retries can create a thundering

12:18 herd attacking and going after these locks.

12:23 We're going to step into a different class of problems.

12:26 Locks are great for those short critical sections.

12:29 They are the wrong tool for long

12:33 running workflows that cross service and network boundaries.

12:38 In Cosmic Ticks, uh purchase flow may

12:41 reserve inventory and may charge a payment,

12:44 send confirmation, update several pieces of state.

12:47 You can't responsibly hold a short lease around that entire journey.

12:52 So let's take a look at this in a workflow version of this problem.

12:57 So I'm going to change my config just a little

13:00 bit to say we're going to use a different purchase mode.

13:02 And in this case, we are going to use Sagas.

13:06 to accomplish these purchases.

13:07 I'm going go back to our seats in our concert venue.

13:12 I'm not going to use best available here.

13:15 Instead, I'm just going to say, let's buy a single ticket.

13:18 So I'm to pick this ticket and I'm going to say, let's buy it.

13:22 And when I do that, it says my purchase succeeded.

13:26 There's no contention on that.

13:27 But what actually happened?

13:29 Well, if we go look, I'm going to turn on my activity log and we're

13:33 going to see I attempted to make this purchase.

13:36 I started the purchase saga for one seat.

13:40 And as part of that workflow, I did several things.

13:42 I held the seat.

13:44 I charged the card.

13:46 The payment was approved.

13:48 And then I issued tickets with the ticketing system.

13:51 And then I sent an email or a confirmation

13:54 notifying Alice that she received and bought these tickets.

13:57 So that was my saga.

13:58 Those were my multiple steps in the workflow.

14:01 But what happens if a failure occurs?

14:04 What happens if maybe the credit card processing system is failing?

14:09 So I just turned on a toggle that says,

14:12 let's fail credit card processing and we are going to buy another ticket.

14:16 So I'm going go down to the lower right section.

14:18 It's a little cheaper.

14:19 I'm going to buy these tickets.

14:21 And what we're seeing now is I started that purchase saga,

14:25 but the purchase was rolled back.

14:27 You'll see up here.

14:28 I don't have that in a red dot anymore.

14:30 I didn't lock up that ticket.

14:32 The saga started.

14:33 I held the seats, I tried to charge the card, but the payment was declined.

14:37 And so it compensated.

14:39 It released the held seats and it returned them back so I could buy them again.

14:44 Um, the, the credit card payment failed.

14:47 Well, what if it fails at a later point?

14:49 What if the card payment goes through, but the ticketing system has a failure?

14:55 the ticketing system is down or something like that.

14:57 Well, let's turn that on.

14:58 So I've turned on the ticketing system failing.

15:01 We still have that same seat available to us because it failed the last time.

15:04 Let's try to buy it again.

15:06 And now in our activity log, we see the purchase was rolled back.

15:08 Uh, but when we tried to buy the ticket, we reserved and held the seat.

15:13 We charged the credit card that went fine,

15:16 but we couldn't issue the ticket because the ticketing system was down.

15:21 And so here we compensated by refunding the card,

15:25 releasing the seats and it's still available.

15:28 All right.

15:30 Let's.

15:31 talk about what just happened here and how we did it.

15:37 So a saga treats this workflow as an explicit state.

15:41 We have one saga document per purchase in this case.

15:45 Each step moves through the state machine

15:48 and each step moves it forward conditionally.

15:51 The document records what completed, what's coming up next,

15:55 and what must be compensated if a recovery is needed.

16:00 Now on retry, resume from the recorded state is what we need to do.

16:05 And then we need to skip the completed states on a failure.

16:08 want to execute compensation by kind of backing up through

16:11 it in reverse order for all of those committed steps.

16:16 Now from code, we can look at it like this.

16:18 So I built a little library here and I'm kind of hiding what's underneath here,

16:21 but I'm going to share that code with you at the end as well.

16:24 So if you Think about it like this, we need to load or create the saga instance.

16:28 And so we have all these steps that we identified,

16:31 hold seats, charge card, issue tickets, send confirmation.

16:34 And in each of those we have, here's the process we're trying to do.

16:39 Here's what we do if we need to compensate.

16:41 So we hold seats, we release seats, we charge card,

16:44 we refund card, we issue tickets, we revoke tickets on and on.

16:49 Then we execute the next steps based on the current state.

16:52 We write the transition with optimistic.

16:54 currency and record completion before we move on.

16:58 On retry, we would resume from the recorded

17:01 state instead of replaying the work blindly.

17:04 And when the saga execution completes, we're going to react to the result.

17:09 And so here we're going to get that result back

17:10 and we're going to do whatever we need to do with that.

17:12 The purchase was completed,

17:13 it was rolled back and we saw those toast notifications in the app.

17:17 Now there's some details that matter when we make this work.

17:20 First, every step must be item potent.

17:24 or guarded by a recorded completion.

17:28 Compensation is deliberate.

17:31 You saw the compensation step, refund card, uh release seats.

17:37 Those are deliberate.

17:38 They're not magic.

17:39 They're not some sort of magical rollback.

17:41 And the compensation intent, it should be written ahead of time.

17:45 So if the process crashes mid step,

17:48 recovery still knows what to do to undo or counteract that.

17:53 ETAG conditional rights protect SAGA state transitions against

17:56 these kind of racing executors that we saw.

18:01 Now suppose the workflow succeeds, but you still need to tell other systems.

18:08 If you save the state and then publish directly,

18:11 there is a crash gap that occurs between the commit and the notification.

18:16 You need the intent to notify to be as durable.

18:20 as the state change itself.

18:21 And so this is where the outbox comes into play.

18:24 So I'm going to go back to my application and we'll take this scenario.

18:31 For example, one of the things you saw

18:34 in the code was there wasn't a compensation step for messaging.

18:38 And so if we simulate a messaging failure here, and in this case,

18:43 we try to buy the tickets, you'll notice the purchase succeeded.

18:47 We got all the way through the saga, at the end,

18:51 we didn't actually send the emails to let

18:53 the users know that they purchased the tickets.

18:56 Now, me as a user, I know I clicked purchase on the tickets.

19:00 I saw the toast notification come through.

19:01 I see it in my account in the app.

19:03 And so the notification is not super important,

19:06 but what if I'm transferring tickets between two users?

19:09 So let's go over to the tickets that I already have.

19:14 And these are my active tickets for this concert.

19:17 And let's say I want to transfer as Alice one of my tickets to Bob.

19:22 So we'll pick this one right here.

19:23 And in this case, I'm going to leave this messaging failure in place

19:30 and I'm just going to try to transfer this ticket to Bob.

19:38 So the tickets say they transferred successfully,

19:41 but we have a messaging failure.

19:44 So Bob doesn't know that the ticket was sent to him.

19:49 He never got a notification of that.

19:51 And I also wanted to send a notification to uh Alice just as a receipt,

19:55 as well as she sent the tickets to Bob, but she didn't get that either.

20:00 But the ticket did leave Alice's active tickets.

20:04 Well, what if we change this and we use an outbox

20:08 instead of just doing it transactionally like we're doing now?

20:11 And we'll grab another one of these tickets.

20:13 I'm going to grab this ticket in the upper left

20:15 hand corner and again I'm going to transfer it to Bob.

20:20 And I'm going just press transfer here.

20:23 Now in this case, the tickets transferred successfully.

20:27 But you'll see we.

20:29 Wrote it in the outbox.

20:31 And so the published failed.

20:33 It's not able to send the notification,

20:34 but it's in the outbox and it knows that it needs

20:38 to, and it's holding onto that because it happened as one transactional right?

20:41 And so if the messaging system gets fixed, I'm going to fix it right now.

20:46 Then what we'll see is.

20:47 because I'm reprocessing and responding to the events in the outbox,

20:51 eventually we'll see the messages succeed and we'll see a message go out

20:55 to Alice and a message go out to Bob and there they are.

20:59 The previous owner got a message delivered and the new

21:02 owner got a message delivered that they've received the transfer.

21:06 So you can think of this as two stages.

21:08 Stage one writes the business state as a pending message together.

21:13 Stage two is a relay that reads the pending

21:17 messages and publishes them and marks them sent.

21:19 The outbox closes the gap between those state changes and the intent to notify.

21:24 So what you're guaranteeing is at least once delivery.

21:28 So downstream handlers still have to be item potent.

21:33 And in code, you can think of it like this.

21:35 When the purchase completes or the saga advances,

21:38 I'm going to create a transaction.

21:41 I'm going to write the domain change in that transaction.

21:44 I'm going to also write the pending

21:47 outbox record in that same durable operation.

21:49 I'll save or execute the batch and then

21:52 I'll have a worker read the unsent messages.

21:54 I'll publish them and I'll mark them as dispatched.

21:59 The outbox is not exactly once.

22:02 It's at least once.

22:04 means retries can duplicate publication.

22:07 So handlers must be item potent.

22:10 You also need operational visibility into relay lag,

22:14 backlog growth, and recovery after crashes.

22:18 Here is the production reality check.

22:22 Consistency level matters.

22:25 Multi-region rights can violate if two regions

22:31 both accept conflicting conditional operations before convergence.

22:36 And hotspots matter.

22:38 How you partition your data matters.

22:42 The retry behavior also matters, and once you add uh outbox,

22:46 at least once delivery and relay backlog become operational concerns,

22:51 not these theoretical ones.

22:52 So for coordination data, correctness is going to convenience every time.

23:00 So the pattern map is simple.

23:02 Use eTAG for single document concurrency.

23:07 Use a lock for short, exclusive critical sections.

23:12 Use a saga for multi-step workflows.

23:16 Use an outbox for reliable post-commit publication.

23:21 Do not use timestamps as ordering truth.

23:24 Do not hold locks across those external calls.

23:28 Do not assume direct publish is safe after.

23:33 If you want more detail, I'd encourage you to visit ericboyd.com and there

23:38 you'll find a companion article with the patterns, code, and resources.

23:42 You'll also find the sample code and the deck links there too.

23:47 Thanks again.