Use AI to Achieve Operational Excellence with the Well-Architected Framework practices

0:06 So I was super excited to have you here today

0:09 talking about AI in operational excellence for the Well-Architected Framework.

0:12 Um, so the Well-Architected framework is a tool

0:16 that's used by cloud architects around the world.

0:19 Um, it's been fairly standardized by now to help provide guidelines,

0:23 provides best practices for the use of, uh, of these technologies.

0:27 But it's not just about picking the right technologies.

0:31 It's also about managing the risks,

0:33 identifying the trade offs that need to be made.

0:37 Um, specifically today we're talking about, uh, about operational excellence.

0:41 Um, operational excellence.

0:43 It sometimes feels like it's already nailed down a bit.

0:47 Um, is that something that's super exciting in this, in this AI space right now?

0:51 It is.

0:52 I think, um, first we need to differentiate two things, right?

0:56 Like the the part of operational excellence

0:58 where you run AI applications and then you need operational excellence and a lot

1:03 of those patterns are being provided by that.

1:06 Right.

1:06 And then there's also leveraging AI as part of operational excellence.

1:10 And I think we're going to talk about the latter one.

1:12 So I think AI actually adds a lot of value to operational excellence,

1:16 which in my mind and going back to your question is, um,

1:20 pretty much the most or one of the most important pillars,

1:23 because a lot of customers I've talked to a lot of companies,

1:27 they don't think about day two, right?

1:30 It's like, hey, how do I build an application?

1:33 Very easy.

1:34 I have my checklist go through it.

1:36 But the real challenge for any type of application is day two.

1:40 Right.

1:40 And even though you're using cloud services, it's not going to change it.

1:43 It's not going to relieve you from your responsibility there.

1:46 That's why I think AI adds a lot of value

1:49 or will add a lot of value to operational excellence in life.

1:53 But with everything, take it with a grain of salt.

1:57 Make sure you know what you're doing.

1:59 Yeah.

1:59 Absolutely.

1:59 Yeah.

1:59 Because it does have some key topics in it, right?

2:02 We're talking about about observability.

2:04 We're talking about deployment practices, troubleshooting.

2:06 That's all in the in the realm of operational excellence.

2:10 Um, and when we talk about AI opportunities, obviously these are boundless.

2:15 Everyone is introducing agents everywhere.

2:17 We need AI everywhere.

2:18 Does it matter?

2:19 Every piece of MCP servers, we need them all in there right now.

2:24 So when we think specifically about

2:27 the AI opportunity in operational excellence, what kind of thing comes to mind?

2:31 Just the sort of boundless fantasy about what does the future look like?

2:36 Well, I think if you think about where to start there.

2:39 Right.

2:40 Because to your point, operational excellence is a broad topic.

2:43 But I believe, um, if you think about it,

2:47 the the key canons of operational excellence,

2:50 meaning being able to operate something very well, you need the data, right.

2:55 And typically this falls under this big term observability.

2:58 So I think in the observability space there is a lot of opportunity for AI.

3:04 We use this internally in our engineering.

3:07 As you know, I'm part of our engineering organizations and we, um,

3:11 leverage a lot of AI across the entire operation spectrum,

3:16 but also across like troubleshooting or artifact generation.

3:20 But I would say like the observability part is, uh, tremendously important.

3:26 Yeah.

3:27 Um, because think about and I think we're going

3:30 to touch a little bit on risks and trade offs.

3:33 Right.

3:34 As we all know, like large language models.

3:36 And by that agents are non-deterministic.

3:39 Right.

3:39 So you're not always getting the same

3:41 outcome and becomes actually really important.

3:44 Um, to know what you're doing, especially if you depend entirely

3:48 on those agents consuming your observability data.

3:51 Exactly.

3:51 Yeah.

3:51 But that being said,

3:52 we have a lot of patterns and practices because we're also learning internally

3:56 how to best use those tools that we are bringing into WAF as well.

4:01 Yeah.

4:01 Yeah, yeah.

4:02 So we can put a number of, of opportunities we see on screen.

4:05 And these are things that we came up with in conversations with customers,

4:09 but also with internal SMEs.

4:10 Um, about what you can do with AI.

4:14 So we're talking about, um, things like architecture simulations,

4:18 um, automated automatic remediation of all sorts of production issues.

4:23 That sounds wonderful, but, um, I'm sure that at Microsoft,

4:27 that's super applicable because we run these large and complex systems, right?

4:31 Um, if I'm a smaller application team, if I'm an architect,

4:35 that's just getting started on on operationalizing my application

4:38 the day two practices, where can I start?

4:41 How can I make this tangible?

4:43 So I really think, uh, first of all, you need to like,

4:47 standardize on a common schema where your signals are

4:51 being or the way your signals are being expressed.

4:54 Right.

4:54 And if you think about like in general, opentelemetry has been a tool of choice,

5:00 has been like the schema of choice for cloud native applications.

5:04 And now open telemetry feeds a lot into like AI.

5:07 We have AI spans.

5:09 Now we add new, more features to Opentelemetry

5:12 to really cater to AI based systems, if you will.

5:18 And from that perspective, I would say start with observability.

5:22 Always start with observability followed by evaluations.

5:25 If you have more than one or as part of math,

5:28 if you leverage more than one agent,

5:30 right, and then put evaluations into it and then security,

5:33 I think that's those are the most important.

5:36 Yeah.

5:36 So so if we if we make it even more tangible.

5:40 So so what can we do now with AI in these scenarios

5:43 with, in observability that we couldn't do two years ago?

5:47 Yeah.

5:47 So that's actually a great question right.

5:50 Because so if you think about how we do things

5:53 internally now and I think that's a very good example.

5:57 So before you had with AI you

6:00 had pattern detection and traditional machine learning right.

6:03 So you could detect a pattern.

6:05 What's happening now is it's basically a combination.

6:08 So your application has certain criteria,

6:10 service level indicators that basically tell the system,

6:13 hey, I'm doing well, right?

6:15 So now if your system is about to fail

6:18 because the service level indicator is switched to yellow,

6:21 you can actually use AI in multiple facets.

6:24 Because one very important aspect is AI can help you find a dependency graph,

6:30 because components or services usually don't fall in isolation,

6:34 there's always an impact, right?

6:36 Or most of the systems don't have an impact,

6:39 so you can actually use it to find a dependency graph.

6:42 You can also use AI now to actually figure out, hey,

6:46 this service is about to fail because the service

6:49 level indicator is the signals are turning yellow, right?

6:52 I need to dispatch a direct responsibility individual right away to look at it.

6:57 Then when it really comes to push, comes to shove,

7:01 let's say that system says, hey, that's a P0.

7:03 You really want to have a human in the loop

7:06 right before you dispatch everything, you stop things.

7:08 You want to have some sort of validation.

7:11 And I think this is where also some of the limitations are still there.

7:16 Even though we have AI as a judge and and patterns, we can work that.

7:21 But I think especially when it comes to very critical things,

7:25 to make sure that fix is not having an impact

7:30 on other things or a lot of compliant security things.

7:34 Right?

7:34 Especially in the compliance area these days,

7:37 you you want to have a human in the.

7:39 Loop, right?

7:39 So and you also need to identify those areas with the right criticality,

7:43 with the right level of governance where you can fit this in.

7:46 So are you saying that introducing AI here is is not

7:49 going to be an approach of move fast and break things.

7:53 See where we end up?

7:55 Well, so that it's a good discussion because move

7:58 fast and break things is actually a good pattern.

8:01 If you think about fail fast, if your system is set up the right way.

8:05 Right.

8:05 So because if you think about traditional cloud architectures,

8:08 we actually embrace a fail fast notion, right?

8:11 Catch which basically means catch the error early.

8:13 You need to think about the same pattern applicable to, um,

8:17 like if you leverage AI as part of your wealth.

8:20 Right.

8:21 So you need to make sure that you know that, uh,

8:24 where the boundaries are, right, and how to mitigate it.

8:28 And that goes back to like what I said before, observability.

8:32 Valuation and is very key.

8:34 Yeah.

8:34 So so the key that you mentioned is making sure

8:37 that your system is set up correctly with the right guardrails.

8:39 And that means that in order to set that up correctly,

8:42 you need to take the right risks and trade offs into account.

8:46 Um, so when we talk about these risks,

8:48 you mentioned earlier that, um, AI is non-deterministic by nature.

8:53 That's going to have a big impact

8:56 on specific capabilities like summarization with correctness issues.

9:00 Um, the root cause analysis that, that have incorrectness, uh, issues,

9:05 what kind of risks and what kind of trade offs are you at Microsoft looking at?

9:11 So we are I mean, there are risks.

9:14 There are obvious risks as you as you highlighted.

9:17 Right.

9:18 Um, there are certain patterns that we

9:21 leverage internally to trying to automate, uh, and minimize those risks.

9:25 So I mentioned AI as a charge rate so that you

9:28 basically have another agent controlling the output of one agent.

9:32 Yeah.

9:32 We also invest heavily in two evils.

9:35 Just speaking about, uh, hallucinations and non-deterministic behavior.

9:38 But then there are certain risks, I think, uh, where we're headed,

9:43 but where you still need to have a human in the loop.

9:47 Right?

9:48 And that's usually in, uh,

9:50 in cases where you have to apply to certain regulations.

9:55 Right.

9:55 We need to make sure that your whatever you do there, uh,

9:59 meets the needs of those regulations, governance and so on and so forth.

10:03 So I think it really comes down

10:05 to finding the right balance there between risks.

10:08 Again, as I said, there's trade offs for a lot of those risks,

10:12 but some of them are really human in the loop,

10:14 for example, as a trader, that's a good one.

10:16 Uh, what about, um, security and privacy considerations?

10:19 Should we just give our AI tools broad access to anything

10:24 they might need to come up with recommendations with remediation for issues?

10:29 That's excellent.

10:29 I mean, it really.

10:30 I hate this answer because I'm an engineer and I'm not in consulting,

10:34 but it really depends, right?

10:36 It depends on where you sit.

10:39 If you're building services, if you operate services,

10:42 if you consume services on top of other services.

10:45 So I think a general rule of thumb is give

10:50 AI as much information as necessary in the security space.

10:56 Obviously adhering to a like security rules,

11:00 boundaries and things like that, but the more the better.

11:03 So if you're if you're running a platform, if you're building a platform,

11:07 for example, what you need to think about then is what is PII, right?

11:11 What kind of data do I really need

11:13 to operate the platform on behalf of the customer?

11:16 And then thinking about maybe using AI to obfuscate like PII.

11:21 Exactly.

11:21 Essentially masking whatever.

11:22 It doesn't need to.

11:24 To have excellence.

11:25 Okay, so let's take a look on, um, on how WAF is here to help.

11:30 So what can the well-architected framework, uh, do for you here?

11:34 Because we we talked about it before.

11:36 It's not about presenting a checklist.

11:39 It's just more about giving you the considerations.

11:42 Right.

11:43 The highlighting, the risks, the the mitigations.

11:45 So if you're that cloud architect on your journey to improve opex with AI,

11:50 um, what are some of the risks and trade offs to discuss?

11:53 First, what do you need to, um, put at the top of your list?

11:58 So I think again, the typical things are like

12:01 if you think about WAF as you describe, right.

12:04 It's a very good checklist to make

12:07 sure you especially in the operational excellence, you check all the boxes.

12:11 Right.

12:11 So now if we think about how AI can enrich

12:14 WAF or how you can leverage AI as part of RAF,

12:18 I think the first mitigations and trade offs you need to think about is like,

12:22 okay, where do I need to use AI?

12:25 Or where do I want to use AI?

12:27 Right?

12:27 Because there are certain things that don't work

12:29 and there is a lot of it's not like turnkey.

12:32 There's a lot of work needed.

12:34 Right?

12:34 Because even at at the symbols level.

12:36 Right.

12:36 What we see customers do is I mean, they experiment.

12:39 What's the right model for my agent?

12:41 Right.

12:41 And so in in regards to the web framework,

12:45 I think using AI makes a living thing.

12:48 So it's more dynamic.

12:50 It's not a static checklist.

12:52 So that's part one.

12:53 You can also have like okay I answered this I've done that.

12:57 What's the next best action.

12:58 But you need to figure out what kind of tools

13:01 you use and what the boundaries are of those tools.

13:04 And that goes typically in the like best practices of a generic system.

13:08 Yeah.

13:09 Yeah.

13:09 Obviously.

13:09 Yeah.

13:09 So as we move to the close out,

13:12 let's take a look at these, these key takeaways from the discussion.

13:15 Right.

13:15 Um, we had several interesting things come to light.

13:18 So obviously AI brings many super interesting opportunities with WAF.

13:21 Um, we had a few discussions.

13:24 There's many more that are on the website.

13:27 Um, number two, the tools are great to improve your operation,

13:30 but you do need to carefully take the risks and the trade offs into account.

13:35 Um, and all of these risks should

13:37 be clearly understood and mitigated to where applicable.

13:40 Yes, I think I think that summarizes it well.

13:44 I the way I would also I would I would add

13:47 is when when it comes to how to evaluate this stuff,

13:50 it's just how you would evaluate any type of AI system.

13:53 So if you're building intelligent apps, right,

13:55 you also have inner loop testing of your agents.

13:58 Right.

13:58 Is this the right output I want?

14:00 Does it the right thing?

14:02 Does it do the right thing?

14:03 I think that's what you should think about when you leverage AI as part of.

14:08 Well, yeah, but absolutely, it's a great addition.

14:10 Um, I feel it's adding a lot of value and it's making the WAF practitioners,

14:16 so to say, really more productive, if you think about.

14:19 Right, is there a lot of time in producing papers and analysis?

14:23 Absolutely.

14:23 Okay.

14:24 Thanks, Boris.

14:24 You're very welcome.