The Joy of Unplugging Cables: Kelly Shortridge on Security Resilience

0:00 [music] [music] [music]

0:13 Hi, I'm Scott Hanselman.

0:14 This is another episode of Hanselminutes in association with the ACM Bytecast.

0:18 And today I have the honor of speaking with Kelly Shortridge.

0:22 She's a chief product officer at Fastly.

0:25 How's it going?

0:26 It's going very well.

0:27 It's a beautiful spring day here in New York.

0:30 It is beautiful spring day.

0:31 I've gotten some sunshine today and I feel a lot better.

0:34 Um everything sucks, but it just sucks slightly less when it's sunny.

0:39 That is true and things are blooming.

0:40 I can't complain.

0:42 Yeah, absolutely.

0:43 Um so you are the author of security chaos engineering,

0:46 sustaining resilience in software and systems.

0:48 And I spent the weekend uh reading the book and trying to understand where

0:53 the intersection of chaos engineering and security

0:56 engineering is because like I remember when

0:58 chaos monkey was a thing and I just got to imagine all the Netflix

1:02 people running around pulling cables and the monkey

1:04 was just messing up their stuff.

1:06 And now I'm trying to understand

1:07 the intersection of security engineering with chaos engineering.

1:10 I wonder if you could help me understand that.

1:13 Yes, I think it's better characterized

1:15 by the umbrella of resilience engineering if anything.

1:18 Um I've actually had the rare and delectable

1:21 pleasure of unplugging cables from Fastly's pops.

1:24 Of course, network continued working perfectly.

1:27 It is a thrill though, I will admit.

1:29 Um part of the title with the book is um

1:32 with a little behind the scenes tea is chaos engineering,

1:36 especially at the time, was a big buzzword.

1:38 The book is certainly more than just chaos engineering.

1:40 That is one tool in kind of the resilience engineering toolkit.

1:44 When we think about resilience,

1:45 it's ultimately about how do you recover from failure

1:48 of any kind and prepare for what's next?

1:51 That what's next could be a threat,

1:53 but equally it could be a business opportunity.

1:55 It could be, you know, massive traffic growth for good reasons or it's a DDoS.

2:00 Um so security really is a subset of the both surprises,

2:05 stressors, opportunities,

2:07 and threats in a very broad sense that we need to think about.

2:10 Um This might be a dumb question.

2:13 It could be a spicy question.

2:14 But like why call it security chaos engineering?

2:18 Is it because those are fun words?

2:19 Because like resilience engineering didn't wouldn't fly off the shelves?

2:23 Cuz it seems very clear that like resilience is really what we want,

2:27 but it's just not a sexy term.

2:29 I mean, I think this is a classic

2:31 tension always um when you're trying to publish,

2:34 you know, a book or frankly a movie, you know, you have to [laughter]

2:38 have some sort of catchy name.

2:39 No, that's a great point.

2:40 Chaos would be an awesome movie name,

2:42 but resilience is like uh it's more of an A24 movie.

2:46 Exactly.

2:46 A24 vibe.

2:47 It's also, you know, at you know, talked about at Davos and it's in, you know,

2:52 the National Association of Corporate

2:54 Directors book around organizational resilience.

2:57 Um it is a great Latin root word for international appeal.

3:00 But I think chaos, people are like, wait a second, chaos can be a good thing?

3:03 That doesn't sound right.

3:05 Absolutely.

3:06 I want to engineer chaos.

3:08 That'll be a very very exciting.

3:10 Now, you have said that uh security

3:12 should be designed for failure, not for prevention.

3:16 And I think that's a really cool way to think about that.

3:19 Um can you think of an example where there's

3:21 a perfectly secure system that still failed in the real world?

3:24 Like what's an example where oh, it still happened and we couldn't stop it?

3:28 I mean, I feel like tons.

3:30 First, no such thing as a perfectly secure system, right?

3:34 I think there's so many esoteric failures out there.

3:36 I think about um the airline industry has learned many years ahead of software

3:41 about the intricate nature of complex systems

3:44 and all the failures that could go wrong.

3:46 But the example I always think about is the fact

3:48 that they designed um I forget which airplane it was, which model.

3:52 They designed it with safety in mind to almost every

3:54 degree except for the fact that in a very bizarre scenario,

3:58 if you spilled or not spilled,

4:01 if you somehow exploded the coffee maker like boiling it too hot or something,

4:06 it happened to be close enough to like the panel with some cables

4:10 that it could cause a like critical failure while the plane was in air.

4:15 Oh my god.

4:16 Right.

4:17 You wouldn't think about that as like that is the trigger to like

4:20 a massive failure that means there has to be like an emergency landing,

4:24 but yet there they were.

4:25 Um so I think looking at real world systems

4:27 and all the just bizarre ways that they can fall apart.

4:30 I remember back in the days with Twitter,

4:33 there was cyber squirrel where it talked about all the power

4:36 plant failures caused by squirrels just doing things squirrels do,

4:40 you know, and how they were almost the more threatening

4:43 advanced persistent threat because of the damage that they wrought.

4:46 I think there's just so many examples of like your best intentions,

4:50 you know, reality is stranger than fiction.

4:52 You're not going to be able to dream up every scenario that's possible.

4:55 So you have to prepare for the idea of, okay, things will go wrong.

4:58 How do we minimize impact and make sure we can evolve to like meet the moment.

5:03 Yeah.

5:03 I think it's so important also to remember

5:05 that like and maybe this is also a little spicy.

5:08 Like it's on you to be responsible for your own resilience.

5:12 And I remember in the early days of the cloud when we were all

5:14 trying to get five nines out of Azure and five nines out of AWS,

5:18 it's like okay, Azure went down.

5:20 I'm going to call somebody and yell at them.

5:21 And it's just like, but how badly do you want your site to be up all the time?

5:25 Do you want it badly enough that you're going to put it in both Azure and AWS?

5:30 How much do you want to copy of like how do you make a plane that doesn't crash?

5:33 Do you fly two planes next to each other?

5:35 And then when one fails, like you jump to the other plane?

5:38 Like it is ultimately on us, is it not?

5:40 And we just need to decide how hard to squeeze.

5:43 I think there is usually a trade-off if you

5:46 want to really simplify it between cost and resilience.

5:49 Um to your point, you know,

5:50 ultimately redundancy is multiple paths to get to the same goal.

5:54 In practice, you need polyglot applications and systems.

5:58 That's pretty expensive to pull off.

6:00 Um I do think though that software has

6:03 kind of a beautiful luxury we sometimes don't leverage.

6:06 To your point about planes,

6:07 sometimes you can run two instances of a service to like

6:12 offload capacity in a way you just can't do with physical systems.

6:15 Same thing with simulating failures, too.

6:18 Again, I think that's a very responsible thing to do.

6:20 Um a lot of complex systems wish they could.

6:24 For instance, you can actually instantiate, you know,

6:27 a real kind of clone of the production system.

6:30 You can't replicate a realistic clone of New York City to see, you know,

6:35 if there's a certain level of trash blocking sewer drains,

6:38 like what level of flooding will cause like deaths.

6:41 Like you can't simulate that with any degree of ethics,

6:44 but you can in the computer world.

6:46 But we're not doing it.

6:47 So I think that's that's part of my call

6:49 to action that was big in the book is like, okay,

6:52 how do we start taking this more serious seriously and really

6:55 leveraging the benefits that the flexibility software begets um gives us.

7:00 So I do think there's to your point, yes,

7:03 some of it is more expensive to do, but in another sense, you know,

7:07 maybe we should be allocating more spend towards

7:10 some of that simulation or just understanding like

7:13 the resilience contours of our systems better when other

7:16 industries are just looking at us shaking as like, why aren't you doing this?

7:20 We wish we could do this.

7:22 Yeah.

7:23 I I've been tr- I've been thinking about resilience

7:24 in my own kind of personal IT uh life.

7:27 I assume you have a home lab and, you know, various sorts.

7:30 Um right now as I talk to you,

7:33 because I had an appointment with you, I am on my backup internet.

7:36 Uh turns out I'm looking at my my UniFi here.

7:38 My my WAN failed over at um at 4:38 a.m.

7:43 and I have yet to diagnose it.

7:45 So I'm on backup internet right now.

7:47 And um when I mentioned that to people, like like muggles,

7:50 like regular people, they're like, you have two internet at your house?

7:53 And I'm like, like this is my job, bro.

7:55 Like I'm here.

7:56 Like I got I've been doing this at this house for 18 years.

8:00 It cost me 45 bucks for Comcast as backup internet.

8:03 I have my fiber, but the backup for $45, it only has to fail once, like today,

8:10 and that made it worth the money for the year

8:13 because otherwise I would have had to cancel on you.

8:15 And that wouldn't happen.

8:17 Yeah, so it's like it was a choice.

8:19 And I feel like there [clears throat] are teams

8:21 that think that they are resilient until the thing happens.

8:24 Um what's the difference between a security team

8:26 that thinks they're resilient and maybe one that actually is?

8:29 Like I feel like there's a lot

8:30 of false confidence and metrics theater that happens.

8:34 Metrics theater.

8:34 That could be an episode in itself.

8:36 Um they're like, oh, what percentage security coverage do we have?

8:40 Nobody knows what that means.

8:41 It's a meaningless metric.

8:43 Um that is a great question.

8:46 I think the giveaway is when the security team feels a sense of control,

8:52 probably means that they don't have a lot of resilience.

8:55 Cuz part of resilience is embracing the fact that like

8:57 there will be things well outside of your control.

9:00 So it's like how do you prepare for that?

9:02 If you were trying to control everything

9:04 and make things as deterministic as possible,

9:06 you have already failed in my view.

9:09 Cuz the world is not deterministic.

9:10 Humans aren't deterministic.

9:12 We would like computers to be deterministic, but they aren't um fully at least.

9:16 Um it's one of the hardest problems in computer science is verifying

9:20 that the software works the way that the designer of the program intended it to.

9:24 Um so whenever I hear a security team say like, well,

9:27 we have full control over the software delivery life cycle,

9:30 I'm like, are you sure?

9:35 [laughter] I just saw a meme I just imagined a meme'd

9:37 version of you and that one guy from HBO is like, you sure about that?

9:41 You sure about that?

9:42 Yep.

9:43 Yeah, or are you sure you're doing the right thing to do?

9:45 Yeah, you're absolutely right.

9:47 Um probably means you're investing in things that make you feel good

9:50 and give you that sense of control and not the things that minimize impact.

9:54 Yep.

9:55 Like and that's not directly security,

9:56 but that old joke of like backups always succeed, it's restores that fail.

10:02 Mhm, yes.

10:03 So, it makes me think about like

10:04 chaos engineering and and infrastructure is about,

10:07 you know, pulling wires and yanking wires is very exciting.

10:10 But, I feel like there's a lot of pull

10:11 the wire moments that are that can happen in security,

10:14 but people are too scared to try.

10:17 Fear, I mean, fear is pervasive in the culture and it's

10:20 a disservice to the industry and the mission for sure.

10:23 I think there are also cases where you know, you could be starting with smaller

10:28 experiments or just testing more basic hypotheses.

10:32 My favorite leveraging um actually Fastly's compute,

10:35 which is kind of like a high-performance serverless,

10:37 you can think of it that way.

10:38 It's just a little function that strips out um cookies just to see like,

10:42 hey, does your login site work?

10:44 Same with like off headers.

10:46 It's just those basic [clears throat] assumptions you hold.

10:48 Like, of course, like we're always going to require this for the login page.

10:52 It's like, well, is that true?

10:54 Are you sure?

10:56 And the especially when you can like duplicate

10:59 the request um which this prototype did um you know,

11:03 it's pretty low impact to the business to run that experiment.

11:07 There are of course things where it's like,

11:08 hey, rmrf like the customer database.

11:11 Yeah, that's going to be a pretty

11:12 poorly designed experiment with high consequences.

11:15 But, there's like such a range in between um that I think

11:19 it's very unfortunate that security practitioners

11:21 are too hesitant to try those experiments,

11:24 especially they're very hesitant to reach out to their peers

11:26 across the island like platform engineering and be like,

11:29 hey, can we co-conspire on developing some of these experiments?

11:32 Cuz there are a lot of jointly held

11:34 assumptions too that aren't always poked and prodded.

11:37 You mentioned about uh determinism and how computers and software is

11:41 not as deterministic as we would love to think that it is.

11:44 Not just because, you know,

11:45 the software pretty much always runs as you wrote it,

11:48 but whether or not your intent was well expressed uh certainly is a problem.

11:52 And then the environment within which it runs, you can't always count on.

11:55 But, I'm finding that uh people seem to be spackling

11:59 or puttying over their systems now with what I'm calling ambiguity loops,

12:03 which are basically using an LLM to deal

12:05 with ambiguity by letting it fill the ambiguity with randomness.

12:11 And I'm curious in your business when now people

12:13 are like running playbooks that aren't scripts, they are prose.

12:19 Like a markdown file is not a script, I think you would agree.

12:22 Yeah.

12:23 Uh how do you feel about that?

12:25 Is there a place for LLMs to live in security and in resilience

12:29 and in chaos or do they just increase increase chaos and and entropy?

12:34 It depends.

12:35 I think you know, they're only going to be

12:40 as good as the corpus that went into them for one.

12:42 And so, unless you can verify like really clean code went into it,

12:46 it's like, well, you know, it can maybe be a good basis for actually

12:50 in some cases chaos experiments or you know,

12:53 specific configurations, integration tests, etc.

12:57 What I will say though um is to me

12:59 the more important litmus test is is this replacing human judgment?

13:04 And if it is, that's probably not a good case for an LLM.

13:08 Um I am pro human judgment and creativity.

13:11 I am pretty anti like rep like very repetitive work,

13:16 very tedious work where you don't need that kind of judgment call.

13:19 LLMs can be very helpful there.

13:21 Um I think, you know,

13:22 they're they're also document intelligence examples, you know,

13:25 who loves going through your compliance

13:27 documents and pulling out relevant information.

13:31 Like LLMs can shine there and that way you

13:32 can focus more on strategically like are we sustaining resilience?

13:37 Like, what are the indicators we should be looking at for that?

13:40 Um but asking LLM like, is our system resilient?

13:43 Probably is not going to be a great outcome.

13:45 Um I think the markdown example is a little

13:47 interesting cuz I am also pro making security more accessible.

13:52 I think we dress it up in a lot of arcane you know,

13:55 key phrases and buzzwords um when really a lot

13:59 of people could benefit the industry and contribute.

14:01 So, maybe simplifying how they can enter and not

14:04 requiring scripting knowledge could be a good thing.

14:06 I also see it as a potential foot gun.

14:08 So, I feel like I'm a little mixed.

14:10 No, I hear you.

14:11 I I'm with you 100% on the like human judgment.

14:13 Like, it cannot be overstated.

14:15 I'm I assume that you're speaking to universities and early

14:17 in career people often when you give your speeches and stuff.

14:20 And you talk to them and they're like, what should I learn?

14:22 It's like, you should learn how to have good taste.

14:24 How do you learn good taste?

14:25 Well, you just got to get in there and get your hands dirty and do the thing.

14:29 Start pulling wires and figure out the system.

14:32 Uh certainly I don't I don't want to outsource things

14:34 to to human judgment and toil like keeping a site up is toil.

14:39 SRE is toil.

14:41 But, SREs that are really good at their job

14:43 are good at their job because of their judgment.

14:46 So, there's going to be this constant

14:47 tension between like I don't think the idea

14:50 that you would replace an SRE with a markdown file makes me very nervous.

14:53 But, an SRE agent that could maybe kick the node and keep

14:56 it running while I drive over there has value to me.

15:01 Yes.

15:01 Buying capacity and buying time,

15:03 I think is a great use case as well to your point.

15:06 Um just like how do we help the human engage better with the system?

15:12 Um even just like the rubber duck problem solving,

15:15 that can be a useful thing for the LLM.

15:17 That does require to your point quite a bit of expertise already though.

15:21 I love that you brought that up.

15:22 I I was talking to someone recently and I gave a whole talk

15:24 and I started brought up rubber duck debugging and like no one got it.

15:28 And like, am I like am I onk suddenly?

15:30 Like, no one gets.

15:31 This is not a generational thing.

15:33 Like, talking to the duck.

15:35 Yeah, your face is saying the same thing.

15:36 Like like those are important moments to brain

15:40 you're talking to yourself in the mirror,

15:41 except now the mirror can talk back and that's really cool.

15:44 I find that to be super helpful.

15:45 Have you used LLMs in that context to like

15:48 figure your thoughts out and talk to yourself?

15:51 Sometimes um I use my cats more often for that.

15:55 Cuz they're they give especially like judgy looks that make you really question,

15:59 you know, what you're throwing down.

16:00 I think LLMs can be helpful though,

16:02 especially um I see a lot of people struggle to get buy-in.

16:06 This is kind of getting into, you know, corporate type stuff.

16:09 But, especially international companies where it's like, hey,

16:12 I want to get buy-in on this resilience initiative.

16:14 How is this going to resonate across different cultural contexts?

16:17 Ooh, that's a good one.

16:19 Right?

16:19 Like, is chaos perceived negatively in certain nations versus others?

16:23 I can tell you for instance when I talk about

16:25 deception and using that as a technique for resilience engineering,

16:28 security engineering, uh American security practitioners,

16:32 not universally, tend to result in like, well, we don't want to be the bad guys.

16:37 Mhm.

16:37 Now, in the EU, they're like, tell me more.

16:40 Yes, please.

16:42 Like, we want the Sutter fudge here.

16:44 Um so, that's kind of fascinating, right?

16:46 And I feel like that's an interesting like

16:49 twist um that I found really useful with LLMs.

16:53 I like that.

16:54 I I that I didn't think about that.

16:55 Yeah, you're right.

16:56 It does see broader than than we do and it can challenge your assumptions,

17:00 especially if you tell it challenge my assumptions

17:03 as opposed to telling you that you're absolutely right.

17:06 Um there's you probably work with like big companies

17:09 like our banks and slower moving things, health care.

17:12 They're a little more conservative.

17:13 I'm curious, does is there an example where traditional compliance actively

17:18 makes systems less secure where they think that they're checking boxes,

17:21 but they're actually hurting themselves?

17:23 Yes, actually a frequent co-conspirator of mine,

17:27 Josiah um Dykstra, wrote a paper, not with me.

17:31 Um it's an excellent paper um about that exact topic.

17:35 I think specifically uh covers HIPAA and maybe one of the others that shows

17:40 that it doesn't being more compliant

17:42 doesn't actually result in better security outcomes.

17:45 I'm very much of the view and I've tried to caution regulators

17:47 as well as like well-intentioned regulation

17:50 in this space very quickly calcifies and ossifies.

17:55 Like, it's what helped in year zero through maybe

17:59 even year three may end up actually eroding resilience long-term.

18:04 Great example, I'll keep the person anonymous,

18:06 very innovative CISO um had to explain I

18:10 think over a few years to his auditors like,

18:13 actually it's a great thing that we don't

18:15 allow SSH access anymore cuz that's what attackers love.

18:18 They love when you leave the door open like that.

18:21 But, on the little compliance checklist for the auditors, they're like,

18:24 okay, but it says you're required to have SSH access.

18:27 He's like, okay, but the security outcome is now better.

18:30 So, that's where sometimes they can actually hold

18:32 companies back either big companies who do want

18:36 to innovate um but basically tying them to their investments

18:40 and their spend to just checking those boxes,

18:43 which is a disservice to their overall mission.

18:46 Yeah, having I always want to assert assumptions.

18:49 I feel like at cuz I work at Microsoft in my day job that my ignorance is kind

18:53 of my superpower cuz someone will throw me into a new

18:55 situation and I'll see a checkbox like, must include SSH.

18:58 I'm like, but why?

18:59 Like and they don't they no one knows.

19:01 Like, I don't know, 13 years ago someone

19:02 wrote that checklist and now it's a thing.

19:05 And then investors see it and compliance people see it and that checkbox

19:08 is the thing that stands between you and some certificate or some badge.

19:13 And that's a problem.

19:15 It's a huge problem and actually bring up a kind of elegant point.

19:17 If you look at what resilience means across all sorts of complex systems,

19:21 but also the ones we're talking about here,

19:24 a lot of when a system is stuck in let's say, it's not elegant,

19:27 like a a less resilient or like unresilient state

19:31 or fragile state is because a lot of the processes

19:34 and practices that they have in place to your point

19:37 are from an equilibrium that no longer exists.

19:40 Right?

19:40 The status quo has moved on.

19:42 The practices haven't.

19:44 And so, you're just continuing to erode resilience as you like stick to this old

19:47 world and have not adapted to the new one and the new context.

19:51 It's the same with I always hear CISOs being like, "Well,

19:53 once we patch the vulnerability or fix it,

19:55 you know, then like it's fine." It's like,

19:57 "Well, if that actually resulted in an outage or a breach,

20:01 it's not actually fine because you

20:02 still haven't addressed the underlying impact.

20:04 You've just patched over the one way attackers got in." Um

20:07 they haven't adapted to that new paradigm and that new equilibrium.

20:11 Um which is hard.

20:12 It's updating your mental model of the system, which is not easy.

20:15 That's why it is so important to have people who will be like, "Well, why?

20:19 Why is that?" You know, just poke poke and prod.

20:22 Um often CISOs and security teams make dashboards cuz they

20:25 want to roll things up and the bigger the company, the bigger the dashboard.

20:28 And then the CISO has to really they don't they can't know the entire stack.

20:32 The stack is now too deep.

20:34 So, what is a an example of a misleading security

20:36 metric or dashboard that might cause someone to make a mistake?

20:40 They're relying on a dashboard, but it's uh maybe a misleading metric.

20:45 So many metrics.

20:46 Certainly that security coverage one or risk coverage.

20:50 Um also the number of vulnerabilities discovered.

20:55 Um I'm trying to remember who it was who talked

21:00 about this where actually when things started to get better,

21:03 it meant that their application development

21:05 teams were surfacing more security issues, which was a good thing.

21:09 And it meant that there was more of that trust mutual trust between teams,

21:12 but it looked like it was getting worse.

21:13 Yeah, see that's a great example.

21:15 That's the whole thing like, you know, "Oh my god,

21:17 all these bugs and all these security issues." That's this good stuff.

21:19 All of that is low-hanging fruit,

21:21 but like they'll assume that something bad has happened

21:24 or something has changed and they're going to then you know,

21:27 correlation and causation are not not not the same.

21:31 Exactly.

21:31 I think and also a lot of the security

21:34 specific metrics don't tell the bigger picture.

21:37 Includes I think about, you know, the poor platform engineering teams who are

21:40 handed a list of a thousand vulnerabilities.

21:42 Turns out a lot of them are in components

21:43 that aren't even exposed to the public internet.

21:45 Should they prioritize those?

21:47 Probably not.

21:48 And meanwhile, in actually there are multiple cases

21:51 of this, so I'll keep them all anonymous,

21:54 but it's surprising actually often this happens.

21:56 Security team will be on them and be like,

21:58 "Fix all of these even if they're not publicly

22:00 exposed." And then the security team actually maintains, you know,

22:03 their creds into their, you know,

22:05 like whatever admin system or security system that has its hooks

22:09 into everything is like in a text file on their desktop.

22:12 It's like, "Well, what do you think is actually the bigger

22:14 issue here in terms of what attackers could leverage?" So,

22:17 there's a lot of that kind of attacker

22:19 math attacker calculus that isn't baked in as well.

22:22 Um And I think there's also even if we think about business context,

22:28 the metrics that a lot of security teams track and even CISOs track aren't

22:32 the ones that the board wants to understand

22:35 or other executives need to understand either.

22:37 Yeah.

22:38 Um if I go to fastly.com and I click on products,

22:41 you've got all the network services and all the things that Fastly's known for.

22:44 There's a whole section on security and there's also, you know,

22:47 you have services and folks that you

22:49 can hire professional services and things like that.

22:51 But what how should I think about what

22:54 security is my responsibility and what is the responsibility

22:58 of the vendor for whom I am paying a lot of money to make things secure?

23:04 I think it depends on the vendor.

23:06 I think in the case of let's say like some sort of SaaS application,

23:10 um let's take it sales and marketing,

23:13 so it's going to have some of your customer data, prospect data.

23:16 Feels reasonable for the most part that like the encryption

23:18 and things like that should be handled by the vendor for sure.

23:22 There are cases I'll use like Fastly where we have

23:25 the platform where you can basically write code, run code, etc.

23:28 Um It's our responsibility and we have done

23:32 this to layer in like memory safety by design.

23:35 Um same with like isolation models, ensuring like safe multi-tenancy.

23:40 It's very much our responsibility.

23:41 Um Making sure that like you don't write

23:45 vulnerabilities into your own code, it's like, "Well,

23:48 that's probably outside of our remit." Um though

23:51 that's where sometimes pro serve can come in.

23:54 Um things like, "Hey,

23:55 you have spun up like a service on Fastly and that connects

23:58 to a database that you have wide open without any ACL." Sorry,

24:02 access control list.

24:03 Um Not really our responsibility cuz we don't touch that component, right?

24:07 Um there are ways that we can help with middleware that runs on our platform.

24:11 Um I do think though there's a fundamental principle

24:16 though in the conversation a lot of people miss,

24:18 which is like you have to own your own dependencies.

24:21 Um and so what you adopt, you do have to basically think about it like,

24:25 "Well, we have to assume at some point something will go wrong with it,

24:29 whether that's a security issue or not." And I do see

24:31 a lot of the like hot potato game happening out there.

24:35 Yeah.

24:35 That's exactly why I asked you that question because I

24:37 think that people pay a lot of money for a platform,

24:41 a cloud platform because they want, as they say, a throat to choke.

24:45 So, like who gets yelled at, right?

24:47 You know, I just get Kelly on the phone.

24:48 I want to know what's going on over there.

24:50 You know, but then it's of course their thing.

24:52 Then they are bringing in, you know, unknown uh who knows,

24:56 unknown node packages from unknown provenance and then they

24:59 haven't think about thought about their entire secure supply chain.

25:03 But at the same time, like I I like your point about

25:05 a a sales and marketing CRM or something like that.

25:08 Is it their job as an app to be in charge

25:10 of like AI bot management or DDoS or even API security?

25:14 Like that would be an example where a cloud

25:16 platform could secure those endpoints and and hide that.

25:21 So, I like the separation of concerns there.

25:24 But I I wonder if people who are

25:26 putting together their own systems think about that.

25:29 Like, "We shouldn't be in charge of API security.

25:31 Let Fastly secure our endpoints." And then

25:34 they just have a nice clean bright line.

25:36 Or is it always layered and they would

25:38 have to do that basically have two layers.

25:43 I think it it really depends on the company and their level of resourcing.

25:46 There are some companies that culturally want to own where things are built more

25:51 of their things and so they'll leverage us more to be able to DIY.

25:55 There's certainly others um where it's like, "Well,

25:58 let's just use what Fastly has, right?" Especially when it comes to security.

26:02 Um Putting in whether it's our WAF or like you

26:05 said the AI bot or like insights we're able to surface.

26:08 Like have that in front of our services, apps, sites, whatever it is.

26:12 Um It really depends on resourcing.

26:15 There's also the element of I'm going to say in newer worlds where I

26:20 have seen so many security leaders thrust

26:22 into a conversation now where their CEO, their CMO, their board is like, "Hey,

26:27 what AI bots are actually trying to scrape

26:29 our stuff so we can monetize it?" The CISO's like,

26:32 "I've never had to think about this before." Trying to DIY that is pretty hard

26:36 and having that expertise is some

26:38 of the most expensive expertise out there right now.

26:40 Using a tool probably makes sense in that case.

26:44 Yeah, I think that's a great point.

26:45 I mean, this is this is the thing.

26:47 What do we do here at the company?

26:49 We do insurance.

26:50 Okay, then why are we doing AI bot management?

26:53 Right.

26:54 not that's not our job, you know.

26:56 So, I I always think about the business and I I think sometimes when

26:59 we are talking about all the things that we've been talking about on this show,

27:02 we don't talk about like, "Why did we actually make this software?"

27:05 We made it to solve X business problem.

27:08 Therefore, what responsibility is mine and what can be

27:11 outsourced by someone who actually knows like what they're doing,

27:14 whether it be Fastly or Azure AWS.

27:15 Let somebody who actually cares about that do

27:19 that while I focus on the business problem.

27:21 Cuz I don't want to do Honestly, I don't want to do the stuff Fastly does.

27:24 That's why Fastly's good at it, you know what I mean?

27:27 Yes, and it's uh laying out your own pop infrastructure,

27:30 especially uh in this day and age of RAM prices being what they are,

27:35 especially if you were a small business,

27:37 it's quite unlikely that you're going to be able to do that.

27:40 Um Yeah.

27:41 Nor should you cuz to your point, it's not your core business.

27:43 I think it's I was thinking when you were talking

27:46 about the example of the um duplicate or secondary internet.

27:50 Um Part of that is because your essentially critical function

27:54 hosting this podcast is to make sure you can record it.

27:57 However, if you were to build your own microphone, I'd be a little bit like,

28:00 "Is that actually your core value add here?" You know?

28:06 No, that's a good example.

28:07 Right?

28:08 It's just understanding like what matters and what

28:10 makes you unique as a business, for sure.

28:13 Yeah, absolutely.

28:14 This is totally random and off topic, but uh I we had a fishing thing happen

28:20 at work yesterday where they send us fishing emails, but it's from the red team.

28:24 And I was so proud of myself.

28:26 I was just like, "I don't think that's real." And I was like,

28:29 "Report fishing." And then it was like, "Congratulations.

28:32 You are, you know, you're you're one of the better people." I don't know.

28:35 There's some number of people at the company that did that does that.

28:38 And I'm always impressed that there's a whole teams out

28:41 there trying to attack us internally that I've never even met.

28:45 You know what I mean?

28:46 The the red hats or the I guess they call them

28:47 blue hats at at at Microsoft cuz our badges are blue.

28:52 So, um I was just somehow I was just

28:54 thinking about there's people in trying to create chaos

28:57 internally at the company and they tried to catch

28:59 me yesterday with a fish and I didn't fall for

29:02 I will say, that has gone wrong in the past and I

29:05 have spoken publicly before it was cool to have this take.

29:08 People were quite angry.

29:09 I remember this is many years ago where I said, "Hey,

29:12 it's maybe not a great thing." Especially during COVID this happened a lot.

29:16 Oh.

29:17 Here's your surprise bonus plan.

29:19 And that's the fishing like simulation Oh no, that would be awful.

29:25 Right, but that was happening.

29:27 That's mean.

29:27 That's just that's kind of punitive.

29:29 I agree.

29:30 Right, click here for more money.

29:31 No, this was not that.

29:33 This was more like, "You have mail waiting

29:35 for you in the mail room." And I'm like, "We don't have a mail room." You know?

29:38 There you go.

29:39 Yeah.

29:41 That's fair.

29:41 You're right.

29:42 I mean, this is the whole like sprinkling um USB keys

29:45 around the bank parking lot It's of way of doing things.

29:48 It's like Beyonce's new album, sprinkle sprinkle,

29:51 and then everyone plugs it in and then owns the entire bank.

29:54 Yeah, something like that.

29:55 I think there are a lot of experiments.

29:57 I think it's always keeping in mind again

29:59 the human element that you don't want to sow distrust.

30:02 Um but again, you can also make the experiments collaborative Mhm.

30:07 Um which is which that can get

30:09 really fun because I've actually You mentioned SREs.

30:12 When I talk to like real attackers, Mhm.

30:15 they're generally not scared of security engineering teams.

30:17 They're scared of SREs cuz SREs will like obsess performance.

30:21 There's that one backdoor, right?

30:22 Was it XEU tills where it was a guy who's like oh,

30:25 performance degraded by I think it was less than 1%.

30:27 What is going on?

30:28 Right.

30:29 Discovered the backdoor.

30:30 Right.

30:31 That was awesome, actually.

30:32 That was pretty cool.

30:34 Right?

30:35 So, I think security teams need to embrace like,

30:37 hey, you may you will have good ideas,

30:39 but like there going to be other very clever people where if you say,

30:42 okay, if you got really mad at the company, how would you attack us?

30:46 They're probably going to have some

30:47 interesting ideas that maybe can become experiments

30:50 or clue you into some gaps maybe you have in your current security investments.

30:57 [snorts] Very cool.

30:57 This is You've given me a lot to think about.

30:59 Um I you know, I I I thought that chaos engineering and security

31:04 chaos engineering and this kind of resilience

31:06 was uh kind of a branding exercise,

31:08 but it feels more concrete after having chatted with you.

31:13 Yes, I mean, again, it was mostly a buzzword and it's

31:16 part of playing the game um that publishers have to play.

31:19 I will say though for a very long time since I was the wee lad as they say,

31:24 uh I've been obsessed with chaos theory.

31:26 Mhm.

31:27 Um and I do think chaos theory,

31:29 which is quite beautiful in the sense of like systems do have an order to them,

31:34 but it's not necessarily predictable.

31:36 It's more like mostly like a fractal or dragon

31:38 curve in many cases as any meteorologist knows well.

31:42 So, we need to focus less on again, do we have control over it?

31:45 Are we able to predict it?

31:46 And more, you know, the quote I love is

31:49 from Susan Elizabeth Howe who's a geologist who said,

31:53 "A building doesn't care whether the earthquake was predicted or not.

31:55 It either stays up or it doesn't." That's good.

31:58 That's very good.

31:59 It's really good.

32:00 That's very good.

32:01 I feel like that's the essence of it.

32:03 Yeah.

32:03 I like that one.

32:05 One of my favorites in the in a similar

32:06 vein is a Babylon 5 uh the avalanche has begun.

32:09 It's too late for the pebbles to vote.

32:12 That is also very good.

32:13 Yes.

32:14 Yes.

32:15 I don't like this.

32:15 This is not a good idea.

32:16 This is happening.

32:16 Sorry, this is happening.

32:16 So, buckle up.

32:17 Well, thank you so much, Kelly Shortridge, for chatting with me today.

32:19 Thank you for the great questions.

32:21 Exactly.

32:21 Well, thank you so much, Kelly Shortridge, for chatting with me today.

32:24 Thank you for the great questions.

32:25 Appreciate it.

32:26 We have been chatting with Kelly Shortridge,

32:28 the chief product officer at Fastly.

32:30 This has been another episode of Hanselminutes

32:32 in association with the ACM Bytecast, and we'll [music] see you again next week.

32:41 [music] [music]