Where Are We Now? Bias in Health AI

0:22 I 100% agree that when you're just sending yourself to OpenAI,

0:26 you are giving over control of something

0:28 that you've developed over to a company.

0:30 I will say, though, that as a health system,

0:34 you have to separate out is the functionality of summary useful

0:39 from the actual control that you're giving up to the company.

0:43 I would argue that for a lot of things that we're using Generative

0:47 AI for I'll give you some examples towards the end of this talk,

0:49 the functionality is increasingly useful.

0:52 One of the things that I think health systems are trying to do,

0:55 including ours, is to have contracts in place with certain companies.

1:00 That might be Microsoft,

1:01 it might be other ones so that we know that if you put data into that mechanism,

1:07 we have a contract that forbids that company from doing a lot of the things

1:11 that would happen if you were to do that on a private account with them.

1:14 Just to say that that's a an applied person's dilemma.

1:20 It's useful.

1:21 If you don't give a appropriate path, everything will be an illicit path,

1:26 and you lose a lot of control on the illicit path,

1:28 sometimes there's an appropriate pathway to go.

1:31 But I'll come back to that in a little bit.

1:33 Hilariously, my one disclosure is I serve

1:36 on the Consumer Health Advisory Panel for Google.

1:38 But I'm actually going to I only have one slide on Google,

1:40 where I actually criticize Google, so I think this is fair.

1:45 With that out of the way,

1:47 what's the actual issue that we're talking about when we're talking about bias?

1:51 In my talk, I'm going to talk very applied,

1:53 talk about give an overview of some of the ways in which

1:56 bias manifests linked to specific papers and concepts in healthcare delivery.

2:01 Then I'll come back to the role of generative

2:03 AI in some of the ways in which I want

2:05 to at least end with ways that we are trying

2:08 to address certain biases in both predictive and generative AI,

2:13 closing with some open controversies,

2:15 some of which doctor Noble talked about really nicely, and some of them,

2:20 which are hot off the presses,

2:21 changing policy that make us re look at the way that care delivery is happening.

2:26 I think the central thesis to a lot of the initial concerns around

2:33 bias in AI come down to the fact that we have constrained resources.

2:38 One of the ways in which people have said, Well,

2:41 if you've got constrained resources and you

2:43 can't deliver certain internions to everyone,

2:46 an equitable way to allocate those resources

2:49 would be to allocate them based on risk.

2:52 That's at least been a working model.

2:55 Even in the statistics, prediction modeling literature, that's a working idea.

2:59 It's a combination of people's preferences and people's risk where

3:03 you use to try to figure out who to intervene on.

3:06 You have a high risk group of patients who

3:08 are not going to show up for their procedure.

3:10 You try to call the highest risk ones to give them extra support.

3:15 I think that viewed from that lens,

3:16 the idea is that you've got maybe 20 people up here,

3:20 and what you're doing is you're sorting them by risk,

3:22 and then at the very high end of you're

3:24 trying to intervene on those patients primarily in recognition

3:28 of the fact that you've got constrained resources and you

3:31 can't provide high touch care to everyone in an equal fashion,

3:35 so you try to provide it in an equitable fashion.

3:37 This, unfortunately,

3:39 is essentially the cause of a lot of the problems of predictive AI.

3:44 We know that predictive AI is biased,

3:47 and there's this great overview paper by Herne Seres and John Gutag that talk

3:51 about all the ways along the modeling life cycle that bias gets introduced.

3:56 There's historical bias, representation bias.

4:00 When you start applying a model and start measuring performance,

4:02 there's a measurement bias.

4:04 Who are you measuring it on?

4:05 Then you do your training, you get benchmarks, you do evaluation,

4:09 and there's an evaluation bias related to the evaluation process.

4:13 Then you aggregate evaluations over populations.

4:16 There's an aggregation bias.

4:18 Something can look better on average, but worse for specific sub-populations.

4:23 Then depending on what you do with the model,

4:24 there's a consequentialist approach of taking a model,

4:28 and now you're doing something with it,

4:29 and now you could introduce bias in the fact

4:31 that certain people get something and certain people don't.

4:34 I want to start with just grounding everyone and from a healthcare lens,

4:38 what are these actual biases?

4:41 Not walking through all of them,

4:42 but I want to walk through a bunch of these just to make these very real,

4:45 mathematical and grasp you can grasp them

4:48 that you know what do they actually mean?

4:50 I think the one non health example I have actually is historical bias.

4:55 We know that there is bias in the world,

4:56 and that's because you look at any embedding model.

4:59 These are models that turn concepts, text, phrases into numbers.

5:05 You can use those numbers, compare those numbers against other sets

5:09 of numbers around occupation or around other concepts.

5:13 You can see how for the concept of Hispanic and Google News embedding,

5:18 it shows up with a very different

5:20 set of occupations than Asian white, for example.

5:25 Now, this is not necessarily a problem with the AI.

5:29 The AI is correctly seeing what is coming out

5:31 on Google News and able to make associations between them.

5:34 But you can see that if this is

5:36 the working understanding of the world embedded in these numbers,

5:38 then that's going to impact everything downstream that you do with those numbers

5:42 when you try to actually apply them and use them in various modeling processes.

5:47 Representation bias is I view it as a variation on sampling bias or maybe

5:55 just small subgroups of people where

5:58 the inadequate representation results in worse models.

6:02 Even if you had a model with everyone in the world in it,

6:05 when you try to apply it in different places,

6:07 there are going to be differences in patients, practice patterns,

6:11 technology that will result in that model not generalizing well.

6:15 In the medical literature and the ML literature,

6:18 sometimes people call this dataset shift.

6:20 Sometimes people call it a failure to generalize, but similar concept.

6:24 But if you then look at who we actually train models on, it's not on everyone.

6:28 It's on a subset of people often very US heavy or European heavy,

6:33 or it's people who come to our health system.

6:35 As a tertiary quarternary, care center,

6:38 there's a lot of folks who don't make it all the way to our health system.

6:42 I think the interesting change that I've seen happen is that it used to be

6:47 the case that we would train models

6:49 on as broad of a swath of patients as possible.

6:52 What I'm seeing actually now is a lot of vendors,

6:54 they do do some of that initially, but then they actually do local fine tuning,

6:57 so where they train and update models locally, it doesn't solve issues,

7:03 especially of subgroups,

7:04 but it's just some of the ways that these things manifest.

7:07 A really prominent example of this was actually

7:09 an example from Deep Mind, where in 2019,

7:12 Deep Mind team put together a model that predicted

7:15 acute kidney injury using data from the VA.

7:18 Actually, doctor Noble, you're alluding to the fact that they actually were

7:21 able to somehow get this data on their servers, which is highly unusual.

7:26 I know that because we, at the same time that they published that paper,

7:31 had a grant from the VA to do exactly what Google had just published,

7:35 which was an interesting place to be in.

7:37 I'll share just in a moment, what we found.

7:40 But even in that result, they had a sensitivity analysis where they found

7:43 that the VA is a 94% male population.

7:49 The model when you actually applied it to a held

7:52 out set of patients at the VA nationally,

7:54 the sensitivity of the model was lower in women as compared to men.

7:58 Some of that's biologic,

7:59 some of that is training data, some of that's representation.

8:02 There's actually a complex slew of factors that contribute

8:06 to that, which is to say that it's well recognized.

8:08 You've got representation bias that is

8:10 going to result in biased model performance.

8:12 Then when you try to use that model, what are you really doing?

8:15 Are people actually getting the equitable care?

8:17 Probably the most famous example of people not getting equitable

8:21 care allocation is this paper by Ziad Obermeyer in health, faculty at Berkeley,

8:27 worked with Sandal Millenn at U Chicago, and they published this paper,

8:34 very famous now, I think it was 2019, 2020, in science that showed that if you

8:39 model people's risk based on their utilization

8:42 of the healthcare system as a proxy for their severity of illness,

8:47 then you're actually going to overweight people with good access

8:49 to care and underweight people with bad access to care.

8:53 I think now I think we'd put it in the category of measurement bias,

8:57 but I think that, there's a couple of different ways to view that.

9:00 Just to recognize that, yes, you have predictive AI,

9:03 trying to do the right thing,

9:05 give people resources are at the highest end of the risk spectrum,

9:10 but if the highest end of the risk spectrum is also linked

9:13 to their financial and socio economic status to be able to get the care,

9:18 then you can inadvertently make that care actually

9:21 further in equitable than what you would have imagined.

9:24 Measurement bias also means that not all measurements are taken equally.

9:29 This is a urine albumin cratin ratio, which is something that we try to do

9:34 to screen patients for early kidney disease who have diabetes.

9:38 The rates of measurement are not the same across populations.

9:42 If you're going to build an AI model

9:44 that is powered by pulling data from the EHR,

9:48 we know that not everyone gets the same level

9:50 of care or the same level of access to care.

9:54 I would view this as the different levels of access

9:57 to care results in different levels of data quality,

10:01 which downstream mean that those models may not perform well

10:04 in certain populations that have lower rates of data availability.

10:10 Learning bias relates to the fact that we often skip over the fact

10:14 that algorithms actually themselves can be biased

10:18 depending on certain decisions that you make.

10:19 I think I say we skip past that because

10:21 maybe there's a set of population who thinks, well math can't be biased.

10:25 Of course, the math can be biased when we talk about risk.

10:28 Some of that is due to upstream bias, but a lot of that actually,

10:31 there are aspects of modeling decisions that can contribute to bias.

10:35 Like some basic really easy examples

10:37 I give are lumping demographic groups together

10:40 into other that's a thing you statistically

10:44 sometimes have to do because you don't have

10:46 actually enough sample to where you can

10:49 actually model that variable in any effective

10:51 way when it's higher cardinality than what

10:54 you actually have available in your data.

10:56 But that can result in two groups that are very different being lumped together.

11:00 Essentially becomes meaningless.

11:02 Another interesting thing is this idea that in machine learning,

11:07 you can either have a model that's too flexible and overfits

11:11 your data or not flexible enough and underfits your data.

11:15 A lot of the recommendations to prevent overfitting,

11:18 overfitting, sounds like a good thing.

11:20 The problem the reason it's not a good thing is

11:22 it makes it look like your model is doing really,

11:25 really, really well when, in fact,

11:27 you're making it worse and not generalize beyond the dataset that you have.

11:31 There's a nice visual if you Google overfitting like memes,

11:35 where it's like a bed that's shaped exactly

11:36 in the shape of someone who's lying down.

11:38 That's great, but the moment that they turn, they're going to fall off.

11:42 It's same idea.

11:43 You perfectly fit your data.

11:45 But actually, that's not what you're trying to learn.

11:47 You're trying to learn a relationship.

11:48 One of the things that we do

11:50 to prevent overfitting is we do penalization, regularization.

11:53 There's all these phrases people use to basically simplify

11:56 your model down a little bit to prevent it from overfitting.

11:59 But unfortunately, when you simplify your model down,

12:01 anything that you've learned that potentially might

12:04 have been useful only for specific groups

12:06 that are maybe under representing your dataset

12:09 that will result in what we call model compression,

12:13 where basically that different treatment,

12:15 which might have been an important thing to capture now goes away,

12:18 and so your model performs better in aggregate,

12:20 but actually does worse for specific sub populations.

12:25 Evaluation bias is just that your evaluation

12:28 data may not reflect the target population.

12:31 I shared this example of Deep Mind, where they you know,

12:36 in 2019, I think published this paper in nature, not nature something,

12:41 but nature nature that show they could predict acute kidney injury accurately.

12:47 They said this is the new way that you should do things.

12:51 We really hope someone goes out there, validates our tool,

12:54 starts using it to save lives and improve kidney injury.

12:57 Unfortunately, they didn't share any data.

12:59 Unfortunately, they also didn't share their model.

13:01 That was a very aspirational thing to say.

13:04 But because we had a grant from the VA to do

13:07 exactly this, we stopped what we were doing and said,

13:09 Let's try to recreate their paper on our like VA servers,

13:14 which are not quite the same workhorse of what Google has access to.

13:19 We had to make some modeling decisions.

13:21 But when we tried to replicate this paper, we found that in fact,

13:26 the situation they alluded to was just in one

13:29 aspect where the model did better in males than females.

13:33 We found that that was actually true across the board.

13:36 Very interestingly, you take that model outside

13:39 the VA and apply it to a different population,

13:41 which we were only able to do because

13:44 we fully re-created their modeling pipeline at the VA.

13:48 We found that it actually performed even worse in women outside the VA.

13:53 Interestingly, when you fine-tune outside

13:56 the VA on a gender balanced population,

13:59 you can actually fix some of those problems.

14:00 We actually got into the fact that you can sometimes take these models,

14:04 fine-tune them, and fix some of these issues.

14:06 But the reality is if you just don't know when you're using it off the shelf

14:10 you may run into this issue where

14:12 your evaluation data doesn't reflect the target population.

14:14 It's papers like this and many similar other ones with similar findings

14:18 that have resulted in basically local fine tuning now becoming almost a norm,

14:23 even for big EHR vendors who you would think of as not thinking about AI,

14:26 but now they do this as a best practice.

14:29 Deployment bias is mismatch between the problem that you think a model

14:33 is solving and the problem it's actually solving or the way it's used.

14:37 We published some work on the epic sepsis model that showed

14:40 that the model wasn't that great at predicting sepsis before it happens,

14:44 was very good at identifying sepsis 3 hours after the fact.

14:48 That's actually not necessarily terrible,

14:51 knowing that someone has sepsis shortly after they have

14:54 it is a point at which you can intervene.

14:57 But the model card for this said the point

14:59 of the model was to predict sepsis before it happens.

15:03 What the model car didn't disclose and which

15:06 we found out after we published this first paper

15:09 is that actually ten of the 80 predictors

15:11 in that model were actually Broad Spectrum IV antibiotics.

15:16 What was happening is people were thinking that someone might have sepsis,

15:20 starting antibiotics.

15:21 The model score goes up because they started antibiotics.

15:24 The model sends you an alert and says,

15:25 You should really think about starting antibiotics.

15:28 The follow up to this, we did a follow

15:31 up investigation with my colleague Jenna Wens at Michigan,

15:34 where we said, What if we look

15:36 at the score before you actually start antibiotics?

15:39 In that case, it was literally no better than a coin flip.

15:43 I think that there was this interesting thing here,

15:46 there's a causal relationship between your clinical recognition of something,

15:50 what it thinks you're doing and what it's trying to recommend.

15:54 This is something I tried to share with this HR vendor,

15:57 a couple of months before our paper came out,

15:59 and I think in the new version of this model that's live today,

16:02 they have taken out all those antibiotics from the model.

16:05 They have locally fine tuned it, which is another issue that we

16:08 found with model generalizability across different centers.

16:11 We have some work under review right now

16:14 looking at the new model versus the original model.

16:17 But things where I think they're really pragmatic things you can

16:19 look at, but I think this is an example of deployment bias.

16:22 High level, what can we do to reduce bias?

16:25 I think we have to recognize our implicit biases.

16:28 I think that's an important thing to do is to talk honestly about what are

16:31 you worried about when you're deploying something

16:33 at our Health AAC committee, as Camille knows,

16:36 we often talk about what are you worried about,

16:38 what are you doing to try to avoid it and try to make

16:40 that concept very real and measurable to the extent that we can.

16:45 We start with the purpose of the model.

16:47 What are you trying to achieve?

16:48 What are you trying to actually fix?

16:49 We'll worry about what the models predicting,

16:51 but what is the problem that's trying to be fixed?

16:53 One of the first questions that we often ask at our AIT shop is,

16:57 why are you trying to solve this with AI?

16:59 Is there not another way to do this problem?

17:01 Usually, you have to be able to map it to either predictive generative AI thing

17:05 very cleanly for me to really buy

17:07 that we actually need AI to solve that problem.

17:09 Making sure your sample size is enriched.

17:11 It's easy to say very hard to do.

17:13 This is more of a conversation we have with our vendors.

17:16 This is not just simply synthetic oversampling

17:18 where you mathematically over represent certain populations.

17:22 There is a paper on SMOTE resampling

17:25 that has been multiply debunked that we've written, other folks have written,

17:30 but it's still a very common practice taught in data science classes.

17:33 That's a hill to die on that'll have

17:35 a separate conversation if you apply these resampling things.

17:40 The reality is they don't actually do what they're supposed

17:43 to, and there's a lot of follow-up papers I can point you to.

17:46 Carefully consider the outcome.

17:48 Is the right outcome.

17:48 That was the key finding with iat Ormeyer's work is it wasn't the right outcome,

17:52 and so everything downstream of that outcome was problematic.

17:57 Carefully consider the predictors.

17:59 What are you including?

18:00 What are you excluding?

18:01 Do those things mean the same thing for all groups?

18:03 Don't only make modeling decisions based on aggregate performance.

18:07 I think when we think about predictive AI, some of the key things.

18:10 In the rest that talk I talk about generative AI,

18:14 just fundamentally, what are we worried about?

18:16 I'll talk about some of the things we're doing

18:18 because I think generative AI is not just a technology.

18:21 It's something that actually enables new models of care

18:24 that I want to just introduce you to.

18:26 Then I'll close with two slides on just some open controversies,

18:29 touching on some of the same ones as Dr.

18:31 Noble, but some other ones,

18:32 I think that you have talked him on previously, but maybe not today.

18:37 What is generative AI?

18:38 I think that one of the things I hear,

18:41 which is not correct is it's just predicting the next word using prior words.

18:46 That's technically true,

18:47 except they're trained in a two phase like there's two Step training process.

18:53 In the first Step, they steal everything off the Internet,

18:56 and they do just predict the next word.

18:57 That's Step 1.

18:59 Step 2, though, and they have datasets like Books

19:02 1 and Books 2 that you just don't want

19:03 to look up and see what's actually in those, which

19:06 is where all these lawsuits are coming from.

19:09 Then there's a step 2,

19:10 where they actually take that base model and train it on instructions,

19:16 and answers to those instructions.

19:19 Those could be questions.

19:20 Those could be tasks.

19:21 That is something that there's a lot of different ways to do it.

19:24 Some of the early work was reinforcement learning with human feedback.

19:29 There are supervised training datasets.

19:31 There are literally datasets that are like question, answer.

19:35 You find tune the language model on that such

19:38 that it's no longer just producing the next word.

19:41 It's actually now trying to follow that instruction,

19:44 follow in air quotes because that's just a part of the training procedure.

19:48 Both of these steps can obviously introduce bias.

19:51 I think instruction tuning is where probably most of the bias that we

19:55 think about I think comes from when you think about M dashes,

19:58 a lot of those things.

20:00 Sure, some of that's in the training data.

20:01 A lot of that is in who's

20:03 actually doing that reinforceable learning with human feedback

20:05 from a labor standpoint that's producing some

20:07 of these idiosyncrasies about the way that these tools perform.

20:13 When we're thinking about healthcare using

20:15 generative AI we often teach prompt frameworks as a way to get these tools

20:19 to actually follow instructions in a reproducible way.

20:22 But, As part of the prompt essentially is

20:27 yet another vector by which you can introduce bias.

20:31 The language models could be bias.

20:33 Some of those biases,

20:34 you can actually almost override with instructions in the prompt.

20:37 I don't want you to do x, y, z, I want you to do this instead.

20:41 Now, they're not always adherent to that, and some of the more complex

20:45 models tend to be more adherent than some of the smaller simpler models,

20:49 which is one issue, I think, with using just the small models.

20:52 But this is something that we have to be aware of.

20:55 When you interface with these tools in your personal capacity,

20:59 and then they do a web search or they do other things,

21:01 a lot of that essentially is copy pasted into the prompt.

21:04 You may not have put it in the prompt,

21:06 but you have to realize that when it's grounded on some external data source,

21:10 whether that's a medical knowledge base, whether that's the web,

21:13 that actually is entering your prompt unbeknownst to you,

21:16 because behind the scenes,

21:18 that's the way the tech works is it gets inserted into the prompt

21:20 and then sent off to the server to come back with a recommended output.

21:25 Now, this seems scary, and so far, it shows you a lot of negative things,

21:29 but I want to highlight some of the ways in which

21:32 this is really low hanging fruit that actually helps us in measurable ways.

21:36 One way is actually in some of the Dr.

21:39 Aaron Bina's work in measuring Step hospital quality.

21:43 What's the normal process for measuring hospital quality for sepsis?

21:47 It's like a 40 plus question questionnaire that requires hours long

21:52 chart review where you're trying to look at, did this happen?

21:55 Did that happen?

21:56 Did this happen?

21:56 What about this and this and this?

21:58 It can take a long time.

22:00 Actually, there's very little feedback look that happens.

22:03 You find out something didn't get treated appropriately.

22:07 There's almost no feedback that happens to the clinician.

22:10 Because it takes so long to do this and the labor costs are so expensive

22:15 to do this, it's about 30,000 per metric

22:18 as estimated by a Hopkins study per year.

22:22 In labor costs.

22:24 Essentially, we relegate ourselves to only reviewing 20 cases per month,

22:29 every health system.

22:30 You basically do a statistically invalid exercise for reasons

22:34 that don't result in any actual quality improvement.

22:37 What Dr.

22:38 Bina's tested here, can we actually have AI do the chart?

22:43 How good would the AI chart review be as compared

22:47 with human experts like the head of our satis commitee for example.

22:51 On individual questions,

22:52 it's 97 plus percent accurate on a fairly small sample size,

22:56 but that fairly small sample size is because

22:58 we have we only do 20 cases a month.

23:01 That's almost just the comedy of the problem of the goal

23:05 standard is so bad that this is what we're estimating against.

23:08 If you look at how much do we agree on every single question,

23:11 it's 90% of the time.

23:13 90% of the time, complete agreement on all 40 plus questions,

23:17 such that that's actually what we would have

23:19 submitted to Medicare anyway as part of our measures.

23:22 Interestingly, in those 10% of cases where the AI is wrong,

23:27 in almost half of those cases, actually, it was the abstractor that was wrong.

23:32 When they went back and looked,

23:33 now that's not to say the other thing doesn't happen as well,

23:36 but I think when you look at the ones

23:37 where the AI got it wrong, and the abstractor,

23:39 again, who's an experienced abstractor goes back and looks at this, they said,

23:43 I missed that one thing because it's a lot of actual

23:46 tedious work and so it's hard to do it well.

23:49 This actually lets you give feedback back to people fairly quickly.

23:53 As of July of last year, we actually use generative AI to do our first

23:58 pass on sepsis quality measurement with a human double check.

24:01 I realize there's issues of automation bias,

24:04 but just to say that the current thing is not that great.

24:07 If you compare what we're doing against the care we currently have,

24:10 this actually gives us a way to be responsive.

24:12 We've actually in the midst of a trial right now that we've recently completed,

24:16 where we actually give this feedback back to the treating doctor and said Hey,

24:19 at the time of discharge, did you know your sepsis quality?

24:22 You actually missed this, this and this?

24:24 That actually is a way that we can,

24:25 for the first time ever give feedback back to clinicians

24:28 in a way that we wouldn't otherwise be able to.

24:30 Going back to that opening slide, what's wrong with this approach?

24:34 This is the approach that we did.

24:36 Well, one of the issues is that predictive AI models are biased.

24:38 The other one is that not everyone can get the intervention.

24:42 I know this from personal experience because my mother-in

24:45 -law was supposed to get a procedure in December.

24:50 The day that we were about to launch this new

24:52 initiative that would have addressed some of the issues she had,

24:55 she found out that she had not held

24:57 one of her medications that she needed to hold,

24:59 had to get the procedure rescheduled.

25:01 That introduces a backlog of patients because you can't

25:04 often get someone in in time to fill that slot.

25:06 Then as you're going to get your own procedure scheduled,

25:09 it's delayed because of these things that happen not infrequently.

25:14 Another emerging model of care,

25:16 and this is evident to the most extreme in the recent

25:20 ARPAH advocate program that was just announced last week is,

25:24 you've got constrained resources.

25:25 Why don't you have a first pass with direct to patient AI to start

25:29 the intervention with escalation to clinicians when

25:33 there's a question that EI can't answer.

25:36 The everyone gets the intervention.

25:41 There's the lowest common denominator,

25:44 but there's a lot of risk that gets introduced by doing direct

25:46 to patient AI because there's no human in the loop for that conversation.

25:51 This is some of the work that we've done in the past,

25:54 I would say six to eight months, really thinking hard about is automation

25:58 of some of these kinds of tasks achievable?

26:01 If it is, what's the evaluation process you need to have in place?

26:05 I would argue when you go from human in the loop to human on the loop,

26:08 you actually want to know in the human in the loop period,

26:11 what does the human actually do?

26:13 if they're presented with information that they need to edit,

26:15 they need to review,

26:16 they need to act on, they need to disagree with, do they actually disagree?

26:20 It's a proxy measure for maybe automation bias, but in the best case scenario,

26:25 it's a proxy measure for, are they actually,

26:27 comfortable with what the AI is doing.

26:30 The moment you go to human on the loop, now you actually the biggest thing I

26:34 worry about is is the escalation pathway appropriate?

26:38 In other words, if the AI doesn't know when

26:41 to escalate or it's inaccurately not escalating or escalating things,

26:44 then you basically can't automate.

26:46 There's no amount of work you can automate.

26:49 Even if you go to an automated fashion,

26:51 this is not to say AI is doing everything.

26:53 It's usually there's a work queue of follow ups.

26:56 I have a question about pregnancy.

26:58 I have a question about the specific medication that has to get

27:02 followed up by a nurse AI we don't want the AI answering that.

27:05 But there's a question about how much parking costs,

27:08 and the AI can answer that and save a nurse 10 minutes on the phone

27:11 because there was a parking question

27:13 or a directions question that the AI could answer.

27:16 We started with AI governance framework and said, How do we do this.

27:20 I credit Dr.

27:21 Shim Namati with some of your early work on this and then Dr.

27:26 Arbina basically we met and said,

27:28 Let's do a failure mode and effects analysis on what

27:31 would it even take for us to believe this is doable?

27:35 This is just a handful of some of the failure modes that we identified,

27:40 but there were several other failure modes that we said.

27:41 These are all the things that we are worried about.

27:44 Thje question was, if they happen, how bad would this problem be,

27:48 and how likely are they to happen?

27:50 We use that to calculate a risk level of that thing

27:54 that determines how much scrutiny that actual use case is going to get.

27:58 What I'm showing you on the right is

27:59 actually one of the nurses red teaming this tool,

28:02 where they told the tool my prescribing doctor had died.

28:05 I don't know if I should hold my medication.

28:07 The tool correctly says this has to get escalated to a nurse.

28:11 But these are the things where everything from innocuous

28:15 things like ordering food at a restaurant and recommendations,

28:18 where it's just burning tokens on a unrelated issue,

28:20 all the way to inaccurate medical information.

28:23 It requires a lot of build work and a lot of testing work.

28:28 This is actually a snapshot of our GI colonoscopy workflow,

28:35 where we actually every day at 4:00 P.M.

28:37 and 5:30 P.M.

28:38 I calls go out, people get a text in the morning,

28:40 we follow AB 30,30, and we disclose it's AI.

28:44 We give them an opt out, we give them options to get call back from a human

28:47 if they don't want to talk to the AI.

28:49 Able to actually get a lot of people on the phone.

28:52 Interestingly, in the time that we've gone

28:55 live between testing and actual live calls,

28:58 we've made over 1,200 calls we've had six plus physicians, 20 nurses,

29:02 patient experience, risk management,

29:04 quality and safety involved in the testing process.

29:07 We're right now looking at patient feedback on how patients feel about it.

29:12 This is some of the feedback we've

29:14 gotten from our nurses that works surprisingly well.

29:18 There's some of our most skeptical folks as am I usually,

29:21 when it comes to these tools.

29:23 About 50% of these patients successfully pick up and complete the entire call.

29:28 Average call time of 7 minutes.

29:30 Our early data, and again,

29:32 we're going to be looking at this more carefully is among

29:35 the people who pick up and talk to go through the whole script,

29:38 their cancellation rate drops tenfold.

29:42 It's one of these things where I think we're trying stuff,

29:47 doing it with guardrails in a safe way.

29:50 Then I think measuring, and I think that measurement is a really

29:53 critical thing because if you find it doesn't work,

29:56 you find an issue that requires you to act and fix it, then you have to do that.

30:00 We actually have real time transcript analysis the moment the person hangs up,

30:04 that transcript gets analyzed for all things

30:06 to make sure we're comfortable with what happened.

30:08 We have human review on that because we're on the human on the loop phase,

30:11 which to me means the moment they hang up,

30:13 a human starts looking at those transcripts to see what happened.

30:16 I'd like to get to a more mature phase where we have AI versus AIR teaming.

30:20 We identify a bunch of threat modes of things we're worried about,

30:23 and then we have AI calling into our AI

30:25 to test it to make sure that as there's updates,

30:27 as there's changes, that we have a way to actually look at it

30:29 and make sure that it hasn't shifted and stopped working in a way.

30:32 In the interest of time, I'm not going to play a clip,

30:34 but I have a clip of the actual example

30:36 of a call going out to one of our testers.

30:39 I know I'm at time, but I'm going to end with some controversy slides,

30:42 two of them, and then I'll close.

30:44 I think some of these we talked about can biased AI harm patients?

30:48 That's a lot of the questions that were raised by Zooters paper.

30:51 I think the answer is unabashedly, yes, it caN.

30:54 I think we have to think about in our governance process how do we measure this?

30:59 How do we make this risk understandable?

31:01 How do we then put it in holistic context against the current state of affairs?

31:06 Because the usual care is usually the compare it to your arm,

31:11 and it's often not great.

31:12 We're often thinking about, what is it currently look like.

31:15 Is it better with AI versus not?

31:18 Sometimes it's not and sometimes it is,

31:20 but we have to have a realistic way to look at that.

31:23 How do we pay for all this?

31:25 I think that there is a real emphasis right now

31:28 on revenue cycle AI that directly earns you more money.

31:32 It's an arms race.

31:33 It's a prisoner's dilemma.

31:35 It will make you more money now,

31:36 but in ten years, when everyone's using the tool,

31:38 you'll just be paying more money for the same pay and what we found is

31:43 that maybe some of these tools make

31:45 your life easier and you can see more patients.

31:47 But there was early work actually at a UCLA,

31:49 first ramiPril trial that showed that the time savings

31:52 are modest for some of these AI scribing tools.

31:55 We wrote the editorial on that.

31:57 Then UCSF actually published a follow up paper showing

32:00 some modest gains in access to care visit volume,

32:04 not because they're making people,

32:05 but because people are able to fit additional patients in because these tools

32:08 really sometimes do help some clinicians

32:10 see patients faster by reducing documentation burden.

32:13 But this is some of the just rubber hits the road issues.

32:17 You probably saw a week or two ago in Utah,

32:20 they legalized AI refills on 180 medications with a specific set

32:26 of controls and actually bumped into the CEO of the company at ARPAH,

32:30 doing some of this work.

32:32 If you look at the guard rules,

32:34 there's actually substantial guard rules in place.

32:36 I think the headlines made it look like AI is just taking over.

32:39 It's going to prescribe.

32:40 I totally get the fact that it's

32:42 concerning that AI would be prescribing medications.

32:45 But in this case, it's refills.

32:47 They have actually a review process that's fairly robust in my view.

32:51 There's a great editorial that Matt Allen,

32:53 one of our medical students wrote on this in the salt lake Tribune,

32:55 that I would encourage you to read.

32:57 I think just things to think about,

33:00 things are changing, what is the counterfactual.

33:02 It's not being able to get into a Dr.

33:03 Not getting your refill and not taking your medication oftentimes.

33:07 I think access is a problem.

33:10 Because we can't use AI tools to translate because of regulatory

33:14 requirements around being fair in how we translate and interpret for patients,

33:19 what it sometimes means is that for non English speaking patients,

33:22 we just send them stuff in English because we can't translate.

33:25 Is that actually better than an inaccurate

33:27 translation or a mostly accurate translation?

33:30 I think that's debatable.

33:31 We've actually put together some policy

33:34 recommendations around this in a regulatory sandbox.

33:37 Can we try this out and see, because this might be a tool we can use to open

33:41 up accessibility into our systems to a broad set of patients.

33:45 My last slide here is just that transparency consent.

33:48 Said, whatever we're doing, we're going to be transparent about it.

33:51 Any direct to patient AI that we have in any form, in your clinical note,

33:55 otherwise, we have a best practice

33:57 for universal AI disclosure at UCsan Diego Health.

34:01 Privacy big issues.

34:02 Colleague of mine, Paige Nong and Jodi Platt published

34:06 on patients trust issues with health systems using AI.

34:08 Cool for Health AI published a survey yesterday they

34:11 did with 1,000 plus patients on the same exact thing.

34:14 Environmental impacts are real.

34:16 I think that there's this great book by Karen How on Empire of AI,

34:20 where she breaks down into this.

34:21 There's been some follow up back and forth where she actually updated

34:24 some sections of her book because there was a unit error with meter

34:28 squared and leaders where actually the impacts stated in the book were actually

34:31 thousandfold less or there's thousandfold less than what was stated in the book.

34:35 Still and then there's some issues around water use.

34:39 Is it consumptive water use versus water that's going to re enter the ecosystem?

34:43 I'm not going to get into these issues, but just to say,

34:45 there are real issues and there are things that are important to follow.

34:47 Then the impending AI bubble.

34:49 OpenAI, investing in Oracle, which is investing in video,

34:51 which is investing in OpenAI, how does that work?

34:54 With that, I'll close.

34:56 Thank you.

35:09 [APPLAUSE]