Artwork for podcast Learning Bayesian Statistics
#117 Unveiling the Power of Bayesian Experimental Design, with Desi Ivanova
Episode 117 • 15th October 2024 • Learning Bayesian Statistics • Alexandre Andorra
00:00:00 01:13:11

Share Episode

Shownotes

Proudly sponsored by PyMC Labs, the Bayesian Consultancy. Book a call, or get in touch!

Our theme music is « Good Bayesian », by Baba Brinkman (feat MC Lars and Mega Ran). Check out his awesome work!

Visit our Patreon page to unlock exclusive Bayesian swag ;)

Takeaways:

  • Designing experiments is about optimal data gathering.
  • The optimal design maximizes the amount of information.
  • The best experiment reduces uncertainty the most.
  • Computational challenges limit the feasibility of BED in practice.
  • Amortized Bayesian inference can speed up computations.
  • A good underlying model is crucial for effective BED.
  • Adaptive experiments are more complex than static ones.
  • The future of BED is promising with advancements in AI.

Chapters:

00:00 Introduction to Bayesian Experimental Design

07:51 Understanding Bayesian Experimental Design

19:58 Computational Challenges in Bayesian Experimental Design

28:47 Innovations in Bayesian Experimental Design

40:43 Practical Applications of Bayesian Experimental Design

52:12 Future of Bayesian Experimental Design

01:01:17 Real-World Applications and Impact

Thank you to my Patrons for making this episode possible!

Yusuke Saito, Avi Bryant, Ero Carrera, Giuliano Cruz, Tim Gasser, James Wade, Tradd Salvo, William Benton, James Ahloy, Robin Taylor,, Chad Scherrer, Zwelithini Tunyiswa, Bertrand Wilden, James Thompson, Stephen Oates, Gian Luca Di Tanna, Jack Wells, Matthew Maldonado, Ian Costley, Ally Salim, Larry Gill, Ian Moran, Paul Oreto, Colin Caprani, Colin Carroll, Nathaniel Burbank, Michael Osthege, Rémi Louf, Clive Edelsten, Henri Wallen, Hugo Botha, Vinh Nguyen, Marcin Elantkowski, Adam C. Smith, Will Kurt, Andrew Moskowitz, Hector Munoz, Marco Gorelli, Simon Kessell, Bradley Rode, Patrick Kelley, Rick Anderson, Casper de Bruin, Philippe Labonde, Michael Hankin, Cameron Smith, Tomáš Frýda, Ryan Wesslen, Andreas Netti, Riley King, Yoshiyuki Hamajima, Sven De Maeyer, Michael DeCrescenzo, Fergal M, Mason Yahr, Naoya Kanai, Steven Rowland, Aubrey Clayton, Jeannine Sue, Omri Har Shemesh, Scott Anthony Robson, Robert Yolken, Or Duek, Pavel Dusek, Paul Cox, Andreas Kröpelin, Raphaël R, Nicolas Rode, Gabriel Stechschulte, Arkady, Kurt TeKolste, Gergely Juhasz, Marcus Nölke, Maggi Mackintosh, Grant Pezzolesi, Avram Aelony, Joshua Meehl, Javier Sabio, Kristian Higgins, Alex Jones, Gregorio Aguilar, Matt Rosinski, Bart Trudeau, Luis Fonseca, Dante Gates, Matt Niccolls, Maksim Kuznecov, Michael Thomas, Luke Gorrie, Cory Kiser, Julio, Edvin Saveljev, Frederick Ayala, Jeffrey Powell, Gal Kampel, Adan Romero, Will Geary, Blake Walters, Jonathan Morgan, Francesco Madrisotti, Ivy Huang and Gary Clarke.

Links from the show:

Transcript

This is an automatic transcript and may therefore contain errors. Please get in touch if you're willing to correct them.

Transcripts

Speaker:

Today I am delighted to host Desi Ivanova, a distinguished research fellow in machine

learning at the University of Oxford.

2

:

Desi's fascinating journey in statistics has spanned from quantitatifiance to the

frontiers of Bayesian experimental design, or BED, BED.

3

:

In our conversation, Desi dives into the deep

4

:

world of BED where she has made significant contributions.

5

:

She begins by elucidating the core principles of experimental design, discussing both the

theoretical underpinnings and the complex computational challenges that arise in its

6

:

application.

7

:

Desi shares insights into the innovative solutions she's developed to make BED more

practical and applicable in real-world scenarios, particularly

8

:

highlighting its impact in sectors like healthcare and technology.

9

:

Throughout the discussion, Desi also touches on the exciting future of BED, especially in

light of recent advancements in AI and machine learning.

10

:

She reflects on the critical role of real-time decision-making in today's data-driven

landscape and how patient methods can enhance the speed and accuracy of such decisions.

11

:

This is Learning Vision Statistics, episode 117.

12

:

recorded September 26, 2024.

13

:

Welcome to Learning Bayesian Statistics, a podcast about Bayesian inference, the methods,

the projects and the people who make it possible.

14

:

I'm your host.

15

:

Alex and Dora.

16

:

You can follow me on Twitter at Alex underscore and Dora like the country for any info

about the show.

17

:

Learnbasedats.com is Laplace to be.

18

:

Show notes, becoming a corporate sponsor, unlocking Beijing Merch, supporting the show on

Patreon.

19

:

Everything is in there.

20

:

That's Learnbasedats.com.

21

:

If you're interested in one-on-one mentorship, online courses or statistical consulting,

feel free to reach out and book a call at topmate.io slash Alex underscore and Dora.

22

:

See you around, folks, and best patient wishes to you all.

23

:

And if today's discussion sparked ideas for your business, well, our team at PMC Labs can

help bring them to life.

24

:

Check us out at PMC-labs.com.

25

:

Hello my dear vegans, today I want to welcome our two new patrons in the full posterior

tier.

26

:

Thank you so much Ivy Hwing and Garrett Clark, your support literally makes this show

possible.

27

:

I am

28

:

looking forward to interacting with you guys in the LBS Slack channel.

29

:

Now, before we start the episode, I have a short story for you guys.

30

:

A few years ago, I started learning machine stats by watching all the tutorials I could

find that a teacher I really liked was teaching.

31

:

That teacher was no other than Chris Fonsbeck, PMC's creator and BDFL.

32

:

And five years down the road, senior

33

:

unpredictable road, am beyond excited to share that I will now be teaching a tutorial

alongside Chris.

34

:

That will happen at Pi Data New York from November 6 to 8, 2024.

35

:

And I would be delighted to see you there.

36

:

We will be teaching you everything you need to know to master Gaussian processes with IMC.

37

:

And of course, I will record a few live LBS episodes while I'm there.

38

:

But

39

:

I'll tell you more about that in the next episode.

40

:

In the meantime, you can get your ticket at pine.at.org slash NYC 2024.

41

:

I can't wait to see you there.

42

:

Okay, on to the show now.

43

:

Desi Ivanova, welcome to Learning Bayesian Statistics.

44

:

Thank you for having me, Alex.

45

:

Pleased to be here.

46

:

Yeah, yeah.

47

:

Thanks a lot for taking the time, for being on the show.

48

:

a to Marvin Schmidt for putting us in contact.

49

:

He was kind enough to do that on the base flow stack where we interact from time to time.

50

:

Today, though, we're not going to talk a lot about advertised Bayesian inference.

51

:

We're going to talk mostly about experimental design, Beijing experimental design.

52

:

So BED or BED, I like the acronym.

53

:

But before that, as usual, we'll start with your origin story, Daisy.

54

:

Can you tell us what you're doing nowadays and also how you ended up working on what

you're working today?

55

:

Yeah, of course.

56

:

So broadly speaking, I work in probabilistic machine learning research, where I've worked

on a few different things, actually.

57

:

So the audience here would be mostly familiar with Bayesian inference.

58

:

So I've worked on approximate inference methods, namely, know, variational inference.

59

:

You mentioned Marvin, right?

60

:

So we've actually collaborated with him on some amortized inference work.

61

:

I've also done some work in causality.

62

:

But my main research focus so far has been in an area called Bayesian experimental design,

as you correctly pointed out, BED for short, a nice acronym.

63

:

So BED, Bayesian experimental design was the topic of my PhD.

64

:

And yeah, will be the topic of this podcast episode.

65

:

Yeah, really, really keen on discussing.

66

:

and very, very close to my heart.

67

:

You know, how I ended up here.

68

:

That's actually a bit quite random.

69

:

So before, before getting into research, right, so before my PhD, I actually worked in

finance for quite a few years as a, as a quantitative researcher.

70

:

At some point,

71

:

I really started missing sort of the rigor in a sense of, you know, conducting research,

you know, being quite principled about, you know, how we measure uncertainty, how we

72

:

quantify robustness of our models and of the systems that we're building.

73

:

And right at the height of COVID, I decided to start my PhD back in 2020.

74

:

And

75

:

Indeed, the area, right, based on experimental design, that was originally not the topic

of my PhD.

76

:

I was supposed to work on certain aspects of variational autoencoders.

77

:

If you're familiar with these types of models, they're not as popular anymore, right?

78

:

So if I had ended up working on variational autoencoders, I guess a lot of my research

would have been, I mean, not wasted, but not as relevant.

79

:

not as relevant today as it was, you know, four or five years ago.

80

:

And how I ended up working with Bayesian experimental design specifically, basically,

approached my supervisor a few months before starting my PhD and I said, Hey, can I can I

81

:

read about something interesting to prepare for a PhD?

82

:

And he was like, yeah, just with these papers on Bayesian experimental design.

83

:

And that's how it happened.

84

:

Really?

85

:

Yeah.

86

:

Okay, cool.

87

:

Yeah, I love these.

88

:

I love asking this question because often, you know, with hindsight bias, when you're a

beginner, you like it's easy to trick yourself into thinking that people like you who are

89

:

experts on a on a particular topic and know that topic really well, because they did a PhD

on on it, like they

90

:

They have been doing that since they were, I don't know, 18 or even 15 or it was like all,

all planned and part of being big plan.

91

:

But most of the time when you ask people, it was not at all.

92

:

And it's the result of experimenting with things and also the result of different people

they have met and, and, and encounters and mentors.

93

:

And so I think this is also very valuable to, to

94

:

tell that to beginners because otherwise they can be very daunting.

95

:

100 % Yeah, I would 100 % agree with that.

96

:

And actually experimenting is good.

97

:

You know, again, we'll be talking about experimental design, I think.

98

:

Yeah, many times, you know, just by virtue of trying something new, you discover, you

know, I actually quite liked that.

99

:

And it actually works better, you know, for whatever purpose it might be, it might be your

commute to work, right?

100

:

There was this

101

:

very interesting research.

102

:

You know, when there is like a tube closure, right, if the metro is getting closed, you

know, some people, like 5 % of people actually discover an alternative route that actually

103

:

is much better for the daily commute.

104

:

But they wouldn't have done that had the closure not happened.

105

:

So almost like being forced to experiment may lead to actually better outcomes, right?

106

:

So it's quite interesting.

107

:

Yeah, yeah, no.

108

:

mean,

109

:

completely agree with that and that's also something I tell to a lot of people who reach

out to me, know, wondering how they could start working on Bayesian stats and often I'm

110

:

like, you know, trying to find something you are curious about, interested in and then

start from there because it's gonna be hard stuff and there are gonna be a lot of

111

:

obstacles.

112

:

So if you're not, you know, really curious about

113

:

what you are studying, it's going to be fairly hard to maintain the level of work that you

have to maintain to, to in the end enjoy what you're doing.

114

:

So experimenting is very important.

115

:

I completely agree.

116

:

and actually do you remember yourself?

117

:

so I'm curious first how Bajan is your work.

118

:

And also if you remember when you were, were introduced to, to Bajan stance.

119

:

When was I introduced to Bayesian stats?

120

:

That must have been probably in my undergrad days.

121

:

I remember I took some courses on kind of Bayesian data analysis, but then I didn't do any

of that during my time in industry.

122

:

Yeah.

123

:

And again, as I said, I ended up working on

124

:

Bayesian experimental design a little bit, a little bit randomly.

125

:

The work itself is, you know, it does use Bayesian principles quite a lot.

126

:

You know, we do Bayesian inference, we do, we start with a Bayesian model, right?

127

:

So the modeling aspect is also quite important.

128

:

You know, it's very important to have a good Bayesian model for all of these things to

actually make sense and work well in practice.

129

:

So I would say overall, the work is quite, quite Bayesian, right?

130

:

Yeah.

131

:

Yeah.

132

:

Yeah.

133

:

Yeah, for sure.

134

:

so actually, I think that's a good segue to introduce now, Bayesian experimental design.

135

:

So it's the first time we not talk, not the first time we talk about it on the show, but

it's a really dedicated episode about about that.

136

:

So could you introduce the topic to our listeners and basically explain and define what

Bayesian experimental design is?

137

:

Yeah, of course.

138

:

So can I actually take a step back and talk a little bit about experimental design first?

139

:

Yeah.

140

:

yeah.

141

:

And then we'll add the Bayesian kind of the Bayesian aspect to it.

142

:

So, you know, when, when I say, I work on Bayesian experimental design, most people

immediately think lab experiments, right?

143

:

For example, you're in a chemistry lab and you're trying to synthesize a new drug or a new

compound or something.

144

:

But actually, you know, the field of experimental design is much, broader than that,

right?

145

:

And to, you know, give a few concrete examples, you can think about surveys, right?

146

:

You may need to decide what questions to ask.

147

:

Maybe you want to tailor your questions as, you know, the survey progresses so that, you

know, you're asking very tailored, customized questions to each of your survey

148

:

participants.

149

:

You can think of clinical trials, right?

150

:

So how do you dose drugs appropriately?

151

:

Or, you know, when should you test for certain properties of these drugs, things like

absorption and so on?

152

:

So all of these things can be, you know, cast as a as an experimental design problem, as

an optimal experimental design problem.

153

:

So in my mind, designing experiments really boils down to optimal or at least intelligent

data gathering.

154

:

Does that make sense?

155

:

So we're trying to kind of optimally collect data in order to kind of learn about the

thing that we want to learn about.

156

:

So some underlying quantity of interest, right?

157

:

And the Bayesian framework, so the Bayesian experimental design framework specifically

takes an information theoretic approach to what intelligent or optimal means in this

158

:

context.

159

:

So as I already mentioned, it is a is a model based approach, right?

160

:

So we start with an underlying Bayesian model that actually describes or simulates the

outcome of our experiment.

161

:

And then the optimality part, right?

162

:

So the optimal design will be the one that maximizes the amount of information about the

thing that we're trying to learn about.

163

:

Yeah.

164

:

That makes sense?

165

:

can actually give a concrete example.

166

:

Maybe that will make it easier for you and for the listeners, right?

167

:

So if you think about, you know, the survey, the survey example, right?

168

:

kind of a simple but I think very easy to understand concept is you know trying to learn

let's say about time value of money preferences of different people right yeah so what

169

:

does that mean imagine your

170

:

a behavioral economist, right?

171

:

And you're trying to understand some risk preferences, let's say, of people.

172

:

Generally, the way that you do that is by asking people a series of questions of the form,

do you prefer some money now or you prefer some money later?

173

:

Right?

174

:

So do you prefer 50 pounds now or you prefer 100 pounds in one year?

175

:

Right.

176

:

And then you can choose, are you going to propose 50 pounds or 60 pounds or 100 pounds

now?

177

:

how much money you're going to propose in what time, right?

178

:

So you're going to do a hundred pounds in one month or in three months or in one year,

right?

179

:

So there is like a few choices that you can make.

180

:

And there is a strong incentive to do that with as few questions as possible because you

end up paying actually the money to the participants, right?

181

:

So basically,

182

:

we can start with an underlying Bayesian model that sort of models this type of

preferences of different human participants in this survey.

183

:

There's plenty of such models from psychology, from behavioral economics.

184

:

And at the end of the day, what we want to learn is a few parameters, right?

185

:

You can think about this model almost like a mechanistic

186

:

model that explains how preferences relate to things like, you know, are described by

things like a discount factor or sensitivity to various other things.

187

:

And by asking these series of questions, we're learning about these underlying parameters

in our Bayesian model.

188

:

Did that make sense?

189

:

Yeah.

190

:

Yeah.

191

:

I understand better now.

192

:

And so I'm wondering, it sounds a lot like, you know, just doing also causal modeling,

right?

193

:

So you write your causal graph and then based on that, you can have a generative model and

then, and fitting the model to data is just one part, but it's not what you start with to

194

:

write the model.

195

:

How is that related?

196

:

Right.

197

:

The fields are, in a sense, closely related in the sense that, you know, in order for you

to uncover kind of the true underlying causal graph, let's say if, you know, you start

198

:

with some assumptions, you don't know if X causes Y or Y causes X or, you know, or

something else, the way that you need to do this is by intervening in the system.

199

:

Right.

200

:

So

201

:

You can only, in a sense, have causal conclusions if you have rich enough data and by rich

enough data we generally mean experimental or interventional data, right?

202

:

So you're totally right in kind of drawing parallels in this, right?

203

:

And indeed we may...

204

:

design experiments that actually maximize information about the underlying causal graph,

right?

205

:

So if you don't know the graph and you want to uncover the graph, you can set up a

Bayesian experimental design framework that will allow you to, you know, select, let's

206

:

say, which nodes in my causal graph should I intervene on, with what value should I

intervene on, so that with as few experiments as possible, with as few interventions as

207

:

possible,

208

:

can I actually uncover the true, the ground truth, right?

209

:

The true underlying causal graph, right?

210

:

And, you know, kind of the main thing that you're optimizing for is this notion of

information content.

211

:

So how much information is each intervention, each experiment actually bringing us, right?

212

:

And...

213

:

And I think that's part of the reason why I find the Bayesian framework quite appealing as

opposed to, I guess, non-Bayesian frameworks.

214

:

You know, it really centers around this notion of information gathering.

215

:

And with the Bayesian model, we have a very precise definition of or a precise way to

measure an information content of an experiment.

216

:

Right.

217

:

If you think about

218

:

Imagine again, we're trying to learn some parameters in a model, right?

219

:

The natural, again, once we have the Bayesian model, the natural way to define information

content of an experiment is to look at, you know, what is our uncertainty about these

220

:

parameters under our prior, right?

221

:

So we start with a prior.

222

:

We have uncertainty that is embedded or included in our prior beliefs.

223

:

We're going to perform an experiment to collect some data, right?

224

:

So perform an experiment, collect some data.

225

:

we can update our prior to a posterior.

226

:

So that's classic Bayesian inference, right?

227

:

And now we can compare the uncertainty of that posterior to the uncertainty of our prior.

228

:

And the larger the drop, the better our experiment is, the more informative our experiment

is.

229

:

And so the best...

230

:

or the optimal experiment in this framework would be the one that maximizes this

information gain.

231

:

So the reduction in entropy, we're going to use entropy as a measure of uncertainty in

this framework.

232

:

So it is the experiment that reduces our entropy the most.

233

:

Does that make sense?

234

:

Yeah.

235

:

Total sense.

236

:

Yeah.

237

:

Total sense.

238

:

That's amazing.

239

:

I didn't know.

240

:

So yeah, I mean, that's, that's pretty natural then to include the causal framework into

that.

241

:

And I think that's one of the most powerful features of experimental design, because I

guess most of the time what you want to do when you design an experiment is you want to

242

:

intervene.

243

:

on a causal graph and see actually if your graph is close to reality or not.

244

:

So that's amazing.

245

:

And I love the fact that you can use experimental design to validate or invalidate your

causal graph.

246

:

That's really amazing.

247

:

Correct.

248

:

100%.

249

:

But I do want to stress that

250

:

The notion of causality is not necessary for the purposes of describing what Bayesian

experimental design is.

251

:

I'll give you a couple of other examples, actually.

252

:

So you may...

253

:

You may want to do something like model calibration.

254

:

Let's say you have a simulator with a few parameters that you can tweak, right?

255

:

So that it, I don't know, produces the best outcomes, right?

256

:

Or is optimally calibrated for the thing that you're trying to measure, right?

257

:

It is like, again, I don't think you need, you know, any concepts of causality here,

right?

258

:

It's you're turning a few knobs.

259

:

And you know, again, you can formulate this as an experimental design problem where, you

you are trying to calibrate your system with as few kind of no turns as possible.

260

:

Yeah.

261

:

Yeah, yeah, yeah.

262

:

That makes a ton of sense.

263

:

Something I'm curious about hearing you talk is, and that's also something you've worked

extensively on, is the computational challenges.

264

:

Can you talk about that?

265

:

What are the computational challenges associated with traditional bed, so Bayesian

experimental design, and how they affect the feasibility?

266

:

of bed in real world applications.

267

:

Yeah.

268

:

Yeah, that's that's an excellent point.

269

:

Actually.

270

:

I Yeah, I see you read some of of my papers.

271

:

So, all right.

272

:

So all of these kind of information objectives.

273

:

So what I just described, you know, we can look at the information content, we can

maximize information, and so on, like, it's all very natural.

274

:

And it's all very mathematically precise and beautiful.

275

:

But working with those information, theoretical objectives is quite difficult in practice.

276

:

And the reason for that is precisely as you say, they're extremely computationally costly

to compute or to estimate, and they're even more computationally costly to optimize.

277

:

And the careful listener would have noticed that I mentioned posterior inference.

278

:

Right.

279

:

So in order to compute the information content of an experiment, you actually need to

compute a posterior.

280

:

Right.

281

:

You need to compute a posterior given your data.

282

:

Now, where the problem lies is that you need to do this before you have collected your

data.

283

:

Right.

284

:

Because you designing an experiment and then only you will be performing it and then

observing the outcome and then you can do.

285

:

the actual posterior update.

286

:

Now, what we have to then do is look at our prior entropy minus our posterior entropy and

integrate over all possible outcomes that we may observe under the selected experiment.

287

:

And we have to do that for a number of experiments to actually find the optimal one.

288

:

So what we end up with is this sort of nesting

289

:

of expectations.

290

:

So we have an expectation, we have an average with respect to all possible outcomes that

we can observe.

291

:

And inside of our expectation, inside of this average, we have this nasty posterior

quantity that, generally speaking, is intractable.

292

:

Unless you're in a very specific case where you have a conjugate model, where your

posterior is available in close form, you actually don't have access to that posterior.

293

:

which means that you will need to do some form of approximation, right?

294

:

Whether it's exact like MCMC or is going to be a variational posterior computation.

295

:

Again, there is many ways of doing this.

296

:

The point is that for each design that you may want to try, you need to compute all of

these posteriors.

297

:

for every sample of your potential outcome, right?

298

:

Of your possible outcome under the experiment.

299

:

So what I was just describing is what is known as a doubly intractable quantity, right?

300

:

So again, this podcast audience is very familiar with Bayesian inference and how Bayesian

inference is intractable in general.

301

:

Now computing...

302

:

the EIG, the sort of computing the objective function that we generally use in Bayesian

experimental design is what is known as doubly intractable objective, which is quite

303

:

difficult to work with in practice, right?

304

:

Now, what this means for sort of real world applications is that you either need to throw

a lot of compute

305

:

on the problem.

306

:

Or you need to do, you know, some, you need to sort of give up on the idea of being

Bayesian optimal, right?

307

:

You may use some heuristics or something else.

308

:

And what this problem really becomes limiting is when we start to think about, you know,

running experiments in real time, for example.

309

:

So the survey example that I started with, you know,

310

:

you know, asking participants in your survey, do you prefer somebody now or somebody

later?

311

:

You know, it becomes quite impractical for you to, you know, run all these posterior

inferences and optimize all of these information theoretic objectives in between

312

:

questions, right?

313

:

So it's a little bit, you know, I asked you the first question now, let me run by MCMC.

314

:

Let me optimize some doubly intractable objective.

315

:

Can you just wait five minutes, please?

316

:

And then I'll get back to you with the next question.

317

:

Obviously, it's not something that you can realistically do in practice.

318

:

So I think, historically, the computational challenge of the objectives that we use for

Bayesian experimental design has really...

319

:

limited the feasibility of applying these methods in kind of real-world applications.

320

:

And how, so how did you, which innovations, which work did you do on that front?

321

:

That make all that better.

322

:

Right.

323

:

So there is a few things that I guess we can discuss here.

324

:

So number one, I mentioned posterior inference, right?

325

:

And I mentioned we have to do many posterior inference approximations for every possible

outcome of our experiment.

326

:

Now, I think it was the episode with Marvin.

327

:

right, where you talked about amortized Bayesian inference.

328

:

So in the context of Bayesian experimental design, amortized Bayesian inference plays a

very big role as well, right?

329

:

So one thing that we can do to sort of speed up these computations is to learn a...

330

:

to learn a posterior that is amortized over all the outcomes that we can observe, all the

different outcomes that we can observe, right?

331

:

And the beautiful part is that we know how to do that really well, right?

332

:

So we have all of these very expressive, variational families.

333

:

that we can pick from and optimize with data that we simulate from our underlying Bayesian

model.

334

:

So this aspect of Bayesian experimental design definitely touches on related fields of

amortized Bayesian inference and simulation-based inference.

335

:

So we're using simulations from our model to learn an approximate posterior.

336

:

that we can very quickly draw samples from, as opposed to having to fit an HMC for every

new data set that we may observe.

337

:

That makes sense.

338

:

Yeah.

339

:

And so I will refer listeners to the episode with Marvin, episode 107, where we dive into

amortized patient inference.

340

:

put that in the show notes.

341

:

I also put for reference three other episodes where we mentioned, you know, experimental

design.

342

:

So episode 34 with Lauren Kennedy, 35 with Paul Burkner and 45 with Frank Harrell, that

one.

343

:

focuses more on clinical trial design.

344

:

But that's going to be very interesting to people who are looking to these.

345

:

And yeah, so I can definitely see how amortized patient inference here can be extremely

useful based on everything you used it before.

346

:

Maybe do you have an example, especially I saw that during your PhD,

347

:

You worked on policy-based patient experimental design and you've developed these methods.

348

:

Maybe that will give a more concrete idea to listeners about what all of these means.

349

:

Exactly.

350

:

One way in which we can speed up computations is by utilizing, as I said, amortized

variational inference.

351

:

Now this will speed up the estimation of our information theoretic objective, but we still

need to optimize it.

352

:

Now, given that we have to do after each experiment iteration, right?

353

:

So we have collected our first data point, we have collected our first data point with a

need to...

354

:

update our model and with this new model under this new model, updated model, we need to

kind of decide what to do next.

355

:

Now, this is clearly also very computationally costly, right?

356

:

The optimization step of our information theoretic objective is quite computationally

costly, meaning that it is very hard to do in real time, right?

357

:

Again, going back to the survey example, you still can do it, right?

358

:

You can estimate it a little bit more quickly, but you still can't optimize it.

359

:

And this is where a lot of my PhD work has actually been focused on, right?

360

:

So developing methods that will allow you to run Bayes Bayesian Optimal Design in real

time.

361

:

Now, how are we going to do that?

362

:

So there is a little bit of a conceptual shift in the way that we think about designing

experiments, right?

363

:

What we will do is rather than choosing

364

:

the design, the single design that we're going to perform right now, right in this

experiment iteration.

365

:

What we're going to do is learn or train a design policy that will take as an input our

experimental data that we have gathered so far, and it will produce as an output the

366

:

optimal design for the next experiment iteration.

367

:

So our design policy is just a function, right?

368

:

It's just a function that takes past experimental data as an input and produces the next

design as an output.

369

:

Does that make sense?

370

:

Yeah, yeah, yeah.

371

:

I can see what that means.

372

:

How do you integrate that though?

373

:

You know, like I'm really curious concretely.

374

:

Yeah.

375

:

what does integrating all those methods, so a multi-spatial inference, variational

inference to the Bayesian experimental design, and then you have the Bayesian model that

376

:

underlies all of that.

377

:

How do you do that completely?

378

:

Yes, excellent.

379

:

When we say the model, I generally mean the underlying Bayesian model.

380

:

This is our model that we use to train our

381

:

let's say, variational amortized posterior.

382

:

And this is the same model that we're going to train our design policy network.

383

:

And I already said it's a design policy network, which means that we're going to be using,

again, deep learning.

384

:

We're going to be using neural networks to actually learn a very expressive function that

will be able to take our data as an input, produce the next design as an output.

385

:

Now, how we do that concretely?

386

:

There is, you know, by now a large number of architectures that we can pick that is

suitable for, you know, our concrete problem that we're considering.

387

:

So one very important aspect in everything that we do is that our policy, our neural

network should be able to take variable size data sets as an input.

388

:

Right?

389

:

Because every time we're calling our policy, every time we want a new design, we will be

feeding it with the data that we have gathered so far.

390

:

Right?

391

:

And so it is quite important to be able to condition on or take as an input variable

length sequences.

392

:

Right?

393

:

And so concretely, how can we do that?

394

:

Well, you

395

:

One kind of standard way of doing things is to basically take our experimental data that

we've gathered so far and embed each data point.

396

:

So we have an X for our design, a Y for our outcome.

397

:

Take this pair and embed it to a fixed dimensional representation, right, in some latent

space.

398

:

Let's say with a small neural network, right?

399

:

And we do that for each

400

:

individual design outcome pair, right?

401

:

So if we have n design outcome pairs, we're gonna end up with n fixed dimensional

representations after we have embedded all of this data.

402

:

Now, how can we then produce the next sort of optimal design for the next experiment

iteration?

403

:

There is many choices, and I think it will very much depend on the application.

404

:

So certain Bayesian models, certain underlying Bayesian models are what we call

exchangeable, right?

405

:

So the data conditional on the parameters can be, the data conditional on the parameters

is IID, right?

406

:

Which means that the order in which our data points arrive doesn't matter.

407

:

And again, the survey example.

408

:

is quite a good example of this precisely, right?

409

:

Like it doesn't really matter which question we ask first or second, you know, we can

interchange them and the outcomes will be unaffected.

410

:

This is very different to time series models where, you know, if we design, if we are

choosing the time points at which to take blood pressure, right, for example,

411

:

If we decide to take blood pressure at t equals five, we cannot then go back and take the

blood pressure at t equals two.

412

:

So the choice of architecture will very much depend on, as I said, the underlying problem.

413

:

And generally speaking, we have found it quite useful to explicitly embed the structure

that is known.

414

:

So if we know that our model is exchangeable, we should be using an appropriate

architecture, which will ensure that the order of our data doesn't matter.

415

:

If we have a time series, we can use an architecture that takes into account the order of

the data.

416

:

So for the first one, we have...

417

:

kind of standard architecture such as I don't know how familiar the audience would be, but

you know, in deep learning, there is an architecture called deep sets, right?

418

:

So basically, take our fixed dimensional representations and we simply add them together.

419

:

Very simple, right?

420

:

Okay, we have our end design outcome pairs.

421

:

We add them together, they're all of them are in in the same fixed dimensional

representation, we add them together.

422

:

Now this is our

423

:

representation or a summary of the data that we have gathered so far.

424

:

We take that and we maybe map it through another small neural network to produce the next

design.

425

:

If we have a time series model, then we can, you know, pass everything through an LSTM or

some form of recurrent neural network to then produce the next design.

426

:

And that will keep sort of the order in

427

:

and it will take the order into account.

428

:

Did that answer your question in terms of like how specifically we think about these

policies?

429

:

Yeah.

430

:

Yeah, that's fascinating.

431

:

And so basically, and we talked about that a bit with Marvin already, but the choice of

neural network is very important depending on the type of data because if you have, many

432

:

time series are complicated, right?

433

:

Like they already are, even if you're not using a neural network, time is always

complicated to work with.

434

:

because there is an autocorrelation, right?

435

:

So you have to be very careful.

436

:

So basically that means changing the neural network you're working with.

437

:

then so concretely, like what, you know, for practitioners, someone who is listening to us

or watching us on YouTube, and they want to start implementing BED in their projects,

438

:

what's practical advice

439

:

you would have for them to get started?

440

:

Like how, why, and also when, you know, because there may be some moments, some cases

where you don't really want to use BED.

441

:

And also what kind of packages you're using to actually do that in your own work.

442

:

So that's a big question, I know, but like, again, repeat it as you give the answers.

443

:

Yeah, yeah, yeah.

444

:

Let me start with kind of...

445

:

If people are looking to implement BASE in their projects, I think it is quite important

to sort of recognize where BASE experimental design is applicable, right?

446

:

So it can be applied whenever we can construct an appropriate model for our experiments,

right?

447

:

So the modeling part, like the underlying BASE model is actually doing a lot of the heavy

lifting in sort of in this framework.

448

:

simply because this is basically what we're to assess quality of the designs, right?

449

:

So the model is informing what a valuable information is.

450

:

And so I would definitely advise not to gloss over that part.

451

:

If your model is bad, if your model doesn't represent the data generating process, in

reality, the results might be quite poor.

452

:

Now, I think it's also good to mention that you don't need to know the exact probability

distribution of the outcomes of the experiment, right?

453

:

So you can, you know, as long as you can simulate, right?

454

:

So you can have a simulator-based model that simply samples outcomes of the experiment,

given the experiment.

455

:

which I think, you know, it simplifies things a little bit.

456

:

You know, don't have to write down exact probability distributions, but still you need to

be able to sample or simulate this outcome.

457

:

So that would be step number one, right?

458

:

So ensuring that you have a decent model that you can start sort of experimenting with,

you know, in the sense of like...

459

:

designing the policies or like training the policies or sort of designing experiments.

460

:

The actual implementation aspect in terms of software, unfortunately, based on

experimental design is not as well developed.

461

:

from software point of view as, for example, amortized Bayesian inferences, right?

462

:

So I'm sure that you spoke about the baseflow package with Marvin, which is a really

amazing sort of open source effort.

463

:

They have done a great job of implementing many of the kind of standard architectures that

you can basically, you know,

464

:

pick whatever works or like pick something that is relatively appropriate for your problem

and it will work out, right?

465

:

I think that is like a super powerful, super powerful framework that includes, know,

latest and greatest architectures in fact.

466

:

Unfortunately, we don't have anything like this for Bayesian experimental design yet, but

I am in touch with the Baystow guys and I'm definitely looking into

467

:

implementing some of these experimental design workflows in their package.

468

:

So I have it on my to-do list to actually write a little tutorial in Baseflow, how you can

use Baseflow and your favorite deep learning framework of choice, whether it's PyTorch or

469

:

JAX or like whatever, TensorFlow, to train sort of a...

470

:

a policy, a design policy along with all of the amortized posteriors and all the bells and

whistles that you may need to run, you know, some pipeline like that.

471

:

Right.

472

:

So I mentioned the modeling aspect, I mentioned the software aspect.

473

:

I think thinking about the problem.

474

:

in like other aspects of like, you going to run an adaptive experiment or are you going to

run a static experiment?

475

:

Right.

476

:

So adaptive experiments are much more complicated than static experiments.

477

:

So in an adaptive experiment, you're always conditioning on the data that you have

gathered so far, right?

478

:

In a static experiment, you just design a large batch of experiments and then you run it

once, you collect your data and then you do your Bayesian analysis from there.

479

:

Right.

480

:

And so

481

:

I generally always recommend starting with the simpler case, figure out whether the

simpler case works, do the proof of concept on a static or non-adaptive type of Bayesian

482

:

experimental design.

483

:

And then, and only then, start to think about, let me train a policy or let me try to do

an adaptive experimental design.

484

:

pipeline.

485

:

I think this is a bit of a common pitfall if I may say, like people tend to like jump to

the more complicated thing before actually figuring out kind of the simple case.

486

:

Other than that, I think, again, I think it's a kind of an active area of research to, you

know, figure out ways to evaluate your designs.

487

:

I think by now we have

488

:

pretty good ways of evaluating the quality of our posteriors, for example, right?

489

:

You have various posterior diagnostic checks and so on that doesn't really exist as much

for designs, right?

490

:

So what does it like, you know, I've maximized my information objective, right?

491

:

I have collected as much information as I can, right?

492

:

According to this information objective.

493

:

But what does that mean in practice, right?

494

:

Like there is no kind of real world information that I can...

495

:

test with, right?

496

:

Like if you're doing predictions, can, you can predict, can observe and then compare,

right?

497

:

And you can compute an accuracy score or a root mean squared error or like whatever makes

sense.

498

:

There doesn't really exist anything like this in, in sort of in, in, in design, right?

499

:

So it becomes much harder to quantify the success of such a pipeline.

500

:

And I think it's, it's, it's a super interesting

501

:

area for development.

502

:

It's part of the reason why I work in the field.

503

:

I think there is many open problems that if we figure out, I think we can advance the

field quite a lot and make data gathering an actual thing, principled and robust and

504

:

reliable so that you run your expensive pipeline, but you end up with

505

:

you kind of want to be sure that the day that you end up with is actually useful for the

purposes that you want to use it.

506

:

yeah, did that answer the question?

507

:

So you have the modeling aspect, you have the software aspect, which we are developing, I

think, you know, we will hopefully eventually get there.

508

:

Think about your problem, start simple.

509

:

Try to think about diagnostics.

510

:

And I think, again, I mentioned, you know, it's very much an open

511

:

an open problem, but maybe for your concrete problem at hand, you might be able to kind of

intuitively say, this looks good or this doesn't look good.

512

:

Automating this is like, it's a very interesting open problem and something that I'm

actively working on.

513

:

Yeah.

514

:

Yeah.

515

:

And thank you so much for all the work you're doing on that because I think it's super

important.

516

:

I'm really happy to see you on the base flow side because yeah, those guys are doing

517

:

Amazing work.

518

:

There is the new version that's now been merged on the dev branch, which is back in

agnostics.

519

:

So people can use it with their preferred deep learning package.

520

:

So I always forget the names, but TensorFlow, PyTorch, and JAX, I'm guessing.

521

:

I'm mostly familiar with JAX because that's the one.

522

:

and a bit of PyTorch because that's the ones we're interacting with in PymC.

523

:

This is super cool.

524

:

I've linked to the Baseflow documentation in the show notes.

525

:

Is there maybe, I don't know, a paper, a blog post, something like that you can link

people to with a workflow of patient experimental design and that way people will get an

526

:

idea of how to do that.

527

:

So hopefully by the time the episode is out, I will have it ready.

528

:

Right now I don't have anything kind of practical.

529

:

I'm very happy to send some of the kind of review papers that are out there on Bayesian

Experimental Design.

530

:

hopefully in the next couple of weeks I'll have the tutorial, like a very basic

introductory tutorial.

531

:

you know, we have a simple model, we have our simple parameters, you know, what we want to

learn, here is how you define your posteriors, here is how we define your policy, you

532

:

know, and then you switch on base flow, and then you know, voila, you have you have you

have your results.

533

:

So yeah, I'm hoping to get a blog post like of this of this sort done in in the next

couple of weeks.

534

:

So once ready, I will thank you.

535

:

I will thank you with that.

536

:

Yeah, for sure.

537

:

yeah, for sure.

538

:

Can't wait and Marvin and I are gonna start working on setting up a modeling webinar

Amazing which is a you know another format I have on the show So this is like, you know,

539

:

I'm like Marvin will welcome and share his screen and show how to do the the amortized

patient inference workflow with base flow also using pinc and all that cool stuff now that

540

:

the new API

541

:

is merged, we're going to be able to work on that together and set up the modeling

webinar.

542

:

So listeners, definitely stay tuned for that.

543

:

I will, of course, announce the webinar a bit in advance so that you all have a chance to

sign up.

544

:

And then you can join live, ask questions to Marvin.

545

:

That's going to be super fun.

546

:

And mainly see how you would do Amortize Bayesian inference, concretely.

547

:

Great.

548

:

Amazing.

549

:

Sounds fun.

550

:

Yeah, that's going to be super fun.

551

:

Something I was thinking about is that your work mentions enabling real-time design

decisions.

552

:

And that sounds really challenging to me.

553

:

So I'm wondering how critical is this capability in today's data-driven decision-making

processes?

554

:

Yeah.

555

:

I do think it really is quite critical, right?

556

:

In most kind of real world practical aspects, practical problems, you really do need to

run to make decisions fairly quickly.

557

:

Right?

558

:

Again, all the surveys as an example, you know, you have anything that involves a human,

and you want to adapt as you're performing the experiment, you kind of need to ensure that

559

:

things are

560

:

you're able to run things in real time.

561

:

And honestly, I think part of the reason why we haven't seen a big explosion or based on

experimental design in practice is partly because we couldn't until recently actually run

562

:

these things fast enough, both because of the computational challenges, now that we know

how to do amortize inference very well, now that we know how to train policies.

563

:

that will produce designs very well.

564

:

I am expecting things to, you know, to improve, right?

565

:

And to start to see some of these some of these methods applied in practice.

566

:

Having said that, I do think and please stop me if that's totally unrelated, but to make

things

567

:

successful in practice, there are a few other things that in my opinion have to be

resolved before, you know, we're confident that we can, you apply, you know, such black

568

:

boxes in a sense, right, because we have all these neural networks all over the place.

569

:

And it's not entirely clear whether all of these things are robust to various aspects of

the complexities of the real world.

570

:

Right.

571

:

So

572

:

things like model mis-specification, right?

573

:

So is your Bayesian model actually a good representation of the thing that you're trying

to study?

574

:

That's a big open problem again.

575

:

And again, I'm going to make a parallel to Bayesian inference actually.

576

:

For inference purposes, model mis-specification may not be as bad as it is for design

purposes.

577

:

And the reason for that is you will still get valid under some of the assumptions, of

course, you will still get valid inferences or like you will still be close.

578

:

You still get the best that you can do under the under the the the assumption of a wrong

model.

579

:

Now, when it comes to design, we have absolutely no guarantees.

580

:

And oftentimes we end up in very pathological situations where because we're using our

model to inform the data collection.

581

:

and then to also evaluate, right, fix that model on the same data that we've gathered.

582

:

If your model is misspecified, you might not even be able to detect the misspecification

because of the way that the data was gathered.

583

:

It's not IID, right?

584

:

Like it's very much a non-IID data collection process.

585

:

And so I think when we talk about practical things,

586

:

we really, really need to start thinking about how are we going to make our systems or the

methods that we develop a little bit more robust to misspecification.

587

:

And I don't mean we should solve model misspecification.

588

:

I think that's a very hard task that is basically unsolvable, right?

589

:

Like it is solvable under assumptions, right?

590

:

If you tell me what your misspecification is, you know, we can improve things, but in

general, this is not

591

:

something that we can sort of realistically address uniformly.

592

:

But yeah, so again, going back to practicalities, I do think it's of crucial importance to

sort of make our pipeline in diagnostics sort of robust to some forms of

593

:

mis-specification.

594

:

Yeah.

595

:

Yeah, yeah, for sure.

596

:

And that's where also

597

:

I really love Amortized Patient Inference because it allows you to do simulation-based

calibration.

598

:

And I find that especially helpful and valuable when you're working on developing a model

because already before fitting to data, you already have more confidence about what your

599

:

model is actually able to do and not do and where the possible pain points would be.

600

:

And I find that.

601

:

super helpful.

602

:

And actually talking about all that, I'm wondering where you see the future of Bayesian

experimental design heading, particularly with advancements in AI and machine learning

603

:

technologies.

604

:

Wow.

605

:

Okay.

606

:

So I do view this

607

:

type of work.

608

:

So this type of research is a little bit orthogonal to all of the developments in sort of

modern AI and machine learning.

609

:

And the reason for this is that we can literally borrow the latest and greatest

development in machine learning and plug it into our pipelines.

610

:

there is a better architecture to do X, right?

611

:

Like we can take that architecture and, you know, utilize it for our purposes.

612

:

So I think

613

:

you know, when it comes to the future of Bayesian experimental design, given, you know,

all of the advancements, I think this is great because it's kind of helping the field even

614

:

more, right?

615

:

Like we have more options to choose from, we have better models to choose from, and kind

of the data gathering aspect will always be there, right?

616

:

Like we will always want to collect better data for the purposes of, you know, our data

analysis.

617

:

And so, you know, the design aspect will still be there and with the better models, we'll

just be able to gather better data, if that makes sense.

618

:

Yeah, that definitely makes sense.

619

:

Yeah, for sure.

620

:

And that's interesting.

621

:

Yeah, I didn't anticipate that kind of answer, that's okay.

622

:

definitely.

623

:

see what you mean.

624

:

Maybe before like, yeah, sorry, but even if you think, you know, now you're everybody's

gonna have their AI assistant, right?

625

:

Now, wouldn't it be super frustrating if your AI assistant takes three months to figure

out what you like for breakfast?

626

:

And like, it's experimenting or like, it's just randomly guessing.

627

:

do you like fish soup for breakfast?

628

:

Like,

629

:

How about I prepare you a fish soup for breakfast or like, or I propose you something like

that, right?

630

:

And so I think again, like this personalization aspect, right?

631

:

Like again, kind of sticking to, I don't know, personal AI assistance, right?

632

:

The sooner or the quicker they are able to learn about your preferences, the better that

is.

633

:

And again, you know, we're learning about preferences.

634

:

Again, I'm gonna refer back to the...

635

:

you know, time value of many preference learning, like it is just a more complicated

version of that.

636

:

Right.

637

:

And so if your latest and greatest AI assistant is able to learn and customize itself to

your preferences much more quickly than otherwise, you know, that's a huge win.

638

:

Right.

639

:

And I think this is precisely where all these sort of principle data gathering techniques

can really shine.

640

:

Once we figure out, you know, the

641

:

the sort of the issues that I was talking about, I mean, that makes sense.

642

:

Maybe to play us out, I'm curious if you have something like applications in mind,

practical applications of bed that you've encountered in your research, particularly in

643

:

the fields of healthcare or technology that you found particularly impactful.

644

:

Right.

645

:

Excellent question.

646

:

And actually, I was going to mention that aspect as, you know, where you see the future of

Bayesian experimental design.

647

:

Part of kind of our blind spots, if I may refer to that as sort of blind spots, is that in

our research so far, we have very much focused on developing methods, developing

648

:

computational methods to sort of make some of these

649

:

based on experimental design pipelines actually feasible to run in practice.

650

:

Now, we haven't really spent much time working with practitioners, and this is a gap that

we're actively trying to sort of close.

651

:

In that spirit, we have a few applications in mind.

652

:

We sort of apply people.

653

:

particularly in the context of healthcare, as you mentioned.

654

:

So clinical trials design is a very big one.

655

:

So again, things like getting to the highest safe dose as quickly as possible, For, again,

being personalized to the human, given their context, given their various characteristics.

656

:

is one area where we're looking to sort of start some collaborations and explore this

further.

657

:

Our group in Oxford have a new PhD student joining that will be working in collaboration

with biologists to actually design experiments for something about cells.

658

:

I don't know anything about biology, so.

659

:

I'm not the best person to actually describe that line of work.

660

:

But hopefully there will be some concrete, exciting applications in the near future.

661

:

So that's applications in biology.

662

:

And finally, you know, there's constantly lots of different bits and pieces like, you

know, people from chemistry saying, hey, I have this thing, can we...

663

:

Can we work on maybe, you know, performing or like setting up a basic experimental design

pipeline?

664

:

I think the problem that we've had so far or I've had so far is just lack of time.

665

:

There's just so many things to do and so little time.

666

:

But I am very much actively trying to find time in my calendar to actually work on a few

applied projects because I do think, you know,

667

:

It's all like developing all these methods is great, right?

668

:

I mean, it's very interesting math that you do.

669

:

It's very interesting coding that you do.

670

:

But at the end of the day, you kind of want these things to make someone life's better,

right?

671

:

Like a practitioner that will be able to save some time or save some money or, you know,

improve their data gathering and therefore, you know, downstream analysis much better.

672

:

and more efficient thanks to some of this research.

673

:

So I hope that answered your question in terms of concrete applications.

674

:

think we'll see more of that.

675

:

But so far, you know, the two things are, yeah, clinical trial design that we're exploring

and some of this biology cell stuff.

676

:

Yeah, yeah, no, that's worrying.

677

:

that's, I mean, definitely looking forward to it.

678

:

That sounds absolutely fascinating.

679

:

yeah, if you can make that.

680

:

happening in important fields like that, that's going to be extremely impactful.

681

:

Awesome, Desi.

682

:

I've already...

683

:

Do you have any sort of applications that you think design, basic experimental design

might be suitable?

684

:

I know you're quite experienced in various modeling aspects based in modeling.

685

:

So, yeah, do you have anything in mind?

686

:

Yeah, I mean a lot.

687

:

So marketing, know is already using that a lot.

688

:

Yeah.

689

:

Clinical trials for sure.

690

:

Also now that they work in sports analytics, well, definitely sports, you know, you could

include that into the training of elite athletes and design some experiments to actually

691

:

test causal graphs and see if pulling that lever during training is actually something

that makes a difference during

692

:

the professional games that actually count.

693

:

yeah, I can definitely see that having a big impact in the sports realm, for sure.

694

:

Nice.

695

:

Well, if you're open to collaborations, can do some designing of experiments once you up

your sports models.

696

:

Yeah.

697

:

Yeah.

698

:

Yeah.

699

:

mean, as soon as I work on that, I'll make sure to reach out because that's going to be...

700

:

It's definitely something I'm work on and dive into.

701

:

So that's gonna be fascinating to work on that with you for sure.

702

:

Sounds fun, yeah.

703

:

Yeah, exactly.

704

:

Very, very exciting.

705

:

well, thanks, Stacey.

706

:

That was amazing.

707

:

I think we covered a lot of ground.

708

:

I'm really happy because Alina had a lot of questions for you.

709

:

But thanks a lot for keeping your answers very...

710

:

focused and not getting distracted by all my decorations.

711

:

Of course, I have to ask you the last two questions I ask every guest at the end of the

show.

712

:

So first one, if you had unlimited time and resources, which problem would you try to

solve?

713

:

it's a really hard one.

714

:

Honestly, I because I know you asked those questions.

715

:

I was like, what am going to say?

716

:

I honestly don't know.

717

:

There's so many things.

718

:

But

719

:

Again, I think it would be something of high impact for humanity in general, probably so

in climate change would be something that I would dedicate my unlimited time and

720

:

resources.

721

:

That's good answer.

722

:

That's definitely a popular one.

723

:

So you're in great company and I'm sure the team already working on that is going to be

very...

724

:

be happy to welcome you.

725

:

No, let's hope.

726

:

I think we need to speed up the solutions.

727

:

Like seeing what is happening, right?

728

:

I think it's rather unfortunate.

729

:

And second question, if you could have dinner with any great scientific mind, dead, alive

or fictional, who would it be?

730

:

Yeah.

731

:

So I think it will be Claude Shannon.

732

:

So, you

733

:

the godfather of information theory or like actually the father of information theory.

734

:

Again, partly because a lot of my research is inspired by information theory principles in

Bayesian experimental design, but also outside of Bayesian experimental designs.

735

:

It sort of underpins a lot of the sort of modern machine learning development, right?

736

:

And

737

:

What I think will be really quite cool is that if you were to have dinner with him, if I

were to have dinner with him and basically tell him like, hey, look at all these language

738

:

models that we have today.

739

:

Like Claude Shannon was the person that invented language models back in 1948, right?

740

:

So that's many years ago.

741

:

And like, literally not even having computers, right?

742

:

So he would, he calculated things by hand and produced output that actually looks like

English, right?

743

:

In 1948.

744

:

And so I think, you know, a brilliant mind like him, you know, just seeing kind of the

progress that we've made since then.

745

:

And like, we actually have language models on computers that behave like humans.

746

:

I'll be super keen to hear like, what is next from him.

747

:

And I think he will have some very interesting answers to that.

748

:

What is the future for information processing and the path to artificial, I guess people

call it artificial general intelligence.

749

:

So what would be the path to AGI from here onwards?

750

:

Yeah, for sure.

751

:

That'd be a fascinating dinner.

752

:

Make sure it comes like that.

753

:

Awesome.

754

:

Well.

755

:

Desi, thank you so much.

756

:

think we can call it a show.

757

:

I learned so much.

758

:

I'm sure my listeners did too, because as you showed, is a very, this is a topic that's

very much on the frontier of science.

759

:

So thank you so much for all the work you're doing on that.

760

:

And as usual, I put resources and a link to your website in the show notes for those who

want to dig deeper.

761

:

Thank you again, Desi, for taking the time and being on this show.

762

:

Thank you so much for having me.

763

:

It was my pleasure.

764

:

This has been another episode of Learning Bayesian Statistics.

765

:

Be sure to rate, review, and follow the show on your favorite podcatcher, and visit

learnbaystats.com for more resources about today's topics, as well as access to more

766

:

episodes to help you reach true Bayesian state of mind.

767

:

That's learnbaystats.com.

768

:

Our theme music is Good Bayesian by Baba Brinkman, fit MC Lass and Meghiraam.

769

:

Check out his awesome work at bababrinkman.com.

770

:

I'm your host.

771

:

Alex and Dora.

772

:

can follow me on Twitter at Alex underscore and Dora like the country.

773

:

You can support the show and unlock exclusive benefits by visiting Patreon.com slash

LearnBasedDance.

774

:

Thank you so much for listening and for your support.

775

:

You're truly a good Bayesian.

776

:

Change your predictions after taking information in.

777

:

And if you're thinking I'll be less than amazing.

778

:

Let's adjust those expectations.

779

:

me show you how to be a good Bayesian Change calculations after taking fresh data in Those

predictions that your brain is making Let's get them on a solid foundation

Chapters

Video

More from YouTube