Help us become the #1 Data Podcast by leaving a rating & review! We are 67 reviews away!

Emily Oster is a Harvard trained economist who built a career on making decisions when the data is bad or missing. I asked her how she does it.

💌 Join 33k+ aspiring data analysts & get my tips in your inbox weekly 👉 https://datacareerjumpstart.com/newsletter

🆘 Feeling stuck in your data journey? Come to my next free "How to Land Your First Data Job" training 👉 https://datacareerjumpstart.com/training

👩‍💻 Want to land a data job in less than 90 days? 👉 https://datacareerjumpstart.com/daa

👔 Ace The Interview with Confidence 👉 https://datacareerjumpstart.com/interviewsimulator

⌚ TIMESTAMPS

00:00 – Let the decision lead the data

04:03 – Instagram isn't evidence

08:39 – What to do when there's no data

10:57 – Your family is a small business

14:33 – Right decision vs right process

30:21 – The New York test score error

39:00 – Rapid fire myths

🔗 CONNECT WITH EMILY

📚 Expecting Better: https://a.co/d/00ExVFqN

📚 Cribsheet: https://a.co/d/09rGjGjs

📚 The Family Firm: https://a.co/d/08RaAOBh

📈 Emily's artifact: https://claude.ai/code/artifact/ecaef324-0efe-401a-9fca-d5d2816e88ee

📊 Education Substack: https://substack.com/@statetestscoreresults

🤝 LinkedIn: https://www.linkedin.com/in/emilyoster

📸 Instagram: https://www.instagram.com/profemilyoster/

🐦 X: https://x.com/ProfEmilyOster

💻 Website: https://parentdata.org/

🔗 CONNECT WITH AVERY

🎥 YouTube Channel

🤝 LinkedIn

📸 Instagram

🎵 TikTok

💻 Website

Mentioned in this episode:

October Cohort of The Accelerator Starts on October 12th

Land your first data job ASAP with The Accelerator. And for this cohort only, get a discount PLUS our premium job board software PLUS our live AI workshops on how to analyze data Learn more here: https://datacareerjumpstart.com/daa

https://datacareerjumpstart.com/daa

Live "How to Become a Data Analyst" Webinar

Come learn the EXACT roadmap to landing your first data job: https://web.datacareerjumpstart.com/webinar

That's Emily Oster, a Harvard

trained economist who is the expert

2

on making data-driven decisions

when there's not always good data.

3

And I'm a huge fan of her data-driven

parenting books, and they've helped me

4

raise my own kids and parent them in

a way that I feel really comfortable.

5

Today, she'll give us the key, the method

to actually making good decisions even

6

when there's poor data, there's not

data, or we're in a lot of uncertainty

7

and there's a lot of unknowns.

8

By the end, you'll have a great framework

for making great decisions like a data

9

analyst, even if you're not one already.

10

So let's go ahead and get into it

11

Emily Oster is the founder and

CEO of ParentData, a professor of

12

economics at Brown, and a three-times

New York Times bestselling author.

13

Emily, welcome to the Data Career Podcast.

14

Thank you for having me.

15

Super excited to have you.

16

I am a big fan.

17

I have the, the books right here.

18

If you guys haven't checked out- Amazing

19

Emily's books before,

definitely, um, check them out.

20

They are amazing.

21

Um, but we live in a really

interesting time, Emily.

22

Uh, there's, like, so much infor-

misinformation going around, um,

23

in parenting and in everything

in politics and finance.

24

Um, you know, everyone's trying

to tell you how you should parent

25

your kids and what decisions

you should make for your kids.

26

Uh, so my question t- for you,

is it possible to make good life

27

decisions in a world where we're

constantly bombarded by different

28

opinions and different data sets?

29

I believe yes, uh, but I think it

requires us to think about the structure

30

of our decisions rather than just

ask the question what the data says.

31

So a lot of times people will come to

me and they'll be like, "Okay, well,

32

just tell me what the data says."

33

It's like, that's not always that helpful

a question, and if your approach to

34

decision-making is to just, like, see

the last piece of data and, like, make

35

a decision based on that, you aren't

necessarily gonna make good decisions,

36

and I think part of what makes our current

information environment so challenging

37

is that people are constantly getting

bombarded with data, and every time they

38

see a new piece of data they're, like,

not necessarily ready to incorporate

39

it into their decisions in a smart way.

40

So I think the answer is yes, we

need data, and we can make good

41

decisions, but we have to have the

decision-making sort of lead the data.

42

So I would tell people, like, you

need to wait until you're ready to

43

make a decision, and then think about

what your choices are, structure the

44

decision, and then you get the data

that you need to make the decision and

45

then make the decision based on that.

46

But I think it's, it's too hard to Only

use data, I guess if that makes sense.

47

Yeah, for sure.

48

I think, I think it's also interesting

when we're talking about data to maybe

49

specify what we're talking about.

50

'Cause, you know, some of the topics that,

that you take on, um, like for instance

51

Encryptshe, is, you know, is breast best?

52

Like, is it actually good to…

53

Is it better to breastfeed your,

your baby, or is bottle feeding okay?

54

Um, another, you know, one of the other

things you tackle is vaccinations.

55

Do vaccinations, you know, cause autism?

56

And I think a lot of people, maybe for

people who are listening to this, they're

57

data nerds and, you know, they're able

to, you know, maybe go out there and

58

try to find some data on, you know,

autism rates and vaccination rates, and

59

maybe, you know, put together some sort

of a statistical analysis to do so.

60

But I think a lot of people are getting

their data from, like, Instagram posts.

61

Yeah.

62

Um, so, like, how do you try to navigate

the, the world where it's, like, a lot,

63

where a lot of data is presented to us

in, like, an Instagram post or something

64

that's, that's maybe not very structural

and, and hard to interpret in the moment?

65

Yeah, I hate data from Instagram posts

because it's always like, "Here's a

66

study that shows blah, blah, blah, blah."

67

And it's like, well, what, like,

is it the only study of this topic?

68

Is it the biggest study of the topic?

69

Is it the best study of this topic?

70

Is it some random thing from 1987

that you pulled out of, like, the

71

journal of, like, made-up results?

72

Which is usually the answer.

73

And so i- Er, data is it's like I get so

frustrated because I think we, we really

74

need to prioritize the best data, but part

of what is very challenging for people

75

is it can be hard to know what that is.

76

And there is a fair amount of

training that goes into the

77

question of like, is this good data?

78

Is this less, less good data?

79

Um, so I guess I would say it

is never a good idea to make a

80

decision about something based

on a single Instagram post.

81

If you are in a position to need to

know whether some relationship is

82

true, are vaccines causing autism,

for example, you need to step way back

83

out of Instagram or out of TikTok or

whatever it is and figure out what are

84

some sources you can go to that are

gonna give you a better, more nuanced,

85

more thoughtful answer to that question.

86

And there are a few things people can

look for in, you know, what makes a

87

good data set, things like, is it big?

88

Is it likely to be randomized?

89

You know, things like that, and

that's, that's kind of the core.

90

But, you know, a single study

says and somebody puts it in a

91

carousel on Instagram, that's a

crappy way to learn about data.

92

That- that's…

93

I mean, that's unfortunate.

94

I wish we could always just like

trust what we, what we see online.

95

Um, but obviously we, we can't.

96

Cannot.

97

That's one of the things that I think,

um, you do a really good job in, in

98

Crib Sheet especially of like, you

know, we're, we're debating, we're

99

debating something like, uh, should

we like co-sleep with our babies or

100

should we sleep train our babies?

101

Um, you know, you pull up like

all these different studies

102

that have been done on that.

103

And some of the studies like that, maybe

let's just say for example, that say,

104

oh, you know, you know, co-sleeping is

like really good for your baby actually.

105

Um, you might throw…

106

Correct me if I'm wrong, but like

you might like throw that study out

107

the window and kind of ignore the

results because maybe it's not a

108

large sample size or maybe it's not

a diverse sample size or, or maybe

109

like the actual experiment was wrong.

110

So even though the results say something,

like you're not necessarily one to

111

just trust, uh, the results kind of

randomly from a, uh, an experiment.

112

Is that kind of correct?

113

Is that kind of your way of thinking?

114

Yeah, definitely.

115

I think that a lot of what distinguishes

the way that I approach sort of large

116

corpuses of data from the way that you

would and it sort of, um, that other

117

people would perhaps, is that I am much

more willing to say, okay, let me find the

118

best studies here and base our conclusions

on the best studies rather than just like

119

every study should get their, their voice.

120

Like some studies don't deserve a voice.

121

Um, and I think that is especially

true when we're outside of

122

the randomization space.

123

So in the parenting, like health, et

cetera space, there is a huge amount of

124

what we're told that is like we're just

comparing people who do one thing to

125

people who do another thing, and those

people are really different on like a

126

billion dimensions, and we're attributing

it to the one topic that we're studying.

127

And that kind of evidence I will

almost always say like, just forget

128

it Like just put it in the trash.

129

And that is actually a place where I

really differ from a lot of even, you

130

know, people who I think have a lot of

training and are in, you know, serious

131

like professors because there are people

who will tell you, "Well, okay, but

132

we have so many studies of something.

133

Like there are so many observations."

134

But if it's not causal, it doesn't

matter how many observations you have.

135

And so I'm, I'm very interested in how

we can have good data, really excellent

136

data that lets us make causal statements.

137

And if we have a study that isn't

gonna let us make causal statements

138

about something we wanna make causal

statements about, I just think

139

we should throw it in the trash.

140

That's all.

141

A- a- and, and I think that's important

for people to realize because, um, I

142

think there's some group out, out there

who don't really take any scientific

143

studies and, like, white papers.

144

Like, they don't ever look at that.

145

Like, they're only in the

Instagram world, you know?

146

Uh, maybe, maybe looking

at aggregations of things.

147

And there's some people who, who

maybe are a lot more, like, prone to,

148

like, "Oh, I really trust science,

um, a- and studies," but it's always

149

important to look into those studies.

150

Um, I'm curious, like, there's not

always a study for, for everything.

151

Right.

152

Um, so, like, what do we do when

we need to make an im- an important

153

decision, we wanna be data-driven

in our approach, but, like, there

154

isn't good data or there's no data?

155

What do we do then?

156

Yeah.

157

So I think the f- first, that's

very hard, and we have to first…

158

I think first it's like there's a radical

acceptance of just saying, like, "Hey,

159

I'm gonna have to make a choice here.

160

There is no option to, like,

wait until the data is better.

161

There just, I have to move

forward with one thing."

162

And if we paralyze ourselves w- with

the view that, like, we can't make

163

decisions until there are better data,

like, the decision will be made for

164

you in some direction by you waiting.

165

So just to recognize, like, sometimes

you'll have to make decisions

166

under uncertainty, and that is

unfortunate, but it is the way it is.

167

I think the second thing I would

tell people is in almost all of those

168

settings, it's not important, right?

169

So if something were really…

170

It's not uniformly true, but if

something is really important,

171

and I talk a lot about parenting,

but, like, really important in

172

parenting, like really, really

matters, you will see it in the data.

173

Like, the things that we know really

matter, like poverty, whether your

174

kid has a stable place to sleep,

whether they have enough to eat, those

175

things really show up in the data.

176

The correlations are really, really big.

177

We have good causal evidence.

178

The kinds of questions where people say,

"Oh, I wish I had better data on this.

179

You know, is it better to enroll

my kid in travel soccer or in,

180

you know, travel lacrosse?"

181

Or where, like, there's no data

on that, but you know what?

182

It's not important.

183

And so I think just, like, dialing

down and asking ourselves, "How

184

likely is it that this thing matters?

185

Given everything else about my family,

like, is this likely to be an important

186

decision for my kids' outcomes?"

187

And most of the time you're gonna say no.

188

And then that actually makes the

decision-making much easier because

189

it allows you to focus on all the

things that do matter for good

190

decision-making, like how will

this logistically affect my family?

191

How much does it cost?

192

Whatever are the things that really

should go into that decision.

193

So I think just reminding yourself you

gotta make decisions when it's uncertain,

194

and a lot of things are not important.

195

And it's okay to say, like,

"This probably isn't important.

196

Maybe it matters a little bit in

one direction or another, but on

197

the whole, it's not the thing that's

gonna break or break my kid," which

198

almost there is nothing like that.

199

I think that's such an interesting

approach, and, uh, I'm just thinking, you

200

know, about the people who are listening.

201

Uh, like so many of us are data

engineers, data analysts, data

202

scientists, and it's like our whole life

is, you know, helping businesses make

203

better decisions with, with the data.

204

Um, and so I think it's hard for me, like

as a data nerd, to be like, oh, you know,

205

sometimes the, the data doesn't matter.

206

Like, or, or even maybe, maybe the

choice, um, doesn't really matter.

207

Um, but, but we're- I, I would…

208

So let me tell you, I think this actually,

there's such a strong, uh, like data

209

analyst parallel here that I would make

for people, which is like, you know, the

210

difference between like great success

and not great success in your business

211

is like did you launch the right product?

212

You know, did you…

213

Like there's some big strategic

decision that is happening, you know,

214

usually above the heads of everyone

and like where somebody at the top

215

is making a big strategic play in one

direction or another, and that's gonna

216

determine like whether the business

is successful or not successful.

217

The job of the data analyst, and I

do this like for my own business, is

218

to be like in the weeds and be like

can I get 1% more if I like send the

219

email in this way or this other way?

220

Or like can I optimize

the pricing in this way?

221

Like let me do an AB test on

this, that, and the other thing.

222

And like those things are really important

for your business, but they're not

223

important like the big strategic question.

224

In the background, they're kind

of optimizations on the margin.

225

These kind of choices that we sometimes

get obsessed with with our kid, they're

226

optimizations on the margin, and that

margin for parenting is really small,

227

and there's so much noise And so

thinking about like, if only I could

228

optimize this tiny thing, it's like,

well actually that's not important.

229

Like, that's one tiny AB test in

one place and like if you get it

230

right, you, you don't get it right in

parenting, it's, it's a lotta noise.

231

So I don't know.

232

That's, that's often how I

think about the parenting piece.

233

I mean, you bring up a good point because

it's like if, should we launch product

234

A or product B, and let's say we choose,

you know, product A, but actually product

235

B was the right choice, but there was

just not data to make that decision.

236

You know, what we're optimizing, let's

just say like a subject line on an email,

237

and you know, you AB test it and you

find, oh, we get a lot more conversions

238

with, you know, this, this subject line.

239

It's like well, the bigger decision

was really we shoulda gone product B.

240

Like, the, the amount of money

you can make on the- Totally

241

you know, having the right subject

line really is probably dwarfed by

242

actually launching the right product.

243

Which that's, that's an

important thing to realize.

244

I think it's important for people

in, in, you know, in their p-

245

careers to realize that as well.

246

That it's like we're, we can analyze

data, um, but we're, we're always trying

247

to do so for business purposes and,

you know, m- move the business forward.

248

So it, whether that's for our, our kids…

249

And, and even like you said, like

travel lacrosse versus travel soccer.

250

Let's say that there was data on that.

251

It's like, well who's to say that

your kid is like the average-

252

Totally … you know, it, it- Soccer

kid … doesn't take into account.

253

Yeah.

254

Yeah.

255

Yeah.

256

It's like- No, it-

257

you don't have data on your kid.

258

Totally.

259

And I think in our, in our parenting

decisions, of course like everything

260

in our family is so much more

complicated than like the subject line

261

of the email that you're optimizing.

262

And so every one of these decisions has

some other decisions associated with it

263

which may actually be more important.

264

So we can get very like laser

focused with our kids on

265

thinking, you know, well let me…

266

What is the right activity or choice or

school to like optimize, you know, some

267

outcome metric that we have attached

to our kid, without stepping back and

268

saying, you know, really what we're

trying to optimize is like that our kid

269

is a, you know, happy, productive adult

who likes us and comes home for meals.

270

And like that's actually a much broader

optimization than like, you know,

271

are we gonna Achieve Junior Olympic

status or whatever, which you won't.

272

One of the things I, I really like, uh,

along these lines and in the, in the book

273

is, you know, you mentioned that parenting

you're gonna have a million decisions,

274

and there's no way to guarantee that

you're gonna make the right decision.

275

In fact, you're probably gonna make

the wrong decision quite often.

276

Um, and instead of optimizing for making

correct decisions, you talk about,

277

um, making decisions through, like a

framework and, and having education

278

around the decision that you are making.

279

Can you walk wa- about the difference

between, like, making right decisions

280

and making the decision the right way?

281

Yeah, absolutely.

282

So, think the most important distinction

is one of those things you can

283

do, and the other one you cannot.

284

So you can never guarantee that

you will make the right decision.

285

It's just, like, not,

that's not available to us.

286

But we can say ex ante that we approach

the decision the right way, and so I

287

talk about, you know, being structured

in how we make our choices, and starting

288

by really outlining what our choices are.

289

So being clear on the

choice that you're making.

290

People often will ask me, you know,

"Well, should I do this or not?"

291

You know, "Should I send my

kid to this school or not?"

292

It's like, well, or not is not an

available schooling option, so, like, you

293

better tell me what's on the other side of

that, because you're never gonna be able

294

to compare something to, like, the vast

array of other things on the planet Earth.

295

So you gotta think about

what your choice is.

296

You've gotta collect the

information that you need.

297

You need to then actually force yourself

to make a decision, and I think that's

298

the hardest part, because in a world in

which we want to make the right decision,

299

we can get, like, paralyzed by the

realization that we cannot guarantee that.

300

And so in order to work ourselves

past that, we often have to really

301

put in place, like, okay, I am

gonna sit down and make a decision.

302

Like, I, this is the date.

303

I'm putting it in my calendar.

304

I'm scheduling it.

305

Like, this is the time we're gonna

make whatever is this choice.

306

Because otherwise you can just,

like, spiral and spiral and spiral

307

forever and never actually make any

choices, and then usually the world

308

will choose for you in some, in

some way if you don't do anything.

309

Um, and I, I think the, the value of

having this kind of structure to a

310

decision process is that on the other

end of it, you can be confident, again,

311

not that you made the right choice, but

that you made the choice the right way.

312

I think that's quite

protective for people.

313

I will say one other thing,

which is I tell people, after

314

you have made this choice, you

should make a plan to revisit it.

315

You know, there are some choices

we can never revisit, you know?

316

W- should I or should I

not have a second child?

317

We- well, once you choose that,

you've pretty much, that's,

318

you're pretty much committed.

319

We're not revisiting that.

320

But many of the choices we

make you can revisit, right?

321

Like, I sent my kid to this school,

but I can choose another school.

322

I sent them to this activity, but I

could choose another activity, like,

323

later, or we could not do this activity.

324

And I think we owe it to ourselves to

plan this kind of revisiting of the

325

choices that we make, because if we

don't, then we will never revisit them.

326

And if we plan to revisit them,

we're much more likely to do so.

327

So much good to unpack there.

328

Um, one, one thing I wanna zero in on

is, like, you, you mentioned if you're

329

deciding between A and B, and you're

like, "I don't have enough data to do

330

that," um, sometimes we don't make the

choice, and that itself is a choice, is

331

what it sounded like you were saying.

332

And oftentimes it's w- the worst

choice of the three, it seems like.

333

Like, it's even worse than deciding wrong.

334

Yeah, I think it, it sort of like, it,

it, it robs you of the opportunity to

335

improve the choices you have, right?

336

So this is like, decisions are

particularly hard if neither

337

choice is really something we want.

338

If we're sort of like, "Well, both of

these choices are kind of poor," we're,

339

we're really reluctant to make them

because there's something very aversive

340

about choosing something you don't want.

341

But if you don't on purpose choose,

you will end up in one of the, in one

342

of the s- the branches, typically.

343

And if you haven't chosen it,

you won't have had an opportunity

344

to make it as good as possible.

345

Like, among the bad options, how can

I make this option the least bad?

346

And that's, uh, that's missed out

on if you just decide to ignore

347

the problem and forget about it.

348

Uh, that's a good lesson for me to learn,

because I am the king of doing that,

349

and, uh, that stings a lot of the time.

350

So, uh- … I'm gonna try

to do better with that.

351

Um, I'm curious, like, so I'm used

to data, quote, unquote, as, like, a

352

table in Excel or, like, in, you know,

in a SQL database type of a thing.

353

Um, but like you said, we don't always

have data to, to make decisions.

354

One of the things that you talk

about in your other book, The

355

Family Firm, is, like, other

ways that you could potentially

356

get data that aren't necessarily

from, uh, like a, a spreadsheet.

357

And, you know, maybe they're not as

high integrity as, as spreadsheet data.

358

Um, but they're still

valuable in making decisions.

359

Um, so, like, one of the things you

mention about is, you know, information

360

about your personal circumstances.

361

And, and you mentioned earlier,

like, you know, if you're making

362

decision A or B, like, which one

are you actually going to like more?

363

Even, even if one's more optimal

than the other, like, which

364

one are you going to like?

365

And the other thing that you mentioned

about is talking to, to others.

366

I just am curious to hear your thoughts

on, like, when you're getting data

367

that isn't necessarily, like, tabular

data, really high quality data, um,

368

it's, it's more, like, around you

data, what, what are you looking for?

369

What signals are you looking for, and

how can you know if you can trust it?

370

Yeah, so I think really here I'm talking

about getting i- e- information, I think.

371

Look, data is just pieces of information.

372

So I'm really telling you, like get

some data on how you feel about stuff.

373

And I think for many people who are

kind of like us, actually maybe you

374

wanna put that in a spreadsheet.

375

Like I'm not averse to the idea that

like you should collect data on your

376

preferences and logistics and constraints

in the same way that you would imagine

377

collecting data on, you know, k- test

score outcomes or, or whatever it is.

378

But I think a big piece of this

is, is kind of th- interrogating,

379

like if I make this choice, what,

what are the actual implications?

380

You know, both how much am

I gonna like my day-to-day?

381

Really think about it.

382

Like if I get up and I face, you know

… Le- let me put a concrete example in it.

383

So a lot of people talk, talk to me

about like choosing between preschools.

384

Like I have this preschool and

it's right close to my house, but

385

it, you know, isn't very fancy.

386

N- only half the teachers have master's

degrees or whatever, or there's this

387

preschool that's like 40 minutes away

and it's like super fancy, right?

388

It's like how do I think about that?

389

Okay, and so one piece of that

data is, you know, how much do

390

we know about differences in

preschool outcomes or whatever.

391

And another piece of that data is

like how do I feel about commuting

392

40 minutes each way with my kid?

393

You know, what's that gonna do to my day?

394

Like what's my day … That's

a piece of data.

395

Like what is my day gonna look like?

396

How are we gonna manage this logistically?

397

How am I gonna feel about, am I gonna

fight with my partner about this?

398

Like are we gonna argue every day

about who is gonna take their kids?

399

That's a piece of data.

400

You know, so there's a bunch of

stuff in there that you want to put

401

in your decision making, even if it

isn't numbers in a spreadsheet, but

402

it is information in a document.

403

Uh, and another piece of that is

asking other people, you know,

404

what do they think about it?

405

How did they experience this?

406

But like-

407

It's…

408

For people who love data and evidence,

there's an aversion to the idea of, like,

409

going with your gut, and people will talk

about this as like those are two choices.

410

You could go with the data,

or you could go with your gut.

411

My view is like the data's not bossy.

412

It's not gonna tell you what to do.

413

It's only an input to decision-making,

which also needs your preferences.

414

People say, "I'm going with my gut."

415

What they really mean is,

"This is the thing I want."

416

Okay.

417

But you can actually bring those things

together, and that's gonna be better

418

than your data or your gut alone.

419

I'm, I'm laughing because I'm literally…

420

Like, we're putting, uh, our,

our daughter in preschool.

421

Literally went through that exact…

422

Do we go to the s- the preschool we think

is, like, better, quote unquote, you know,

423

but it's further away, or do we just kinda

do one that, that's close by and near?

424

So, uh, I'm laughing 'cause I went

through that, that exact analysis- Yeah

425

recently.

426

I will tell you, we went to the, we…

427

The better one that's further away.

428

It's not 40 minutes away.

429

If it was 40 minutes away, I don't

think I would be able to do that.

430

But now that we're in the process, and,

like, my wife is the one who takes her

431

the majority of the time, it's like,

"Oh, we don't really like this commute.”

432

So, you know, next year or next

kid or next semester, I don't know

433

if we're gonna do this or not.

434

Um, so sometimes you make a decision

the best you can, and you make- Yeah

435

… maybe a wrong decision, but it's just

another data point where it's like, "Oh,

436

actually, I don't like being in the car

for-" Yeah "… X amount of minutes a

437

day" We learned this- And we're done

438

totally, and then I think that's

why you need the, the other step

439

because if you don't plan to revisit

that choice, then you, you're not

440

gonna do it 'cause you're not gonna

wanna admit that you made a mistake.

441

And it's like it's not

that you made a mistake.

442

You did a trial, and, like, maybe

that trial turns out like, well,

443

we learned something that we wanna

do something differently next time.

444

A- and I think it's important that not

only we do this with, with preschool, but

445

when I worked for ExxonMobil, I was a data

scientist there, and one of my jobs was

446

to make machine learning algorithms to

predict how much gasoline we should buy

447

in each, you know, one of the thousands of

different stores we have across America.

448

And I, I built, you know, a machine

learning algorithm that, that was

449

the most accurate we could make it,

um, to predict, you know, gasoline.

450

But it wasn't like, hey, the, the

gasoline prediction that my machine

451

learning model puts out is what we order.

452

That, that number goes to, uh, a trader,

a buyer of gasoline, and they can

453

totally ignore my number, and they can,

you know, use my number as, as aid.

454

So it's so interesting that, like, you

know, these preschool decisions are

455

kind of the same framework that, you

know, multi-million, billion, you know,

456

in- industry decisions are making.

457

I'm sure you've seen that kind

of with, with your research and,

458

and, and your analysis as well.

459

Totally.

460

And I think we under even…

461

It's, it's sort of interesting always for

me to watch people who are so good at this

462

approach to their job, who are so, like…

463

And then I'm like, "Well, can't you…"

464

Like, you should just be

porting that into your life.

465

Like, it's the same thing.

466

Like whe- you know, when you are running,

when you are married to someone and you

467

have, or you like have a partner and

you have kids together and you have two

468

jobs and whatever, like you're running

a small to medium sized enterprise.

469

And there actually are like quite a

lot of tools from how companies do

470

that, that I think will make this

easier for, for people, even though

471

it's like a really weird thing to say.

472

And when you tell people that they're

like, you know, they're like, "Well,

473

do you boss your husband around?"

474

It's like, "Of course I boss him around,"

but that's not because of the, it's

475

not because of this enterprise idea.

476

That's awesome.

477

And I love that.

478

And that's, you know, one of the, the

things you talk about in, in The Family

479

Firm is like, we should, we should really

kind of organize our family, uh, like

480

we r- organize teams and, and businesses

in a fun way, not like in a boring way.

481

Yeah.

482

But like- Fun way … we

should have objectives.

483

We should have goals.

484

Like, we should, you know, make decisions

that are for the optimal happiness

485

and health o- of our family, which

I think I've, I've been reading it.

486

I've really been, um, enjoying that.

487

Um, I'm curious, so you know,

we talked about like making

488

data-driven decisions as, as parents.

489

I'm curious, like where else in our

personal lives, um, that we can make

490

data-driven decisions and kind of

adopt the approach that you've taken

491

in, you know, in breastfeeding and in,

you know, sh- what kids, where, where

492

should we send our kids to school?

493

What other places in our life can we

live data-drivenly other than parenting?

494

Yeah, I mean, I think the other obvious

one that people like is health, um,

495

is sort of like there are a lot of

health decisions that you have to make.

496

Uh, and there's a lot of data on those.

497

Uh, and I think it has many of the

same issues that parenting has.

498

You know, there's some of the

data is better than others.

499

Um, and You know, we, you have

to make decisions that take

500

into account your constraints.

501

Um, and I actually think this is

a place for me where we are…

502

A mu- much of the discourse

misses the idea of constraints.

503

We're like really good in the health

space about talking about like how

504

to optimize, like, you know, let's

track every da, da, da, da, da.

505

Like, you know, how do you like get

all of your n- numbers to be exactly

506

optimal in these various ways?

507

But without helping people sort of see

like, okay, well, you probably don't

508

have 17 hours a day to like fully

optimize a 27-step life protocol.

509

Uh, and so how do we incorporate the

data with the constraints and ask,

510

you know, what are, what are the

most effective things, uh, to do?

511

And I think the other place is just how,

in like sort of general, like how do we

512

choose our jobs and how do we, you know,

operate our like professional lives?

513

Um, but I think health is the, health

is the other obvious space for me.

514

Health is a really interesting one,

um, because obviously, like, it-

515

it's … If you don't have health,

you have nothing in your life, right?

516

Right.

517

'Cause, like, I think we all have

known someone that's, that's lost

518

health, and we just see, you know,

how much of a detriment to life that

519

is and how, how difficult things are.

520

So that makes sense to optimize it.

521

Um, just, like, a, a concrete example

of that is, and I know you, you talk

522

about, um, this in, in your books.

523

Um, but I got diagnosed with ADHD last

year, and I had a, I had a decision.

524

It's like, do I try, you know, after

trying, you know, six months of

525

non-medication ways to, to, to solve the

problem, like, do I try medication or not?

526

Yeah.

527

Um, and you know, one of the things

that I do is, like, I pretty much

528

constantly wear, uh, an Apple Watch.

529

And so I've been taking, you know, ADHD

medicine for, for almost nine months now.

530

And one thing I've seen is my

resting heart rate has, has risen,

531

like, five beats per minute.

532

Yeah.

533

And it's like, okay.

534

Um, that's, like, a side

effect of taking ADHD medicine.

535

It's like, do I, do I like that?

536

Do … Is, is the health risk that with

my heart worth the effects of m- you

537

know, maybe me getting more work done

or maybe me being a more patient parent?

538

Um, those types of things.

539

And it's hard because it's like there's

not really all that data out there

540

that can support … Or, or maybe

there is a bunch of data out there,

541

but it's like which one do I trust, and

how do I apply it to my personal life

542

and my business and my family life?

543

Um, so I guess making decisions

under uncertainty with

544

health definitely makes- Yeah

545

makes a lot of sense.

546

Yeah, and un- under uncertainty, and

they're really also under constraints.

547

Like, you're, like, really what

you're describing is a constraint,

548

which is like you, like, if you do

this one thing, it has this effect.

549

But y- like, you're … It's trade-off,

uh, and we're not that good at trade-offs.

550

And a lot of the health messaging

in particular sort of doesn't like

551

to acknowledge the existence of

trade-offs, so they're just like,

552

"Do all 4,000 of these things."

553

And it's like, okay, but I, I, I can't do

that, or it's, like, literally impossible

554

to do two of these things at the same

time, and so now I have to pick one.

555

And, you know, people ask me,

like, "Should I sleep or exercise?"

556

Like, I only have 30 minutes.

557

I have to pick sleep or exercise.

558

Like, which thing is better?

559

And, uh, that's, that's a hard question.

560

What's the answer?

561

Hmm, kind of, kind of depends how much

sleep you're getting, but probably sleep.

562

Yeah.

563

I, and ob- obviously there's so

many factors that, you know, that

564

go into your life, and it's hard

to, to make a blanket statement.

565

It's funny that you're, you're

mentioning this, 'cause my other job

566

at ExxonMobil, my other problem that

I solved at ExxonMobil, uh, and I

567

didn't solve this problem on my own.

568

We worked as a really big team to do this.

569

But we, we made mathematical models

of the entire refinery, um, which

570

was, like, 140,000 equations.

571

Um, and we were trying to optimize, you

know, how much money the refinery can

572

make with all these different constraints.

573

Like, we can't…

574

We need to make sure that our, our

pollution's abov- uh, below this level.

575

You know, our, our tower

can only take this many-

576

Yeah … barrels of crude every day.

577

And that was a really

hard problem to solve.

578

Yeah.

579

Um, and we knew, we knew

all the, the math behind it.

580

It's like you can't really do that in

your life, 'cause, like, you can't really

581

model life outcomes, I don't think.

582

Nope.

583

Um, maybe you can.

584

I don't know.

585

No, and you have…

586

And, and of course, then when people try

to, they come up with, like, crazy things.

587

Somebody sent me a paper the other

day which, in which these people,

588

like, tried to assign a number of

minutes of life to, like, each food.

589

So, like, if you have one

Diet Coke, it costs you, like,

590

this many minutes of life.

591

But it's like, that's a cra- like,

first of all, that's bananas.

592

Like, you definitely can't do that.

593

It's all of the data is from correlation.

594

It's not causal, whatever.

595

But it also just, like,

didn't make any sense.

596

It was like a Diet Coke costs you 12

minutes, but, like, a peanut butter

597

sandwich gains you, like, 33 minutes.

598

And it's like, okay, well, if

I eat them together, can I get

599

fif- like how does this work?

600

But it was so, like, so much in the

space of people just want an answer.

601

They wanna know, like,

okay, how much is…

602

Like, what's the cost of this Diet Coke?

603

And the answer is, like- We don't, you

know, we don't have it, probably zero.

604

Uh, or have it with a peanut butter

sandwich, and then I get to negative 17.

605

That's awesome.

606

I love that.

607

That's, that's very cool.

608

Um, okay.

609

I saw something really cool that you,

um, posted on your Twitter recently

610

and your Substack, and we'll make

sure to have a link to your social

611

in the description down below.

612

Um, but it was a really cool analysis

you've done recently on New York education

613

data and kind of like their testing data.

614

Um, and one of the things you

actually published that caught

615

my eye was a Claude artifact.

616

Yeah.

617

So for those who are unfamiliar

with Claude, it's basically

618

like ChatGPT, but it's from a

different company called Anthropic.

619

Um, I really like it for doing things

like data analysis, and it creates

620

these things called artifacts, which are

basically, you can think as like a, a, a

621

published something, a URL that goes to

some sort of a page that has text on it.

622

And in your case, you were analyzing

data, so it had text and graphs and

623

different analyses and like that.

624

Um, and I wanna talk about the

New York- Yeah … education

625

study, uh, study that you did.

626

But first, I want to kind of walk me

through your, like, data pipeline.

627

Like, how, how did this, like, come to be?

628

Like, where are you getting your data?

629

How are you analyzing it?

630

How are you publishing it?

631

I was really curious about that, if

you don't mind sharing maybe, like,

632

a high, high, uh, view of that.

633

Sure, yeah.

634

So in, in that case, actually, the

key to that entire analysis is one of

635

the projects I d- I do is something

called the Education Data Center,

636

uh, which is a, a project where we,

uh, try to clean and organize all

637

the state-level test score data.

638

So if your kids are in, you know,

public school in grades three through

639

eight in the US, they will take,

uh, math and ELA tests every year.

640

That's like an important

part of accountability.

641

Uh, but the state's data

is, like, a hot mess.

642

Like, every state is issuing

it in a different thing.

643

If you wanna have the data from Montana,

it's 3,000 separate spreadsheets.

644

Like blah, blah, blah, blah.

645

And so one of, one thing I really

care about data transparency.

646

Uh, and so in this project we,

like, download all of this stuff,

647

and we have, so we have like a, a

website where you can sort of get

648

all the microdata for, for this.

649

And I mention that because that's

a sort of core, like, backend

650

pipeline for a project like this.

651

On that particular project, there's

a, this thing that happened in New

652

York with the test scores, which we

can talk more about, but where, like,

653

basically somebody called me, some

reporter called, and they were like,

654

"Here are the, you know, here are the

test scores that are gonna come out.

655

Like, what do you think?"

656

And I looked at them, and I

was just like, "They're wrong.

657

Like, I don't like, I don't know

what to tell you, but, like,

658

data doesn't look like that.

659

Like, somebody made a

mistake probably last year."

660

And then I got really, like- exercise.

661

I, like, I really love the, the

piece of data where you try to,

662

like, learn what's going on.

663

Just like, I just wanted

to know what's going on.

664

Like, I couldn't, like, let it go.

665

And so then my data pipeline is, you know,

in the, in the back end, I'm basically

666

using Claude with an API pulling down

this, this raw data and kind of writing

667

code in Python to, like, figure out,

try to figure out what's going on.

668

And this is a place where the, these AI

tools and sort of Claude in particular

669

has really changed how quickly I could do

something like this because, you know, on

670

the back end, I could have written this

code in Stata on my own and so on, but I

671

probably would not have had the bandwidth

to do it without a kind of LLM tool.

672

Uh.

673

It was awesome, first off.

674

Uh, it was super cool.

675

It was so fun.

676

It was like, it was so cool.

677

Yeah, so basically what, what I'm

hearing is like you, you obviously

678

know how to do this analysis.

679

You could obviously do it from scratch.

680

And I love the idea of from scratch.

681

It's like none of us are actually

doing this by hand and paper.

682

Right.

683

Like, that's probably from scratch.

684

No.

685

So it's like, oh, like-

My dad was an economist.

686

He used to like, I think, do this by…

687

They would like multiply the

matrices, but that was a while ago.

688

See, but that, that's kind of

my point, is it's like, oh, and

689

then R came out, and then Python,

or and then, then Stata- Right

690

or whatever.

691

Uh, you know, and it's like now we just

have Claude and ChatGPT, which I just see

692

as like a new tool, like you said, that

enables this- Yeah … type of analysis.

693

Um, and it's like I don't think we

could've taken a random Joe off the

694

street and, you know, had them create

this analysis that you created.

695

I don't think this analysis, the

AI could've created on its own.

696

Um, so it's cool to get like

a little bit of glimpse on, on

697

how you're using AI to do that.

698

Um, so that, that's very cool.

699

Um, I do wanna get into like the,

the details of, of, of what happened

700

in this, in this, I wanna call

it a study, but it's not a study.

701

These test results.

702

So basically, um, if I'm understanding,

uh, correctly, the test results…

703

L- l- let's make it as simple as possible.

704

The test results were around a certain

level, and then the next year they jumped

705

up like 10%, from like 43 to like 52%.

706

And maybe we'll pop up the, the

graph on the screen that your

707

Claude created to, to show people.

708

Um, and then they've fallen back down

to normal levels- Yeah … this year.

709

Yeah.

710

And so what you're arguing, if I'm not

mistaken, is basically something happened

711

in that middle year where it's like,

no, we didn't see improvements of 9%.

712

Like, something weird happened.

713

Like, there was some error in the testing

or some error in the analysis- Yeah

714

where it's like…

715

And this is important because it

looks like the state's doing a great

716

job, we're really improving, when

in reality it didn't improve at

717

all, and that's, that's really big

implications on like funding and money.

718

Is that correct?

719

Yeah, absolutely.

720

I think the, so, so two

things I would add to that.

721

So one is it, it really can't

be that they s- the scores went

722

up this much and down this much.

723

Like, this is a place where we

have so much data on how much

724

test scores like this vary.

725

We know so much about

just what is going on.

726

And, and the numbers here would imply

that like the tip, the, the sort of

727

across the entire state in basically every

school across every demographic group,

728

like fourth graders in one year learned

two-thirds of a year more, and then in the

729

next year they lost all of that and more.

730

Like, it's just like this

isn't, you know, it's, it…

731

No.

732

This is not right.

733

And I think that's a piece where

probably the person part of this is

734

really like I, like I have so much

experience with this, I can just look

735

at that and be like, "That's wrong."

736

And now I can go into, you know, some

LLM and be like, "Okay, I'm sure this,

737

like I'm pretty sure this is wrong.

738

Like, let's try to, try to understand it."

739

Um, and okay, so that's the first piece.

740

And then, yeah, the question is what,

uh, like what, what happened behind this?

741

And it really does matter because as

you say, you know, funding decisions

742

are made on these, on these numbers.

743

And for example, New York allocated,

you know, some three-year funding grants

744

of $250,000 a year across schools-

based on these flawed test scores,

745

which like basically made more middle

schools get this and some elementary

746

schools not get these like large grants.

747

There's like a lot of money behind,

millions and millions of dollars

748

behind these scores, and some

of them are, in my view, wrong.

749

That's, uh, crazy, and thank you

for, um, you know, organizing this

750

and trying to suss out these things.

751

That's, that's really important, I think.

752

Yeah.

753

I w- I mean, look, it was…

754

It- I think it is important, but it was

also very interesting and fun because

755

it required really, like, getting

into, you know, like, well, what…

756

Like, what did you do wrong?

757

Like, what exactly?

758

And that's, and that's where I think

the, the ability to move quickly with

759

these LLMs and the ability to have

an LLM read, like, you know, like,

760

600 pages of technical documentation

and be like, "Okay, you know, here,

761

like, let's kind of problem solve.

762

Like, where could possibly

this have, have fallen apart?"

763

And I got much further than I think

I would have been able to alone.

764

I still need…

765

I, I'm not done.

766

I mean, I'm, I'm done, but I'm…

767

We're not done.

768

But I think that somebody is

gonna figure out what actually ha-

769

happened, and hopefully fix it.

770

Super cool.

771

I hope.

772

We'll, we'll include a link to

your, um, Substack, 'cause I know

773

you have, like, a whole Substack

dedicated to the education stuff- Yes

774

um, and that analysis as well.

775

And I actually wanna talk about the

education data, 'cause one of the things

776

I do is I run a, I run a data boot

camp, um, where I try to help people

777

learn how to become data analysts.

778

And one of the projects we do, the

second project that we do, 'cause I'm

779

really, like, hands-on, project-based,

is we actually analyze, uh, the data

780

from Massachusetts and the- Nice

781

the, the results that they get from that.

782

And so, one, I'm familiar with how messy

and how many Excel spreadsheets and how-

783

Yeah … how hard it is to, like, join

that data and- Massachusetts is actually

784

very good relative to the average state.

785

It's not bad.

786

One of the reasons I chose it,

'cause it's the second project.

787

We don't wanna get too hard

and, like, actually joining

788

and bajillion different things.

789

Um, but first off, I was, like, super

stoked to see, like, oh, lookit, this

790

is, like, a really viable project that

people are doing similar things in real

791

life- Yeah … 'cause we try to create

a dashboard based off of, like, what's

792

happening in the schools and who's

doing well and who's not doing well.

793

So I was stoked to see that, that.

794

And the second thing, like, no, no

pressure obviously, but one of the

795

things we do is we have a team of data

analysts on our, uh, you know, on our

796

program who are always happy to analyze

data, especially for good causes.

797

So if there's ever, you know, another

school that's cheating or doing

798

something wrong and you don't have

the bandwidth, we're happy to, to,

799

to, to do an internship project

and, and- Oh, that'd be awesome

800

analyze that data and try to give

you- All right … our results.

801

That would be very fun.

802

Yeah.

803

I think we- Yeah … we are, in that

project, we are so focused on the,

804

like, just getting the data together,

and I think that sometimes we…

805

Like, there isn't bandwidth to do the,

like, okay, can we really understand

806

why these things changed in the way

they, they did, other than mistakes.

807

Okay.

808

Well, I'm, I'm serious.

809

Maybe we'll talk offline

on, on how to do that.

810

Yeah.

811

'Cause I have so many people

that would voluntarily do some

812

pretty interesting analysis.

813

Um, okay, that was awesome.

814

Okay.

815

Um, the, the last thing I wanna

do with you is play a game.

816

And, uh, you know, you are the

queen at, like, taking a complex

817

problem and being like, "Oh, you

know, this is what the data says.

818

You know, this is, this is, like,

whether the data's good or not, this is

819

what maybe you should do in your life."

820

And- Okay … uh, you're,

you're very good at nuance.

821

But I wanna ask you a few rapid

fire myths, and you tell me in one

822

sentence, uh, whether it's true or not.

823

Does that sound good?

824

Okay.

825

Yeah.

826

Great.

827

Okay.

828

Number one, uh, is

Tylenol safe in pregnancy?

829

Yes.

830

Tylenol is safe in pregnancy.

831

Okay.

832

Perfect.

833

That's easy.

834

Number two, is breast best?

835

Breastfeeding has some early life

benefits, but many of the benefits that

836

you are sold on, like IQ and obesity and

so on, are not supported in the best data.

837

Okay.

838

Number three, are phone

bans in school a good idea?

839

Yes, but not because they're going

to dramatically change a lot of test

840

scores, but because they are good for

kids' interactions with each other.

841

Awesome.

842

Number four, do cell phones cause cancer?

843

No.

844

And you know how we would know?

845

If brain cancer had gone up a lot

over time instead of actually what has

846

happened, which is that it's gone down.

847

That's good news.

848

All right.

849

Uh, is red meat bad for your health?

850

No.

851

Do vaccines increase autism odds?

852

No, they do not.

853

Um, should you take creatine?

854

Yes, if you are strength training.

855

There is no point if you

are a sedentary person.

856

Okay.

857

And last one, can you

get Botox while pregnant?

858

You can, but no one's gonna do it for you.

859

Okay.

860

There you go.

861

Uh, well, if you guys want the more

nuanced, data-driven, longer answers

862

to all these questions, you'll find

a bunch of in-depth articles, uh,

863

on Emily's website, parentdata.org.

864

It's actually one that I, I use pretty

often when I have a question in my

865

life, especially when parenting.

866

Mm-hmm.

867

Um, I subscribe to Emily's newsletter.

868

We'll have links to those down below.

869

I find them incredibly

helpful, uh, with parenting.

870

But even if you don't have kids, I

think you'll find Emily's style of

871

taking data and life information and

making good decisions or at least

872

having good frameworks for making,

uh, good decisions really helpful.

873

So we'll have a bunch of Emily's

links in the description down below.

874

And, and once again, check out Emily's

books, The Family Firm and Crib Sheet.

875

And there's another one that's Expecting

Better, is that what it's called?

876

Yeah, Expecting Better.

877

Okay.

878

That's the OG.

879

It's about pregnancy.

880

I, I was too late to have that one.

881

I don't have that one, but, uh, maybe

next kid, we'll, we'll get that one.

882

Um- Next kid, next kid.

883

E- Emily- Thanks … thanks so much

for coming on the Data Career Podcast.

884

Thanks for having me