Shownotes
Help us become the #1 Data Podcast by leaving a rating & review! We are 67 reviews away!
Emily Oster is a Harvard trained economist who built a career on making decisions when the data is bad or missing. I asked her how she does it.
💌 Join 33k+ aspiring data analysts & get my tips in your inbox weekly 👉 https://datacareerjumpstart.com/newsletter
🆘 Feeling stuck in your data journey? Come to my next free "How to Land Your First Data Job" training 👉 https://datacareerjumpstart.com/training
👩💻 Want to land a data job in less than 90 days? 👉 https://datacareerjumpstart.com/daa
👔 Ace The Interview with Confidence 👉 https://datacareerjumpstart.com/interviewsimulator
⌚ TIMESTAMPS
00:00 – Let the decision lead the data
04:03 – Instagram isn't evidence
08:39 – What to do when there's no data
10:57 – Your family is a small business
14:33 – Right decision vs right process
30:21 – The New York test score error
39:00 – Rapid fire myths
🔗 CONNECT WITH EMILY
📚 Expecting Better: https://a.co/d/00ExVFqN
📚 Cribsheet: https://a.co/d/09rGjGjs
📚 The Family Firm: https://a.co/d/08RaAOBh
📈 Emily's artifact: https://claude.ai/code/artifact/ecaef324-0efe-401a-9fca-d5d2816e88ee
📊 Education Substack: https://substack.com/@statetestscoreresults
🤝 LinkedIn: https://www.linkedin.com/in/emilyoster
📸 Instagram: https://www.instagram.com/profemilyoster/
🐦 X: https://x.com/ProfEmilyOster
💻 Website: https://parentdata.org/
🔗 CONNECT WITH AVERY
🎵 TikTok
💻 Website
Mentioned in this episode:
🔍 Find Your Next Data Job at FindADataJob.com
This episode is brought to you by FindADataJob.com — the #1 job board for aspiring data analysts. It's my own job board, where we post dozens of new data analyst roles every single day. Every listing comes with the stuff other boards leave out: salary reports, skill reports, and a seniority score, so you know exactly what you're applying to before you hit submit. Stop scrolling through a hundred irrelevant postings and go find your next data job faster at FindADataJob.com.
Transcripts
That's Emily Oster, a Harvard
trained economist who is the expert
2
:on making data-driven decisions
when there's not always good data.
3
:And I'm a huge fan of her data-driven
parenting books, and they've helped me
4
:raise my own kids and parent them in
a way that I feel really comfortable.
5
:Today, she'll give us the key, the method
to actually making good decisions even
6
:when there's poor data, there's not
data, or we're in a lot of uncertainty
7
:and there's a lot of unknowns.
8
:By the end, you'll have a great framework
for making great decisions like a data
9
:analyst, even if you're not one already.
10
:So let's go ahead and get into it
11
:Emily Oster is the founder and
CEO of ParentData, a professor of
12
:economics at Brown, and a three-times
New York Times bestselling author.
13
:Emily, welcome to the Data Career Podcast.
14
:Thank you for having me.
15
:Super excited to have you.
16
:I am a big fan.
17
:I have the, the books right here.
18
:If you guys haven't checked out- Amazing
19
:Emily's books before,
definitely, um, check them out.
20
:They are amazing.
21
:Um, but we live in a really
interesting time, Emily.
22
:Uh, there's, like, so much infor-
misinformation going around, um,
23
:in parenting and in everything
in politics and finance.
24
:Um, you know, everyone's trying
to tell you how you should parent
25
:your kids and what decisions
you should make for your kids.
26
:Uh, so my question t- for you,
is it possible to make good life
27
:decisions in a world where we're
constantly bombarded by different
28
:opinions and different data sets?
29
:I believe yes, uh, but I think it
requires us to think about the structure
30
:of our decisions rather than just
ask the question what the data says.
31
:So a lot of times people will come to
me and they'll be like, "Okay, well,
32
:just tell me what the data says."
33
:It's like, that's not always that helpful
a question, and if your approach to
34
:decision-making is to just, like, see
the last piece of data and, like, make
35
:a decision based on that, you aren't
necessarily gonna make good decisions,
36
:and I think part of what makes our current
information environment so challenging
37
:is that people are constantly getting
bombarded with data, and every time they
38
:see a new piece of data they're, like,
not necessarily ready to incorporate
39
:it into their decisions in a smart way.
40
:So I think the answer is yes, we
need data, and we can make good
41
:decisions, but we have to have the
decision-making sort of lead the data.
42
:So I would tell people, like, you
need to wait until you're ready to
43
:make a decision, and then think about
what your choices are, structure the
44
:decision, and then you get the data
that you need to make the decision and
45
:then make the decision based on that.
46
:But I think it's, it's too hard to Only
use data, I guess if that makes sense.
47
:Yeah, for sure.
48
:I think, I think it's also interesting
when we're talking about data to maybe
49
:specify what we're talking about.
50
:'Cause, you know, some of the topics that,
that you take on, um, like for instance
51
:Encryptshe, is, you know, is breast best?
52
:Like, is it actually good to…
53
:Is it better to breastfeed your,
your baby, or is bottle feeding okay?
54
:Um, another, you know, one of the other
things you tackle is vaccinations.
55
:Do vaccinations, you know, cause autism?
56
:And I think a lot of people, maybe for
people who are listening to this, they're
57
:data nerds and, you know, they're able
to, you know, maybe go out there and
58
:try to find some data on, you know,
autism rates and vaccination rates, and
59
:maybe, you know, put together some sort
of a statistical analysis to do so.
60
:But I think a lot of people are getting
their data from, like, Instagram posts.
61
:Yeah.
62
:Um, so, like, how do you try to navigate
the, the world where it's, like, a lot,
63
:where a lot of data is presented to us
in, like, an Instagram post or something
64
:that's, that's maybe not very structural
and, and hard to interpret in the moment?
65
:Yeah, I hate data from Instagram posts
because it's always like, "Here's a
66
:study that shows blah, blah, blah, blah."
67
:And it's like, well, what, like,
is it the only study of this topic?
68
:Is it the biggest study of the topic?
69
:Is it the best study of this topic?
70
:Is it some random thing from 1987
that you pulled out of, like, the
71
:journal of, like, made-up results?
72
:Which is usually the answer.
73
:And so i- Er, data is it's like I get so
frustrated because I think we, we really
74
:need to prioritize the best data, but part
of what is very challenging for people
75
:is it can be hard to know what that is.
76
:And there is a fair amount of
training that goes into the
77
:question of like, is this good data?
78
:Is this less, less good data?
79
:Um, so I guess I would say it
is never a good idea to make a
80
:decision about something based
on a single Instagram post.
81
:If you are in a position to need to
know whether some relationship is
82
:true, are vaccines causing autism,
for example, you need to step way back
83
:out of Instagram or out of TikTok or
whatever it is and figure out what are
84
:some sources you can go to that are
gonna give you a better, more nuanced,
85
:more thoughtful answer to that question.
86
:And there are a few things people can
look for in, you know, what makes a
87
:good data set, things like, is it big?
88
:Is it likely to be randomized?
89
:You know, things like that, and
that's, that's kind of the core.
90
:But, you know, a single study
says and somebody puts it in a
91
:carousel on Instagram, that's a
crappy way to learn about data.
92
:That- that's…
93
:I mean, that's unfortunate.
94
:I wish we could always just like
trust what we, what we see online.
95
:Um, but obviously we, we can't.
96
:Cannot.
97
:That's one of the things that I think,
um, you do a really good job in, in
98
:Crib Sheet especially of like, you
know, we're, we're debating, we're
99
:debating something like, uh, should
we like co-sleep with our babies or
100
:should we sleep train our babies?
101
:Um, you know, you pull up like
all these different studies
102
:that have been done on that.
103
:And some of the studies like that, maybe
let's just say for example, that say,
104
:oh, you know, you know, co-sleeping is
like really good for your baby actually.
105
:Um, you might throw…
106
:Correct me if I'm wrong, but like
you might like throw that study out
107
:the window and kind of ignore the
results because maybe it's not a
108
:large sample size or maybe it's not
a diverse sample size or, or maybe
109
:like the actual experiment was wrong.
110
:So even though the results say something,
like you're not necessarily one to
111
:just trust, uh, the results kind of
randomly from a, uh, an experiment.
112
:Is that kind of correct?
113
:Is that kind of your way of thinking?
114
:Yeah, definitely.
115
:I think that a lot of what distinguishes
the way that I approach sort of large
116
:corpuses of data from the way that you
would and it sort of, um, that other
117
:people would perhaps, is that I am much
more willing to say, okay, let me find the
118
:best studies here and base our conclusions
on the best studies rather than just like
119
:every study should get their, their voice.
120
:Like some studies don't deserve a voice.
121
:Um, and I think that is especially
true when we're outside of
122
:the randomization space.
123
:So in the parenting, like health, et
cetera space, there is a huge amount of
124
:what we're told that is like we're just
comparing people who do one thing to
125
:people who do another thing, and those
people are really different on like a
126
:billion dimensions, and we're attributing
it to the one topic that we're studying.
127
:And that kind of evidence I will
almost always say like, just forget
128
:it Like just put it in the trash.
129
:And that is actually a place where I
really differ from a lot of even, you
130
:know, people who I think have a lot of
training and are in, you know, serious
131
:like professors because there are people
who will tell you, "Well, okay, but
132
:we have so many studies of something.
133
:Like there are so many observations."
134
:But if it's not causal, it doesn't
matter how many observations you have.
135
:And so I'm, I'm very interested in how
we can have good data, really excellent
136
:data that lets us make causal statements.
137
:And if we have a study that isn't
gonna let us make causal statements
138
:about something we wanna make causal
statements about, I just think
139
:we should throw it in the trash.
140
:That's all.
141
:A- a- and, and I think that's important
for people to realize because, um, I
142
:think there's some group out, out there
who don't really take any scientific
143
:studies and, like, white papers.
144
:Like, they don't ever look at that.
145
:Like, they're only in the
Instagram world, you know?
146
:Uh, maybe, maybe looking
at aggregations of things.
147
:And there's some people who, who
maybe are a lot more, like, prone to,
148
:like, "Oh, I really trust science,
um, a- and studies," but it's always
149
:important to look into those studies.
150
:Um, I'm curious, like, there's not
always a study for, for everything.
151
:Right.
152
:Um, so, like, what do we do when
we need to make an im- an important
153
:decision, we wanna be data-driven
in our approach, but, like, there
154
:isn't good data or there's no data?
155
:What do we do then?
156
:Yeah.
157
:So I think the f- first, that's
very hard, and we have to first…
158
:I think first it's like there's a radical
acceptance of just saying, like, "Hey,
159
:I'm gonna have to make a choice here.
160
:There is no option to, like,
wait until the data is better.
161
:There just, I have to move
forward with one thing."
162
:And if we paralyze ourselves w- with
the view that, like, we can't make
163
:decisions until there are better data,
like, the decision will be made for
164
:you in some direction by you waiting.
165
:So just to recognize, like, sometimes
you'll have to make decisions
166
:under uncertainty, and that is
unfortunate, but it is the way it is.
167
:I think the second thing I would
tell people is in almost all of those
168
:settings, it's not important, right?
169
:So if something were really…
170
:It's not uniformly true, but if
something is really important,
171
:and I talk a lot about parenting,
but, like, really important in
172
:parenting, like really, really
matters, you will see it in the data.
173
:Like, the things that we know really
matter, like poverty, whether your
174
:kid has a stable place to sleep,
whether they have enough to eat, those
175
:things really show up in the data.
176
:The correlations are really, really big.
177
:We have good causal evidence.
178
:The kinds of questions where people say,
"Oh, I wish I had better data on this.
179
:You know, is it better to enroll
my kid in travel soccer or in,
180
:you know, travel lacrosse?"
181
:Or where, like, there's no data
on that, but you know what?
182
:It's not important.
183
:And so I think just, like, dialing
down and asking ourselves, "How
184
:likely is it that this thing matters?
185
:Given everything else about my family,
like, is this likely to be an important
186
:decision for my kids' outcomes?"
187
:And most of the time you're gonna say no.
188
:And then that actually makes the
decision-making much easier because
189
:it allows you to focus on all the
things that do matter for good
190
:decision-making, like how will
this logistically affect my family?
191
:How much does it cost?
192
:Whatever are the things that really
should go into that decision.
193
:So I think just reminding yourself you
gotta make decisions when it's uncertain,
194
:and a lot of things are not important.
195
:And it's okay to say, like,
"This probably isn't important.
196
:Maybe it matters a little bit in
one direction or another, but on
197
:the whole, it's not the thing that's
gonna break or break my kid," which
198
:almost there is nothing like that.
199
:I think that's such an interesting
approach, and, uh, I'm just thinking, you
200
:know, about the people who are listening.
201
:Uh, like so many of us are data
engineers, data analysts, data
202
:scientists, and it's like our whole life
is, you know, helping businesses make
203
:better decisions with, with the data.
204
:Um, and so I think it's hard for me, like
as a data nerd, to be like, oh, you know,
205
:sometimes the, the data doesn't matter.
206
:Like, or, or even maybe, maybe the
choice, um, doesn't really matter.
207
:Um, but, but we're- I, I would…
208
:So let me tell you, I think this actually,
there's such a strong, uh, like data
209
:analyst parallel here that I would make
for people, which is like, you know, the
210
:difference between like great success
and not great success in your business
211
:is like did you launch the right product?
212
:You know, did you…
213
:Like there's some big strategic
decision that is happening, you know,
214
:usually above the heads of everyone
and like where somebody at the top
215
:is making a big strategic play in one
direction or another, and that's gonna
216
:determine like whether the business
is successful or not successful.
217
:The job of the data analyst, and I
do this like for my own business, is
218
:to be like in the weeds and be like
can I get 1% more if I like send the
219
:email in this way or this other way?
220
:Or like can I optimize
the pricing in this way?
221
:Like let me do an AB test on
this, that, and the other thing.
222
:And like those things are really important
for your business, but they're not
223
:important like the big strategic question.
224
:In the background, they're kind
of optimizations on the margin.
225
:These kind of choices that we sometimes
get obsessed with with our kid, they're
226
:optimizations on the margin, and that
margin for parenting is really small,
227
:and there's so much noise And so
thinking about like, if only I could
228
:optimize this tiny thing, it's like,
well actually that's not important.
229
:Like, that's one tiny AB test in
one place and like if you get it
230
:right, you, you don't get it right in
parenting, it's, it's a lotta noise.
231
:So I don't know.
232
:That's, that's often how I
think about the parenting piece.
233
:I mean, you bring up a good point because
it's like if, should we launch product
234
:A or product B, and let's say we choose,
you know, product A, but actually product
235
:B was the right choice, but there was
just not data to make that decision.
236
:You know, what we're optimizing, let's
just say like a subject line on an email,
237
:and you know, you AB test it and you
find, oh, we get a lot more conversions
238
:with, you know, this, this subject line.
239
:It's like well, the bigger decision
was really we shoulda gone product B.
240
:Like, the, the amount of money
you can make on the- Totally
241
:you know, having the right subject
line really is probably dwarfed by
242
:actually launching the right product.
243
:Which that's, that's an
important thing to realize.
244
:I think it's important for people
in, in, you know, in their p-
245
:careers to realize that as well.
246
:That it's like we're, we can analyze
data, um, but we're, we're always trying
247
:to do so for business purposes and,
you know, m- move the business forward.
248
:So it, whether that's for our, our kids…
249
:And, and even like you said, like
travel lacrosse versus travel soccer.
250
:Let's say that there was data on that.
251
:It's like, well who's to say that
your kid is like the average-
252
:Totally … you know, it, it- Soccer
kid … doesn't take into account.
253
:Yeah.
254
:Yeah.
255
:Yeah.
256
:It's like- No, it-
257
:you don't have data on your kid.
258
:Totally.
259
:And I think in our, in our parenting
decisions, of course like everything
260
:in our family is so much more
complicated than like the subject line
261
:of the email that you're optimizing.
262
:And so every one of these decisions has
some other decisions associated with it
263
:which may actually be more important.
264
:So we can get very like laser
focused with our kids on
265
:thinking, you know, well let me…
266
:What is the right activity or choice or
school to like optimize, you know, some
267
:outcome metric that we have attached
to our kid, without stepping back and
268
:saying, you know, really what we're
trying to optimize is like that our kid
269
:is a, you know, happy, productive adult
who likes us and comes home for meals.
270
:And like that's actually a much broader
optimization than like, you know,
271
:are we gonna Achieve Junior Olympic
status or whatever, which you won't.
272
:One of the things I, I really like, uh,
along these lines and in the, in the book
273
:is, you know, you mentioned that parenting
you're gonna have a million decisions,
274
:and there's no way to guarantee that
you're gonna make the right decision.
275
:In fact, you're probably gonna make
the wrong decision quite often.
276
:Um, and instead of optimizing for making
correct decisions, you talk about,
277
:um, making decisions through, like a
framework and, and having education
278
:around the decision that you are making.
279
:Can you walk wa- about the difference
between, like, making right decisions
280
:and making the decision the right way?
281
:Yeah, absolutely.
282
:So, think the most important distinction
is one of those things you can
283
:do, and the other one you cannot.
284
:So you can never guarantee that
you will make the right decision.
285
:It's just, like, not,
that's not available to us.
286
:But we can say ex ante that we approach
the decision the right way, and so I
287
:talk about, you know, being structured
in how we make our choices, and starting
288
:by really outlining what our choices are.
289
:So being clear on the
choice that you're making.
290
:People often will ask me, you know,
"Well, should I do this or not?"
291
:You know, "Should I send my
kid to this school or not?"
292
:It's like, well, or not is not an
available schooling option, so, like, you
293
:better tell me what's on the other side of
that, because you're never gonna be able
294
:to compare something to, like, the vast
array of other things on the planet Earth.
295
:So you gotta think about
what your choice is.
296
:You've gotta collect the
information that you need.
297
:You need to then actually force yourself
to make a decision, and I think that's
298
:the hardest part, because in a world in
which we want to make the right decision,
299
:we can get, like, paralyzed by the
realization that we cannot guarantee that.
300
:And so in order to work ourselves
past that, we often have to really
301
:put in place, like, okay, I am
gonna sit down and make a decision.
302
:Like, I, this is the date.
303
:I'm putting it in my calendar.
304
:I'm scheduling it.
305
:Like, this is the time we're gonna
make whatever is this choice.
306
:Because otherwise you can just,
like, spiral and spiral and spiral
307
:forever and never actually make any
choices, and then usually the world
308
:will choose for you in some, in
some way if you don't do anything.
309
:Um, and I, I think the, the value of
having this kind of structure to a
310
:decision process is that on the other
end of it, you can be confident, again,
311
:not that you made the right choice, but
that you made the choice the right way.
312
:I think that's quite
protective for people.
313
:I will say one other thing,
which is I tell people, after
314
:you have made this choice, you
should make a plan to revisit it.
315
:You know, there are some choices
we can never revisit, you know?
316
:W- should I or should I
not have a second child?
317
:We- well, once you choose that,
you've pretty much, that's,
318
:you're pretty much committed.
319
:We're not revisiting that.
320
:But many of the choices we
make you can revisit, right?
321
:Like, I sent my kid to this school,
but I can choose another school.
322
:I sent them to this activity, but I
could choose another activity, like,
323
:later, or we could not do this activity.
324
:And I think we owe it to ourselves to
plan this kind of revisiting of the
325
:choices that we make, because if we
don't, then we will never revisit them.
326
:And if we plan to revisit them,
we're much more likely to do so.
327
:So much good to unpack there.
328
:Um, one, one thing I wanna zero in on
is, like, you, you mentioned if you're
329
:deciding between A and B, and you're
like, "I don't have enough data to do
330
:that," um, sometimes we don't make the
choice, and that itself is a choice, is
331
:what it sounded like you were saying.
332
:And oftentimes it's w- the worst
choice of the three, it seems like.
333
:Like, it's even worse than deciding wrong.
334
:Yeah, I think it, it sort of like, it,
it, it robs you of the opportunity to
335
:improve the choices you have, right?
336
:So this is like, decisions are
particularly hard if neither
337
:choice is really something we want.
338
:If we're sort of like, "Well, both of
these choices are kind of poor," we're,
339
:we're really reluctant to make them
because there's something very aversive
340
:about choosing something you don't want.
341
:But if you don't on purpose choose,
you will end up in one of the, in one
342
:of the s- the branches, typically.
343
:And if you haven't chosen it,
you won't have had an opportunity
344
:to make it as good as possible.
345
:Like, among the bad options, how can
I make this option the least bad?
346
:And that's, uh, that's missed out
on if you just decide to ignore
347
:the problem and forget about it.
348
:Uh, that's a good lesson for me to learn,
because I am the king of doing that,
349
:and, uh, that stings a lot of the time.
350
:So, uh- … I'm gonna try
to do better with that.
351
:Um, I'm curious, like, so I'm used
to data, quote, unquote, as, like, a
352
:table in Excel or, like, in, you know,
in a SQL database type of a thing.
353
:Um, but like you said, we don't always
have data to, to make decisions.
354
:One of the things that you talk
about in your other book, The
355
:Family Firm, is, like, other
ways that you could potentially
356
:get data that aren't necessarily
from, uh, like a, a spreadsheet.
357
:And, you know, maybe they're not as
high integrity as, as spreadsheet data.
358
:Um, but they're still
valuable in making decisions.
359
:Um, so, like, one of the things you
mention about is, you know, information
360
:about your personal circumstances.
361
:And, and you mentioned earlier,
like, you know, if you're making
362
:decision A or B, like, which one
are you actually going to like more?
363
:Even, even if one's more optimal
than the other, like, which
364
:one are you going to like?
365
:And the other thing that you mentioned
about is talking to, to others.
366
:I just am curious to hear your thoughts
on, like, when you're getting data
367
:that isn't necessarily, like, tabular
data, really high quality data, um,
368
:it's, it's more, like, around you
data, what, what are you looking for?
369
:What signals are you looking for, and
how can you know if you can trust it?
370
:Yeah, so I think really here I'm talking
about getting i- e- information, I think.
371
:Look, data is just pieces of information.
372
:So I'm really telling you, like get
some data on how you feel about stuff.
373
:And I think for many people who are
kind of like us, actually maybe you
374
:wanna put that in a spreadsheet.
375
:Like I'm not averse to the idea that
like you should collect data on your
376
:preferences and logistics and constraints
in the same way that you would imagine
377
:collecting data on, you know, k- test
score outcomes or, or whatever it is.
378
:But I think a big piece of this
is, is kind of th- interrogating,
379
:like if I make this choice, what,
what are the actual implications?
380
:You know, both how much am
I gonna like my day-to-day?
381
:Really think about it.
382
:Like if I get up and I face, you know
… Le- let me put a concrete example in it.
383
:So a lot of people talk, talk to me
about like choosing between preschools.
384
:Like I have this preschool and
it's right close to my house, but
385
:it, you know, isn't very fancy.
386
:N- only half the teachers have master's
degrees or whatever, or there's this
387
:preschool that's like 40 minutes away
and it's like super fancy, right?
388
:It's like how do I think about that?
389
:Okay, and so one piece of that
data is, you know, how much do
390
:we know about differences in
preschool outcomes or whatever.
391
:And another piece of that data is
like how do I feel about commuting
392
:40 minutes each way with my kid?
393
:You know, what's that gonna do to my day?
394
:Like what's my day … That's
a piece of data.
395
:Like what is my day gonna look like?
396
:How are we gonna manage this logistically?
397
:How am I gonna feel about, am I gonna
fight with my partner about this?
398
:Like are we gonna argue every day
about who is gonna take their kids?
399
:That's a piece of data.
400
:You know, so there's a bunch of
stuff in there that you want to put
401
:in your decision making, even if it
isn't numbers in a spreadsheet, but
402
:it is information in a document.
403
:Uh, and another piece of that is
asking other people, you know,
404
:what do they think about it?
405
:How did they experience this?
406
:But like-
407
:It's…
408
:For people who love data and evidence,
there's an aversion to the idea of, like,
409
:going with your gut, and people will talk
about this as like those are two choices.
410
:You could go with the data,
or you could go with your gut.
411
:My view is like the data's not bossy.
412
:It's not gonna tell you what to do.
413
:It's only an input to decision-making,
which also needs your preferences.
414
:People say, "I'm going with my gut."
415
:What they really mean is,
"This is the thing I want."
416
:Okay.
417
:But you can actually bring those things
together, and that's gonna be better
418
:than your data or your gut alone.
419
:I'm, I'm laughing because I'm literally…
420
:Like, we're putting, uh, our,
our daughter in preschool.
421
:Literally went through that exact…
422
:Do we go to the s- the preschool we think
is, like, better, quote unquote, you know,
423
:but it's further away, or do we just kinda
do one that, that's close by and near?
424
:So, uh, I'm laughing 'cause I went
through that, that exact analysis- Yeah
425
:recently.
426
:I will tell you, we went to the, we…
427
:The better one that's further away.
428
:It's not 40 minutes away.
429
:If it was 40 minutes away, I don't
think I would be able to do that.
430
:But now that we're in the process, and,
like, my wife is the one who takes her
431
:the majority of the time, it's like,
"Oh, we don't really like this commute.”
432
:So, you know, next year or next
kid or next semester, I don't know
433
:if we're gonna do this or not.
434
:Um, so sometimes you make a decision
the best you can, and you make- Yeah
435
:… maybe a wrong decision, but it's just
another data point where it's like, "Oh,
436
:actually, I don't like being in the car
for-" Yeah "… X amount of minutes a
437
:day" We learned this- And we're done
438
:totally, and then I think that's
why you need the, the other step
439
:because if you don't plan to revisit
that choice, then you, you're not
440
:gonna do it 'cause you're not gonna
wanna admit that you made a mistake.
441
:And it's like it's not
that you made a mistake.
442
:You did a trial, and, like, maybe
that trial turns out like, well,
443
:we learned something that we wanna
do something differently next time.
444
:A- and I think it's important that not
only we do this with, with preschool, but
445
:when I worked for ExxonMobil, I was a data
scientist there, and one of my jobs was
446
:to make machine learning algorithms to
predict how much gasoline we should buy
447
:in each, you know, one of the thousands of
different stores we have across America.
448
:And I, I built, you know, a machine
learning algorithm that, that was
449
:the most accurate we could make it,
um, to predict, you know, gasoline.
450
:But it wasn't like, hey, the, the
gasoline prediction that my machine
451
:learning model puts out is what we order.
452
:That, that number goes to, uh, a trader,
a buyer of gasoline, and they can
453
:totally ignore my number, and they can,
you know, use my number as, as aid.
454
:So it's so interesting that, like, you
know, these preschool decisions are
455
:kind of the same framework that, you
know, multi-million, billion, you know,
456
:in- industry decisions are making.
457
:I'm sure you've seen that kind
of with, with your research and,
458
:and, and your analysis as well.
459
:Totally.
460
:And I think we under even…
461
:It's, it's sort of interesting always for
me to watch people who are so good at this
462
:approach to their job, who are so, like…
463
:And then I'm like, "Well, can't you…"
464
:Like, you should just be
porting that into your life.
465
:Like, it's the same thing.
466
:Like whe- you know, when you are running,
when you are married to someone and you
467
:have, or you like have a partner and
you have kids together and you have two
468
:jobs and whatever, like you're running
a small to medium sized enterprise.
469
:And there actually are like quite a
lot of tools from how companies do
470
:that, that I think will make this
easier for, for people, even though
471
:it's like a really weird thing to say.
472
:And when you tell people that they're
like, you know, they're like, "Well,
473
:do you boss your husband around?"
474
:It's like, "Of course I boss him around,"
but that's not because of the, it's
475
:not because of this enterprise idea.
476
:That's awesome.
477
:And I love that.
478
:And that's, you know, one of the, the
things you talk about in, in The Family
479
:Firm is like, we should, we should really
kind of organize our family, uh, like
480
:we r- organize teams and, and businesses
in a fun way, not like in a boring way.
481
:Yeah.
482
:But like- Fun way … we
should have objectives.
483
:We should have goals.
484
:Like, we should, you know, make decisions
that are for the optimal happiness
485
:and health o- of our family, which
I think I've, I've been reading it.
486
:I've really been, um, enjoying that.
487
:Um, I'm curious, so you know,
we talked about like making
488
:data-driven decisions as, as parents.
489
:I'm curious, like where else in our
personal lives, um, that we can make
490
:data-driven decisions and kind of
adopt the approach that you've taken
491
:in, you know, in breastfeeding and in,
you know, sh- what kids, where, where
492
:should we send our kids to school?
493
:What other places in our life can we
live data-drivenly other than parenting?
494
:Yeah, I mean, I think the other obvious
one that people like is health, um,
495
:is sort of like there are a lot of
health decisions that you have to make.
496
:Uh, and there's a lot of data on those.
497
:Uh, and I think it has many of the
same issues that parenting has.
498
:You know, there's some of the
data is better than others.
499
:Um, and You know, we, you have
to make decisions that take
500
:into account your constraints.
501
:Um, and I actually think this is
a place for me where we are…
502
:A mu- much of the discourse
misses the idea of constraints.
503
:We're like really good in the health
space about talking about like how
504
:to optimize, like, you know, let's
track every da, da, da, da, da.
505
:Like, you know, how do you like get
all of your n- numbers to be exactly
506
:optimal in these various ways?
507
:But without helping people sort of see
like, okay, well, you probably don't
508
:have 17 hours a day to like fully
optimize a 27-step life protocol.
509
:Uh, and so how do we incorporate the
data with the constraints and ask,
510
:you know, what are, what are the
most effective things, uh, to do?
511
:And I think the other place is just how,
in like sort of general, like how do we
512
:choose our jobs and how do we, you know,
operate our like professional lives?
513
:Um, but I think health is the, health
is the other obvious space for me.
514
:Health is a really interesting one,
um, because obviously, like, it-
515
:it's … If you don't have health,
you have nothing in your life, right?
516
:Right.
517
:'Cause, like, I think we all have
known someone that's, that's lost
518
:health, and we just see, you know,
how much of a detriment to life that
519
:is and how, how difficult things are.
520
:So that makes sense to optimize it.
521
:Um, just, like, a, a concrete example
of that is, and I know you, you talk
522
:about, um, this in, in your books.
523
:Um, but I got diagnosed with ADHD last
year, and I had a, I had a decision.
524
:It's like, do I try, you know, after
trying, you know, six months of
525
:non-medication ways to, to, to solve the
problem, like, do I try medication or not?
526
:Yeah.
527
:Um, and you know, one of the things
that I do is, like, I pretty much
528
:constantly wear, uh, an Apple Watch.
529
:And so I've been taking, you know, ADHD
medicine for, for almost nine months now.
530
:And one thing I've seen is my
resting heart rate has, has risen,
531
:like, five beats per minute.
532
:Yeah.
533
:And it's like, okay.
534
:Um, that's, like, a side
effect of taking ADHD medicine.
535
:It's like, do I, do I like that?
536
:Do … Is, is the health risk that with
my heart worth the effects of m- you
537
:know, maybe me getting more work done
or maybe me being a more patient parent?
538
:Um, those types of things.
539
:And it's hard because it's like there's
not really all that data out there
540
:that can support … Or, or maybe
there is a bunch of data out there,
541
:but it's like which one do I trust, and
how do I apply it to my personal life
542
:and my business and my family life?
543
:Um, so I guess making decisions
under uncertainty with
544
:health definitely makes- Yeah
545
:makes a lot of sense.
546
:Yeah, and un- under uncertainty, and
they're really also under constraints.
547
:Like, you're, like, really what
you're describing is a constraint,
548
:which is like you, like, if you do
this one thing, it has this effect.
549
:But y- like, you're … It's trade-off,
uh, and we're not that good at trade-offs.
550
:And a lot of the health messaging
in particular sort of doesn't like
551
:to acknowledge the existence of
trade-offs, so they're just like,
552
:"Do all 4,000 of these things."
553
:And it's like, okay, but I, I, I can't do
that, or it's, like, literally impossible
554
:to do two of these things at the same
time, and so now I have to pick one.
555
:And, you know, people ask me,
like, "Should I sleep or exercise?"
556
:Like, I only have 30 minutes.
557
:I have to pick sleep or exercise.
558
:Like, which thing is better?
559
:And, uh, that's, that's a hard question.
560
:What's the answer?
561
:Hmm, kind of, kind of depends how much
sleep you're getting, but probably sleep.
562
:Yeah.
563
:I, and ob- obviously there's so
many factors that, you know, that
564
:go into your life, and it's hard
to, to make a blanket statement.
565
:It's funny that you're, you're
mentioning this, 'cause my other job
566
:at ExxonMobil, my other problem that
I solved at ExxonMobil, uh, and I
567
:didn't solve this problem on my own.
568
:We worked as a really big team to do this.
569
:But we, we made mathematical models
of the entire refinery, um, which
570
:was, like, 140,000 equations.
571
:Um, and we were trying to optimize, you
know, how much money the refinery can
572
:make with all these different constraints.
573
:Like, we can't…
574
:We need to make sure that our, our
pollution's abov- uh, below this level.
575
:You know, our, our tower
can only take this many-
576
:Yeah … barrels of crude every day.
577
:And that was a really
hard problem to solve.
578
:Yeah.
579
:Um, and we knew, we knew
all the, the math behind it.
580
:It's like you can't really do that in
your life, 'cause, like, you can't really
581
:model life outcomes, I don't think.
582
:Nope.
583
:Um, maybe you can.
584
:I don't know.
585
:No, and you have…
586
:And, and of course, then when people try
to, they come up with, like, crazy things.
587
:Somebody sent me a paper the other
day which, in which these people,
588
:like, tried to assign a number of
minutes of life to, like, each food.
589
:So, like, if you have one
Diet Coke, it costs you, like,
590
:this many minutes of life.
591
:But it's like, that's a cra- like,
first of all, that's bananas.
592
:Like, you definitely can't do that.
593
:It's all of the data is from correlation.
594
:It's not causal, whatever.
595
:But it also just, like,
didn't make any sense.
596
:It was like a Diet Coke costs you 12
minutes, but, like, a peanut butter
597
:sandwich gains you, like, 33 minutes.
598
:And it's like, okay, well, if
I eat them together, can I get
599
:fif- like how does this work?
600
:But it was so, like, so much in the
space of people just want an answer.
601
:They wanna know, like,
okay, how much is…
602
:Like, what's the cost of this Diet Coke?
603
:And the answer is, like- We don't, you
know, we don't have it, probably zero.
604
:Uh, or have it with a peanut butter
sandwich, and then I get to negative 17.
605
:That's awesome.
606
:I love that.
607
:That's, that's very cool.
608
:Um, okay.
609
:I saw something really cool that you,
um, posted on your Twitter recently
610
:and your Substack, and we'll make
sure to have a link to your social
611
:in the description down below.
612
:Um, but it was a really cool analysis
you've done recently on New York education
613
:data and kind of like their testing data.
614
:Um, and one of the things you
actually published that caught
615
:my eye was a Claude artifact.
616
:Yeah.
617
:So for those who are unfamiliar
with Claude, it's basically
618
:like ChatGPT, but it's from a
different company called Anthropic.
619
:Um, I really like it for doing things
like data analysis, and it creates
620
:these things called artifacts, which are
basically, you can think as like a, a, a
621
:published something, a URL that goes to
some sort of a page that has text on it.
622
:And in your case, you were analyzing
data, so it had text and graphs and
623
:different analyses and like that.
624
:Um, and I wanna talk about the
New York- Yeah … education
625
:study, uh, study that you did.
626
:But first, I want to kind of walk me
through your, like, data pipeline.
627
:Like, how, how did this, like, come to be?
628
:Like, where are you getting your data?
629
:How are you analyzing it?
630
:How are you publishing it?
631
:I was really curious about that, if
you don't mind sharing maybe, like,
632
:a high, high, uh, view of that.
633
:Sure, yeah.
634
:So in, in that case, actually, the
key to that entire analysis is one of
635
:the projects I d- I do is something
called the Education Data Center,
636
:uh, which is a, a project where we,
uh, try to clean and organize all
637
:the state-level test score data.
638
:So if your kids are in, you know,
public school in grades three through
639
:eight in the US, they will take,
uh, math and ELA tests every year.
640
:That's like an important
part of accountability.
641
:Uh, but the state's data
is, like, a hot mess.
642
:Like, every state is issuing
it in a different thing.
643
:If you wanna have the data from Montana,
it's 3,000 separate spreadsheets.
644
:Like blah, blah, blah, blah.
645
:And so one of, one thing I really
care about data transparency.
646
:Uh, and so in this project we,
like, download all of this stuff,
647
:and we have, so we have like a, a
website where you can sort of get
648
:all the microdata for, for this.
649
:And I mention that because that's
a sort of core, like, backend
650
:pipeline for a project like this.
651
:On that particular project, there's
a, this thing that happened in New
652
:York with the test scores, which we
can talk more about, but where, like,
653
:basically somebody called me, some
reporter called, and they were like,
654
:"Here are the, you know, here are the
test scores that are gonna come out.
655
:Like, what do you think?"
656
:And I looked at them, and I
was just like, "They're wrong.
657
:Like, I don't like, I don't know
what to tell you, but, like,
658
:data doesn't look like that.
659
:Like, somebody made a
mistake probably last year."
660
:And then I got really, like- exercise.
661
:I, like, I really love the, the
piece of data where you try to,
662
:like, learn what's going on.
663
:Just like, I just wanted
to know what's going on.
664
:Like, I couldn't, like, let it go.
665
:And so then my data pipeline is, you know,
in the, in the back end, I'm basically
666
:using Claude with an API pulling down
this, this raw data and kind of writing
667
:code in Python to, like, figure out,
try to figure out what's going on.
668
:And this is a place where the, these AI
tools and sort of Claude in particular
669
:has really changed how quickly I could do
something like this because, you know, on
670
:the back end, I could have written this
code in Stata on my own and so on, but I
671
:probably would not have had the bandwidth
to do it without a kind of LLM tool.
672
:Uh.
673
:It was awesome, first off.
674
:Uh, it was super cool.
675
:It was so fun.
676
:It was like, it was so cool.
677
:Yeah, so basically what, what I'm
hearing is like you, you obviously
678
:know how to do this analysis.
679
:You could obviously do it from scratch.
680
:And I love the idea of from scratch.
681
:It's like none of us are actually
doing this by hand and paper.
682
:Right.
683
:Like, that's probably from scratch.
684
:No.
685
:So it's like, oh, like-
My dad was an economist.
686
:He used to like, I think, do this by…
687
:They would like multiply the
matrices, but that was a while ago.
688
:See, but that, that's kind of
my point, is it's like, oh, and
689
:then R came out, and then Python,
or and then, then Stata- Right
690
:or whatever.
691
:Uh, you know, and it's like now we just
have Claude and ChatGPT, which I just see
692
:as like a new tool, like you said, that
enables this- Yeah … type of analysis.
693
:Um, and it's like I don't think we
could've taken a random Joe off the
694
:street and, you know, had them create
this analysis that you created.
695
:I don't think this analysis, the
AI could've created on its own.
696
:Um, so it's cool to get like
a little bit of glimpse on, on
697
:how you're using AI to do that.
698
:Um, so that, that's very cool.
699
:Um, I do wanna get into like the,
the details of, of, of what happened
700
:in this, in this, I wanna call
it a study, but it's not a study.
701
:These test results.
702
:So basically, um, if I'm understanding,
uh, correctly, the test results…
703
:L- l- let's make it as simple as possible.
704
:The test results were around a certain
level, and then the next year they jumped
705
:up like 10%, from like 43 to like 52%.
706
:And maybe we'll pop up the, the
graph on the screen that your
707
:Claude created to, to show people.
708
:Um, and then they've fallen back down
to normal levels- Yeah … this year.
709
:Yeah.
710
:And so what you're arguing, if I'm not
mistaken, is basically something happened
711
:in that middle year where it's like,
no, we didn't see improvements of 9%.
712
:Like, something weird happened.
713
:Like, there was some error in the testing
or some error in the analysis- Yeah
714
:where it's like…
715
:And this is important because it
looks like the state's doing a great
716
:job, we're really improving, when
in reality it didn't improve at
717
:all, and that's, that's really big
implications on like funding and money.
718
:Is that correct?
719
:Yeah, absolutely.
720
:I think the, so, so two
things I would add to that.
721
:So one is it, it really can't
be that they s- the scores went
722
:up this much and down this much.
723
:Like, this is a place where we
have so much data on how much
724
:test scores like this vary.
725
:We know so much about
just what is going on.
726
:And, and the numbers here would imply
that like the tip, the, the sort of
727
:across the entire state in basically every
school across every demographic group,
728
:like fourth graders in one year learned
two-thirds of a year more, and then in the
729
:next year they lost all of that and more.
730
:Like, it's just like this
isn't, you know, it's, it…
731
:No.
732
:This is not right.
733
:And I think that's a piece where
probably the person part of this is
734
:really like I, like I have so much
experience with this, I can just look
735
:at that and be like, "That's wrong."
736
:And now I can go into, you know, some
LLM and be like, "Okay, I'm sure this,
737
:like I'm pretty sure this is wrong.
738
:Like, let's try to, try to understand it."
739
:Um, and okay, so that's the first piece.
740
:And then, yeah, the question is what,
uh, like what, what happened behind this?
741
:And it really does matter because as
you say, you know, funding decisions
742
:are made on these, on these numbers.
743
:And for example, New York allocated,
you know, some three-year funding grants
744
:of $250,000 a year across schools-
based on these flawed test scores,
745
:which like basically made more middle
schools get this and some elementary
746
:schools not get these like large grants.
747
:There's like a lot of money behind,
millions and millions of dollars
748
:behind these scores, and some
of them are, in my view, wrong.
749
:That's, uh, crazy, and thank you
for, um, you know, organizing this
750
:and trying to suss out these things.
751
:That's, that's really important, I think.
752
:Yeah.
753
:I w- I mean, look, it was…
754
:It- I think it is important, but it was
also very interesting and fun because
755
:it required really, like, getting
into, you know, like, well, what…
756
:Like, what did you do wrong?
757
:Like, what exactly?
758
:And that's, and that's where I think
the, the ability to move quickly with
759
:these LLMs and the ability to have
an LLM read, like, you know, like,
760
:600 pages of technical documentation
and be like, "Okay, you know, here,
761
:like, let's kind of problem solve.
762
:Like, where could possibly
this have, have fallen apart?"
763
:And I got much further than I think
I would have been able to alone.
764
:I still need…
765
:I, I'm not done.
766
:I mean, I'm, I'm done, but I'm…
767
:We're not done.
768
:But I think that somebody is
gonna figure out what actually ha-
769
:happened, and hopefully fix it.
770
:Super cool.
771
:I hope.
772
:We'll, we'll include a link to
your, um, Substack, 'cause I know
773
:you have, like, a whole Substack
dedicated to the education stuff- Yes
774
:um, and that analysis as well.
775
:And I actually wanna talk about the
education data, 'cause one of the things
776
:I do is I run a, I run a data boot
camp, um, where I try to help people
777
:learn how to become data analysts.
778
:And one of the projects we do, the
second project that we do, 'cause I'm
779
:really, like, hands-on, project-based,
is we actually analyze, uh, the data
780
:from Massachusetts and the- Nice
781
:the, the results that they get from that.
782
:And so, one, I'm familiar with how messy
and how many Excel spreadsheets and how-
783
:Yeah … how hard it is to, like, join
that data and- Massachusetts is actually
784
:very good relative to the average state.
785
:It's not bad.
786
:One of the reasons I chose it,
'cause it's the second project.
787
:We don't wanna get too hard
and, like, actually joining
788
:and bajillion different things.
789
:Um, but first off, I was, like, super
stoked to see, like, oh, lookit, this
790
:is, like, a really viable project that
people are doing similar things in real
791
:life- Yeah … 'cause we try to create
a dashboard based off of, like, what's
792
:happening in the schools and who's
doing well and who's not doing well.
793
:So I was stoked to see that, that.
794
:And the second thing, like, no, no
pressure obviously, but one of the
795
:things we do is we have a team of data
analysts on our, uh, you know, on our
796
:program who are always happy to analyze
data, especially for good causes.
797
:So if there's ever, you know, another
school that's cheating or doing
798
:something wrong and you don't have
the bandwidth, we're happy to, to,
799
:to, to do an internship project
and, and- Oh, that'd be awesome
800
:analyze that data and try to give
you- All right … our results.
801
:That would be very fun.
802
:Yeah.
803
:I think we- Yeah … we are, in that
project, we are so focused on the,
804
:like, just getting the data together,
and I think that sometimes we…
805
:Like, there isn't bandwidth to do the,
like, okay, can we really understand
806
:why these things changed in the way
they, they did, other than mistakes.
807
:Okay.
808
:Well, I'm, I'm serious.
809
:Maybe we'll talk offline
on, on how to do that.
810
:Yeah.
811
:'Cause I have so many people
that would voluntarily do some
812
:pretty interesting analysis.
813
:Um, okay, that was awesome.
814
:Okay.
815
:Um, the, the last thing I wanna
do with you is play a game.
816
:And, uh, you know, you are the
queen at, like, taking a complex
817
:problem and being like, "Oh, you
know, this is what the data says.
818
:You know, this is, this is, like,
whether the data's good or not, this is
819
:what maybe you should do in your life."
820
:And- Okay … uh, you're,
you're very good at nuance.
821
:But I wanna ask you a few rapid
fire myths, and you tell me in one
822
:sentence, uh, whether it's true or not.
823
:Does that sound good?
824
:Okay.
825
:Yeah.
826
:Great.
827
:Okay.
828
:Number one, uh, is
Tylenol safe in pregnancy?
829
:Yes.
830
:Tylenol is safe in pregnancy.
831
:Okay.
832
:Perfect.
833
:That's easy.
834
:Number two, is breast best?
835
:Breastfeeding has some early life
benefits, but many of the benefits that
836
:you are sold on, like IQ and obesity and
so on, are not supported in the best data.
837
:Okay.
838
:Number three, are phone
bans in school a good idea?
839
:Yes, but not because they're going
to dramatically change a lot of test
840
:scores, but because they are good for
kids' interactions with each other.
841
:Awesome.
842
:Number four, do cell phones cause cancer?
843
:No.
844
:And you know how we would know?
845
:If brain cancer had gone up a lot
over time instead of actually what has
846
:happened, which is that it's gone down.
847
:That's good news.
848
:All right.
849
:Uh, is red meat bad for your health?
850
:No.
851
:Do vaccines increase autism odds?
852
:No, they do not.
853
:Um, should you take creatine?
854
:Yes, if you are strength training.
855
:There is no point if you
are a sedentary person.
856
:Okay.
857
:And last one, can you
get Botox while pregnant?
858
:You can, but no one's gonna do it for you.
859
:Okay.
860
:There you go.
861
:Uh, well, if you guys want the more
nuanced, data-driven, longer answers
862
:to all these questions, you'll find
a bunch of in-depth articles, uh,
863
:on Emily's website, parentdata.org.
864
:It's actually one that I, I use pretty
often when I have a question in my
865
:life, especially when parenting.
866
:Mm-hmm.
867
:Um, I subscribe to Emily's newsletter.
868
:We'll have links to those down below.
869
:I find them incredibly
helpful, uh, with parenting.
870
:But even if you don't have kids, I
think you'll find Emily's style of
871
:taking data and life information and
making good decisions or at least
872
:having good frameworks for making,
uh, good decisions really helpful.
873
:So we'll have a bunch of Emily's
links in the description down below.
874
:And, and once again, check out Emily's
books, The Family Firm and Crib Sheet.
875
:And there's another one that's Expecting
Better, is that what it's called?
876
:Yeah, Expecting Better.
877
:Okay.
878
:That's the OG.
879
:It's about pregnancy.
880
:I, I was too late to have that one.
881
:I don't have that one, but, uh, maybe
next kid, we'll, we'll get that one.
882
:Um- Next kid, next kid.
883
:E- Emily- Thanks … thanks so much
for coming on the Data Career Podcast.
884
:Thanks for having me
