Transcripts
Hey, Megan, let's do a podcast. Great idea. What
Speaker:should we talk about?
Speaker:Hey, it's Megan from TorranceLearning and another
Speaker:installment in our Tangents with TorranceLearning Podcast. And one
Speaker:of the things that is super, super cool about my
Speaker:job and the work that I do is I get to talk to all sorts
Speaker:of really interesting people. And the
Speaker:cool part about having the podcast is that I get to share those conversations
Speaker:with everybody. And so what I'd love to do is
Speaker:introduce my new big nerd of a
Speaker:friend, Chris Grady, and have a conversation that
Speaker:may feel at first like it comes out of left field, and then all of
Speaker:a sudden it's going to say, oh, my gosh, this is like what we do
Speaker:here in Allen Date. So, Chris, welcome.
Speaker:Well, thanks for having me. And thank you for introducing me as a giant nerd.
Speaker:That is my official title. Chris Grady, Giant Nerd. I
Speaker:think maybe you need a little bit more of an official title than that, but
Speaker:I don't mind Giant nerd. Yeah, well, for people listening, I'm a
Speaker:researcher, and I'm a former senior advisor at usaid, where I help design and evaluate
Speaker:development programs, which, one might say has
Speaker:nothing to do with learning development other than that the word development
Speaker:is in both. And yet in both cases.
Speaker:Stop me if I'm wrong here. In both cases, we are doing a
Speaker:whole lot of effort aimed at a whole bunch of people
Speaker:or a targeted group of people in order to get them to do or
Speaker:change a behavior that we feel is
Speaker:good for them or good for us or good for something. Is that a fair
Speaker:description? Yeah, that's fair. I mean, most interventions at
Speaker:USAID were about behavior change. Right. We want people to do something different than
Speaker:they're doing. And that's most of the interventions in the world. Right. If you're doing
Speaker:an intervention, it's because you want something to change, and that's usually people's behavior.
Speaker:And then the key question is, did it work? And how do we know if
Speaker:it worked? And so that's what I specialize in. Okay, that's
Speaker:fascinating. And one of the things that so you
Speaker:presented earlier this year at an
Speaker:ATD intensive on measurement and
Speaker:analytics, and we. What was really cool was your
Speaker:entire conversation around experiments.
Speaker:And experiment is a word that generally
Speaker:feels very unsettling in the learning and development world because we want to
Speaker:come across as professionals. We know what is right for us.
Speaker:And while we love to iterate,
Speaker:that's part of what we do in our Agile process.
Speaker:We don't often call it an experiment.
Speaker:What do you mean when you say the word experiment?
Speaker:I mean something, I guess, very technical. You've got some group of people
Speaker:and you split them and then you do something to one group and don't do
Speaker:something to the other group. And that way you can learn the effect of the
Speaker:thing you did. Boom. Experiment.
Speaker:So if I were to say, gosh, it
Speaker:sounds like drug trials or university
Speaker:research or something like that. Is that a good
Speaker:analogy? Exactly. Drug trials, university
Speaker:research, they all almost always use experiments as their main tool for
Speaker:learning. So drug trials, right. You give the drug to some people and not others,
Speaker:randomly assign who gets it and who doesn't, so you can learn the effect of
Speaker:the drug. You can apply that same logic anywhere. Right. The same logic applies
Speaker:whether it's a social innovation, a drug trial, a diet, or
Speaker:anything. Anything you want you can do an experiment on and learn the effect of
Speaker:it. Okay, this kind of sounds
Speaker:like the scientific method. Yes. Yeah, it's very motivated
Speaker:by the scientific method. Right. You have a hypothesis and you want to test that
Speaker:hypothesis, and the experiment is the test. Okay. And
Speaker:then generally don't you end up with more questions at the end of all these
Speaker:things? Yeah, that's the best part. Every
Speaker:experiment, you analyze it, you get results, and then you want to know why did
Speaker:that happen? Or something will pique your interest in your brain, your curiosity.
Speaker:Then you can do another experiment. A never ending cycle of experiments. That's my
Speaker:dream. Megan, you are a big nerd.
Speaker:Okay, okay. So, all right. When
Speaker:I'm thinking about this process and how it unfolds, when does
Speaker:somebody decide they need to test
Speaker:this or measure this or design an experiment? Right.
Speaker:In my mind, not really knowing a lot about how these things work,
Speaker:somebody comes up with a good idea, somebody goes and they gets grant money, or
Speaker:they get budget money, or they get money money to be able
Speaker:to go do a thing that they think is a good idea and they start
Speaker:doing it. At what point in that process
Speaker:is the experiment designed? Like at the beginning, before
Speaker:you start doing the thing, before you start even designing the thing. Or does
Speaker:somebody say at the end, like, hey, I wonder if that worked? Which is, by
Speaker:the way, where we unfortunately often
Speaker:end up in learning and development, although we're trying to work our way back up
Speaker:that cycle. Yeah, unfortunately it's true. Most people start
Speaker:to wonder if it worked after they did it. But that is too late for
Speaker:an experiment. Right. Because to do an experiment, you have to randomize who gets the
Speaker:thing. And so you can't, after you've Already done the
Speaker:thing, it's too late to randomize it. So what often happens is
Speaker:you do a thing for usaid, it would be some development
Speaker:program, right? So a way to like help a country collect more taxes.
Speaker:And then you want to know if it worked. So what you have to do
Speaker:then is do it again, right? Maybe in a different place, in a different location
Speaker:where you can randomize it. And so I think that's actually a good process
Speaker:because you do it once almost as a pilot. You kind of figure out how
Speaker:to implement something and you get a sense that it might have worked, right?
Speaker:And so then you want to know, you want some rigorous evidence that it did
Speaker:work, and then you do an experiment on it. So I think that's a good
Speaker:workflow. Like first do something as kind of a pilot, and then if you think
Speaker:it works, you want to scale it up and do it everywhere and do it
Speaker:a lot, then it experiment. So you've got more than just a proof of
Speaker:concept, you have rigorous evidence that it was effective. Okay, so this is a
Speaker:multi step. So this is interesting. So the way
Speaker:learning project works, we're often like, go do this thing and we do a bunch
Speaker:of analysis and we do some design and some development work. And then
Speaker:there's generally a pilot group if we're smart, or a beta test or something.
Speaker:And you're saying maybe
Speaker:you're not saying this, I'm interpreting this, maybe like
Speaker:we do that pilot, we do that beta, but then
Speaker:that next iteration is actually an experiment where we
Speaker:say like, is this actually. So we've proven it kind of works, and then we
Speaker:should test whether or not it actually has the intended effect on the
Speaker:audience. Yeah, absolutely. Because it's very easy to
Speaker:delude ourselves into thinking something works when it doesn't. Because we want it to work.
Speaker:Right. If I'm at USAID and I've helped design a program, I
Speaker:think it's going to work. I designed it to be effective. And so
Speaker:unless I have rigorous evidence that it doesn't work, I'm going to assume it does.
Speaker:And so sometimes we need an experiment to check ourselves. And another good
Speaker:reason to pilot is that you want to know if the intervention
Speaker:works or doesn't work. Not because you're
Speaker:learning to implement it, but because it being implemented well is working
Speaker:or not. Because something you might try and think of some example, but
Speaker:something might be failing not because it doesn't work, because you don't know how to
Speaker:do it yet, like riding a bike, right? It might take some time to Figure
Speaker:out how to ride the bike. And then after that, you want to test if
Speaker:you're faster or slower riding than running. If you only test it, when someone's
Speaker:learning to ride a bike, it's going to look like they're slower because they're falling
Speaker:off the bike. But once you learn to ride the bike, it's much faster,
Speaker:obviously. So you have that same issue when you're piloting a program, right? When
Speaker:people pilot, they're essentially falling off the bike over and over until they learn how
Speaker:to pedal. And so you don't want to be testing them falling off the bike.
Speaker:You want to test the bicycle. You know what I mean? I totally do. I
Speaker:totally do. And it occurs to me this is, we're falling
Speaker:into this. It's a little bit intentional, but
Speaker:a client of ours, government client, asked us
Speaker:to build an AI practice module. Yeah,
Speaker:it's pretty fancy. It's actually super constrained
Speaker:because for us, we want to make sure that it works and it doesn't go,
Speaker:like, off the rails. But
Speaker:the, the first asked was, can you build an, an
Speaker:mvp, basically a proof of concept. Can it do its thing? And we
Speaker:shopped that proof of concept around a bunch of, you know, different
Speaker:stakeholders. And at the same time, we're, every time we're, we're
Speaker:shopping it and demoing it, we're punching that. Kicking the
Speaker:tires, whatever you do to tires, right? Don't puncture tires. That's a bad idea. But
Speaker:we're kicking the tires and, and, and, and really
Speaker:kind of working on this. And then our next
Speaker:iteration, we're actually building out because
Speaker:this audience happens to be really, really
Speaker:pure, like as, as pure as you're going to get in L and D. And
Speaker:we're going to talk about that in a second. I'll tell you a little bit
Speaker:about our world, but we have a large
Speaker:population, about a thousand learners who all have exactly
Speaker:the same job, and they're taking exactly the same learning
Speaker:program. And we are going to give half of them a randomized group. Half
Speaker:of them will get the AI practice, half of them will get the same
Speaker:exact module, but it won't be AI. It'll just be
Speaker:straight up. And then we'll be able to assess
Speaker:their performance. At the end on. I think we have eight different metrics
Speaker:that we're looking at. But all of that
Speaker:is before we spend the big money to scale it up
Speaker:100 times the size to a much, much larger audience. So
Speaker:it was interesting in the design of that. I was hanging out with another big
Speaker:Nerd. And I said, oh,
Speaker:I'm kind of feeling awkward here because
Speaker:we've got this AI practice. We're going to give it to half the audience and
Speaker:the other half isn't going to get it. But what if the other half, like,
Speaker:but then they might fail the test and they might not get their job and
Speaker:all this stuff. And this big nerd's
Speaker:response was, you're assuming that this is going to be better.
Speaker:Exactly. Like, oh, my gosh, what a butt punch.
Speaker:Right? I was like, oh.
Speaker:But yeah, like, yeah, I was assuming that this beautiful
Speaker:thing we're making was going to be better. And how much better to
Speaker:test it than to make that assumption? Right. So,
Speaker:so how do we. And this kind
Speaker:of came up during the ATD intensive too. Right. Most people
Speaker:say it might be infeasible. We don't have the budgets, we don't have the
Speaker:time. It's unfair to, you know, if, if.
Speaker:How do we. How do we make ourselves feel better? Or how do we
Speaker:design an experiment that's feasible in an
Speaker:environment? Well, we have to train everybody. Got
Speaker:any ideas? How have you tackled this? Yeah, it's a good question.
Speaker:Also, the ethics of an experiment came up a lot at usaid,
Speaker:but I think that this ethical question comes up a lot when you
Speaker:assume that the thing works, because then it would be wrong to not give it
Speaker:to everybody. So what we've done is, well, we don't know if it works yet,
Speaker:so let's do the experiment and then if it works, give it to the control
Speaker:group who didn't receive it. You can always give it to them later. Right.
Speaker:And you also find often, if you think about it, it's
Speaker:almost unethical not to do the experiment, because what if what you're doing
Speaker:not only doesn't have any effect, but it's harmful and you've
Speaker:actually done something harmful to people. You want to know that and you want to
Speaker:not be able to do it. And if something's ineffective, you don't want to waste
Speaker:a bunch of money on it because that's not helping people. So the best way
Speaker:to help people do the experiment early, figure out if something works or not, and
Speaker:then roll it out to everybody. So that same logic should apply in
Speaker:training. Like your example with an AI training module.
Speaker:What if the AI was worse and it made people worse at their jobs? You'd
Speaker:want to know that. So you don't roll it out to everyone. And if it's
Speaker:completely ineffective, you'd also probably want to know that too, because it's probably costly to
Speaker:roll it out to everyone and so that money could be better spent elsewhere.
Speaker:Totally, totally. Because it's a lot more expensive to build AI training than
Speaker:straight up elearning. Yeah. Okay,
Speaker:so that's really helpful perspective and
Speaker:probably tweaks the messaging that a lot of L
Speaker:and D people need to be engaging with their business
Speaker:on learning and development is
Speaker:often. Especially when it's employee development. Right. It's a, it's a.
Speaker:Well, it is not even often. It is a cost center. It is an
Speaker:expense to the business and not often
Speaker:seen as a source of competitive advantage.
Speaker:It's not often seen as an. Yeah, we say it's an investment in
Speaker:people and it is an investment in people. But
Speaker:when times are tough, what gets cut? It's training. Right.
Speaker:Marketing is also an investment in the business that doesn't get cut as
Speaker:much. So we're kind of in this tricky
Speaker:spot. What that means though, is that a
Speaker:lot of times people want to, when they say like, oh,
Speaker:Megan, I want to measure some stuff, I say, oh, great, why do you want
Speaker:to measure? And they want to prove their worth to the
Speaker:organization does that. To me, that sounds like, oh,
Speaker:that sounds biased. Am I thinking of that right? Yeah.
Speaker:When you're doing the experiment, you. You want to be accepting of
Speaker:either result. Like if something works or doesn't work.
Speaker:Yeah. And you wouldn't want to go in desiring a certain
Speaker:outcome. Right. Because you're going to. There's unconscious ways you can
Speaker:kind of make that outcome be achieved. There's lots of
Speaker:little effects. Like if you bring someone into a lab and
Speaker:you want a certain, you want them to respond a certain way, they tend
Speaker:to pick up on that, even if what you're doing is unconscious. And then they'll
Speaker:respond that way. And so it would look like whatever you did had the effect.
Speaker:But it's people's social intelligence coming out. That's why you often see
Speaker:in medical trials and other rigorous studies, double blind.
Speaker:So the experimenter, when someone comes in, the experimenter doesn't know if
Speaker:that person's in the treatment group or control group because the experimenter knowing
Speaker:biases the respondent. That's why in a drug trial, they'll often
Speaker:give people a pill in the control group, but it's just a salt pill. It
Speaker:doesn't have any medicine in it. Because they don't want either
Speaker:side to know which group they're in because people will manifest something themselves.
Speaker:That sounds like you might have a similar problem here. If you want to Find
Speaker:something, you're often going to find it. But it's difficult then to
Speaker:unbiase yourself. How do we unbiased ourselves? We're human beings, we're all biased.
Speaker:So, yeah, I don't have a good answer for how to do that unless you
Speaker:can find some way to bind yourself. Maybe you
Speaker:remove yourself a bit from the actual analysis of the
Speaker:experiment or implementing it and have somebody who
Speaker:doesn't have a stake in the results being kind of a positive outcome do
Speaker:that. But that, yeah, that's tough because who
Speaker:else is going to implement and analyze the experiment if not, you don't
Speaker:have a good answer. Megan? Well, you got me in a pickle.
Speaker:We say, like all good questions spawn more questions.
Speaker:Well, I think one of the things that in many organizations,
Speaker:right, we're starting to get data scientists brought into,
Speaker:or at least data analysts brought into the learning and development
Speaker:team. I even heard last year a team that
Speaker:had combined and they called it Learning, Design and Analysis. And I thought,
Speaker:oh, my gosh, that's amazing. And, and, and some people
Speaker:are finding in their organizations right there. They have a
Speaker:finance team or a marketing teams already doing this kind of analysis
Speaker:on surveys and stuff, or
Speaker:they're engineers or they're R and D folks who can have maybe a little bit
Speaker:more. They're a little bit more detached, but they also have the statistical
Speaker:analysis and tools to be able to do some of this work
Speaker:that is helpful. It also
Speaker:occurred to me, I wonder if there's like college interns or
Speaker:people who would love to have to work on this kind of thing.
Speaker:Yeah, I bet college interns would. I mean. So I've been going back
Speaker:thinking about what you would ask. I think we need to, in our minds, often
Speaker:reframe things we think. Like, if you do something and you find
Speaker:no effect, that's bad. No, that's great. You've learned that
Speaker:you don't need to do that, you know, and that's great. You can do something
Speaker:else. That gives you the freedom to, okay, let me design something new. Let me
Speaker:design something better. So we shouldn't go into it thinking, oh, if I find
Speaker:that what I did was ineffective, that I failed. No, that's like, that gives you
Speaker:another opportunity. You've learned something valuable and that's really helpful to the business to know
Speaker:they don't need to. Don't throw your money down that hole. Right. That is
Speaker:fantastic. Yes. Find out which money, which holes to throw the money at.
Speaker:Which hole? Does the money become a money tree or. I don't Know that,
Speaker:that, that works totally. So
Speaker:if we think about how
Speaker:we gather data, right. One
Speaker:of the things, and I'm think surveys as
Speaker:a. There are lots of different ways in which we can gather data.
Speaker:The learning and development team often doesn't have access
Speaker:to actual on the job performance data or
Speaker:doesn't have access to or the on the job performance is
Speaker:not quantified, it's not instrumented to be quantified and
Speaker:measurable. And so
Speaker:surveys is a tool that is often at our
Speaker:disposal. We can send surveys, we know who's taken a training, we can
Speaker:send them a survey. Any pitfalls there? As we think about
Speaker:designing surveys to send out to people, oh my gosh, there's
Speaker:so many pitfalls with any, any measurement is going to have lots of pitfalls. So
Speaker:there's a whole field called measurement validity where they try to validate
Speaker:measures and surveys. Obviously there's, there's going to be several possible
Speaker:issues that come up. The first thing, people just
Speaker:might not know the, you know, you're going to ask them their attitude on something.
Speaker:They'll come up with some attitude, but that might not
Speaker:be their real attitude. First, they might not know. Second, they might not want to
Speaker:tell you and they might not even let themselves know that they
Speaker:don't want to tell you. So are people lying to you or to themselves?
Speaker:There's like a lot of, in political science, we often study if
Speaker:people voted or not. If you ask the average person,
Speaker:they're going to tell you they voted. Something like 75% of people tell you they
Speaker:voted even when the actual voter turnout's like 50%
Speaker:because it's socially desirable to say you voted. And so if you just ask that
Speaker:question, clearly measurement validity is going to be low because
Speaker:we know that it doesn't correspond to people's actual behavior. So you have to figure
Speaker:out a way to ask people the question in a way that they can and
Speaker:want to respond honestly. So there's lots of tools for doing
Speaker:that. But that's a huge challenge. One, the simplest one,
Speaker:is to make the socially undesirable things seem totally acceptable. Right?
Speaker:So you do that with framing. You frame the question in a way that it's,
Speaker:you know, hey, we know a lot of people don't have time to vote. Did
Speaker:you do you have to have time to vote this year? And so then if
Speaker:they say no, it's like you've already kind of pre built in the excuse.
Speaker:So it's okay if you didn't vote because we know lots of people have
Speaker:Trouble getting there. So it's not that you're lazy or you didn't want to vote.
Speaker:It's like your life is busy. So little things like that can make it more
Speaker:acceptable for people to answer honestly. And then there's tools to
Speaker:grant people anonymity so that there's no way to
Speaker:trace their answer back to themselves. There's several ways.
Speaker:Okay, now we're going to get into really deep measurement validity, but there's several ways
Speaker:people do this. One is called, like, a randomized response technique. And the
Speaker:simplest way to explain this is you give someone a. You ask someone a yes
Speaker:or no question, then you give them a coin. You have them flip the coin
Speaker:so you don't see it. You say, if it's heads, just say yes. Just say
Speaker:yes. It doesn't matter what your actual response is. If it's a tails, answer
Speaker:this question. And so then if they say yes or no,
Speaker:or if they say yes, you're not actually sure if they said yes because of
Speaker:the coin or because that was their real answer. So they have plausible deniability
Speaker:that if they say yes to the socially undesirable thing, like, did they smoke
Speaker:marijuana? It's not that they actually smoked, it's that the coin told
Speaker:them to say yes. Right. There's other techniques that are similar, but those
Speaker:are ways if people don't want to tell you something,
Speaker:but you can get them to tell you by granting them anonymity.
Speaker:That's kind of cool. Well, I was even thinking, your first way, you're like, hey,
Speaker:there's a lot of reasons why people will come to a class and not be
Speaker:able to apply that on the job. Were you able to
Speaker:apply anything on the job that seems like a perfectly plausible thing?
Speaker:Yeah. They're going to give you way more accurate and honest
Speaker:responses than if you just say, did you apply this on the job? Because it's
Speaker:clear if you ask that you want them to say yes. You know, clear,
Speaker:your sociopaths will say, like, no. Yeah,
Speaker:okay. This is fascinating. And we could be here
Speaker:all day long, so looking
Speaker:forward to the opportunity to continue talking with you about it, But I want to
Speaker:shift gears slightly. We like to. To wrap up these.
Speaker:These conversations because you are more than your work. But I'm
Speaker:fascinated with just, like, work rhythms and
Speaker:rituals and the way people engage with their craft. And
Speaker:you've spent a lot of time in academia, like getting a PhD and all that.
Speaker:Right. Does that influence your. Your work
Speaker:rhythms, do you think in semesters or trimesters, what Are your
Speaker:fidgets? What are your work snacks? What if you were to
Speaker:sum up, what is Chris Grady's work style?
Speaker:What's it like? What's it like inside your head?
Speaker:Intense focus, which is different than any other. In my
Speaker:personal life, I'm very non focused and kind of chaotic. Go with the flow.
Speaker:When I work, I just disappear into my mind for hours. I often,
Speaker:I'll like even forget to eat and eight hours will go by and I'll
Speaker:realize I haven't moved in eight hours, which is very rare because if I'm not
Speaker:working, I'm constantly moving. I exercise constantly. I can't sit and watch
Speaker:tv. If my wife wants to watch a program, she wants to cuddle. She's usually
Speaker:disappointed because I'll wind up doing yoga during the program. But while I'm working,
Speaker:I just laser focus. Especially if I'm coding and doing analysis,
Speaker:the hours fly by. And I think that that was very
Speaker:beneficial in grad school where you need often like a lot of deep thinking and
Speaker:focused time for long periods of time. And
Speaker:it's harder to do that in a work world, right. Where I'm like, oh, I
Speaker:have half hour now, then I have a meeting for 30 minutes and then I
Speaker:need to, I don't know, I have 30 more minutes to do something and then
Speaker:an hour meeting. So I think for me academia was great because it let
Speaker:me use my natural rhythm of just I'm going to work on one thing for
Speaker:eight hours straight and that was a benefit instead of a detriment, which in the
Speaker:work world I've got to adjust and be more flexible. I don't know, there's a
Speaker:lot of people who would love the ability to focus like that. Yeah, I have
Speaker:reverse add.
Speaker:That's fantastic. Chris, this has been awesome.
Speaker:A window into your mind, window into your work.
Speaker:Applicability to my own work and probably the work
Speaker:of a lot of people who are listening. So I just want to thank you
Speaker:so much and looking forward to continuing the
Speaker:conversation. Thank you a lot, Megan. It was fun.
Speaker:So how'd that go, Megan? You know, that was super fun because it's the first
Speaker:time we've talked to somebody outside of the L and D
Speaker:industry and yet all the things that he
Speaker:does and talks about are applicable and appropriate
Speaker:for us. And we could just lift some of these techniques
Speaker:and there's some, some really easy, easy takeaways. Even if you took
Speaker:only one thing away, there's probably half a dozen one things that you could
Speaker:take away and do without any hardship
Speaker:or at least to have that conversation with your leader. So I think that
Speaker:was, that was super fun. And maybe we should find some other folks outside of
Speaker:L and D and see what we can learn from them. I know you've got
Speaker:one more thing to share, Megan. Yeah, there is one more thing.
Speaker:So here's, here's one more thing that I was thinking about as we're having this
Speaker:conversation with Chris is thinking
Speaker:back to the conversation with Will Telheimer. And as he's got
Speaker:a lot of work around, right? The learning transfer evaluation model. And
Speaker:it's, you know, where I've been really moving into. It's his, his wording around,
Speaker:right. Learning is competitive advantage for the business. And that's really resonated with me.
Speaker:And I know I brought that up here. But also
Speaker:connecting in, I want to go back now and reread performance
Speaker:Focused learning surveys by Will, because I think the combo
Speaker:of what Will's talking about and what Chris Grady is talking about
Speaker:is a really powerful combination for getting really
Speaker:valid measurement in our industry. This is Meg
Speaker:Fairchild and Megan Torrance, and this has been a
Speaker:podcast from Torrance Learning. Tangents is the official
Speaker:podcast of Torrance Learning, as though we have an unofficial one.
Speaker:Tangents is hosted by Meg Fairchild and Megan Torrance.
Speaker:It's produced by Dean Casteel and Meg Fairchild,
Speaker:engineered and edited by Dean Casteel with original
Speaker:music also by Dean Castile. This episode was
Speaker:fact checked by Meg Fairchild.
