Shownotes
In this episode, Dr Bisson talks to Professor Matthew Inglis about the importance of randomised controlled trials in education research. In other words, why is it crucial to allocate pupils to an intervention randomly and at the pupil level (rather than, say, a whole class) in order to be able to draw strong conclusions from a research study. Join us as we discuss how to get teachers better informed about this and other research methods.
Can you help us by completing our survey? We would like to know what you liked about this episode, what you would like to hear about next and also whether you have encountered any barriers in your language learning journey. Click here for the link to this 5-minute survey.
Follow Dr Bisson on twitter: @mjbisson
Or get in touch with her via email: marie-josee.bisson@dmu.ac.uk
The Language Scientists Podcast website: languagescientists.our.dmu.ac.uk
Link to Dr Bisson's research lab: sites.google.com/view/languagelab-dmu
Visit Dr Inglis research webpage or get in touch via email: m.j.inglis@lboro.ac.uk
Link to research article mentioned in the podcast
Transcripts
Dr Marie-Josee Bisson
If you have a connection to languages, this is the podcast for you. Whether you're a language learner, a language teacher, a language researcher, or anyone who's interested in languages. I'm Dr.Marie-Josee Bisson and alongside my colleague Dr. Kaitlyn Zavaleta, we are the language scientist and this is our podcast. We are senior lecturers in psychology at De Montfort University, and we conduct research in the area of language learning. Throughout the series, we hope to translate the science behind language learning into informative and useful practical advice. So sit back and enjoy. Today we are joined by Professor Matthew Inglis from Loughborough University and he's here to talk to us about randomised controlled trials. So welcome Matthew.
Dr Matthew Inglis
Hello there. Thanks. Thanks for having me.
Dr Marie-Josee Bisson
Matthew completed is undergraduate degree in mathematics and his postgraduate studies in education at the University of Warwick. He is now a professor of mathematics cognition. Now, before we jumped into today's topic, Matthew, I'd like you to tell us a little bit about your language background.
Dr Matthew Inglis
Well, this is where I appear hopelessly unqualified for your podcast, I'm afraid. My language background is pretty mega. So, I mean, obviously I studied, studied other languages in school, so I did German at GCSE, but not with a great deal of success. I'm afraid I wouldn't classify myself as very linguistic at all, although perhaps of interest, as I did a couple of years ago. I did start learning Esperanto while I was commuting to work on the train. I was, I had to occupy my mind while I was on the train, which I enjoyed and I still do from time to time, although that got a bit of a hit because of the pandemic and I had to stop commuting and it couldn't turn out. It was very closely tied to the physical location I was doing.
Dr Marie-Josee Bisson
So, but that's really interesting. So you don't classify yourself as a language learner or somebody that does language learning yet? Actually, from what you're saying, I would say that you are.
Dr Matthew Inglis
Well, yeah, maybe. Well, I mean, you know, you're the expert. I'm going to take that as a, as a compliment.
Dr Marie-Josee Bisson
Everybody's a language learner. So why Esperanto? That's an unusual choice.
Dr Matthew Inglis
Yeah, partly I was I was enthused by this claim. It's very easy to learn. And I thought, well, you know, I've struggled with proper languages, so let's try the, you know, the one that's most easy. Partly I was quite intrigued by the politics of it, so I was quite attracted by this idea of having a universal language that is sort of neutral for, you know, everyone is equally excluded from that. It seemed to me a nice idea. And also there was a Duolingo version of it that came out and that made, you know, gave me something to do on the train, you know, almost like a crossword or a word or something. So I guess all of those reasons, really.
Dr Marie-Josee Bisson
So you're using it as a mental workout?
Dr Matthew Inglis
Yeah, yeah, yeah. I think that's probably right. But it is also interesting. I mean, I do there's also interesting research on Esperanto, actually. I don't know if you're familiar with some of these. There is this claim that you're better at learning if you're an English speaker at school and you have one year of learning Esperanto and then one year of learning French, you're better at French at the end of that two years than if you have two years of learning French, which seems an extraordinary claim and you sort of think cannot possibly be true. But I mean, you know, people have done studies on this and the thought is the finding. I don't know how much one should trust such findings, but it's you know, so it's kind of got some interesting properties, you know.
Dr Marie-Josee Bisson
Something to look into.
Dr Matthew Inglis
Yeah, it's super regularity of it.
Dr Marie-Josee Bisson
So the way it's been designed, it's, it's meant to appeal to speakers of other European languages though.
Dr Matthew Inglis
It's very European centric. That's true.
Dr Marie-Josee Bisson
Yeah, that's really interesting. I find that each podcast we ask people about their linguistic backgrounds and we discover so such interesting things. So thank you very much for sharing that with us. Now I normally also ask our guests to tell us how they became a language researcher, but Matthew is a professor of mathematics cognition. So you might be wondering, why did I ask Matthew here on this podcast? It is because I was lucky enough to spend two years working with Matthew during my postdoc at Loughborough University, and I learnt so much about how to conduct rigorous research and I was hoping that today he would share some of his knowledge with us. So one of the things I wanted to discuss with you today, Matthew, was using randomised controlled trials in research studies, so I wonder if you can tell us a little bit about that.
Dr Matthew Inglis
led trials or RCT until about:Dr Marie-Josee Bisson
To encourage researchers to use this.
Dr Matthew Inglis
Exactly. And that's been a massive, massive change in education research in the UK. So the vast majority of education research funding now is comes through this organisation, the Education Endowment Foundation. So there's been a big, big shift towards doing randomised controlled trials in education and we're just kind of at the point now where people are sort of thinking through what the consequences of that been and whether that's a positive thing, a negative thing like what the, what's the effects of this big shift? And I think there's both big positives and some negatives as well and it's quite interesting to think about about them.
Dr Marie-Josee Bisson
Okay. So I feel like there's a lot of things we need to unpack from what you've just said for for everybody listening, maybe we can take things from the other perspective and you could tell us what happens if we don't randomly assign people to interventions.
Dr Matthew Inglis
. Plato made this claim up to:Dr Marie-Josee Bisson
So could you talk us through maybe one of your research projects where you've completed all the research, where you've used randomised controlled trials and tell us what happened?
Dr Matthew Inglis
Yeah. So I mean maybe the one that you worked on is a good is a good one. So that's a nice example where.
Dr Marie-Josee Bisson
That is a great example. Yeah.
Dr Matthew Inglis
I mean you, you did a very good job of making sure that worked well. Well, one example of that we were interested in whether two different ways of teaching algebra essentially to early secondary school students. And from the mathematics, education, literature. There were two quite different approaches to teaching algebra that had been proposed, and they had sort of different philosophical underpinnings. And we were interested in I mean, some people sort of pejoratively describe such a study as a horse race, you know, but sometimes actually you do want to you do want to find the outcome of a horse race. You know, if you're someone who needs to make a decision about how to teach. Sometimes you just would like some information about which all things being equal is likely to be the most promising approach. So that study we got, it's a group of schools who are willing to participate and we are at the pupil level. And that's an important point. We randomly allocated pupils within each class. You've got a class of 30 kids. We took 15 of them into one class and 15 of them into another class and taught one group one way and the other group the other way. And then at the end, we compared that performance on a on an algebra test, essentially. Now, sometimes that's really hard to do, that kind of study at randomisation at the pupil level.
Dr Marie-Josee Bisson
I was just about to interrupt you and tell you that this was not that easy to achieve, actually.
Dr Matthew Inglis
It was easy for me to achieve because I had just asked you to do it. But I imagine it wasn't so easy.
Dr Marie-Josee Bisson
It was hard. Is is to actually and this is one of the reasons I wanted to do this podcast is because it's not always clear to teachers why we insist on allocating ourselves, the pupils to do different interventions. Because when you're a teacher, the easiest thing to do and I totally understand why teachers don't have a lot of time and and it's disruptive to teaching when we just pick out just a few students each time from from the class. So for them, it's much easier to allocate a whole class to an intervention than to just pick students out. So it was really difficult to explain the reasons why we needed to do it that way. So sorry I interrupted you there. I just thought it was really important to mention that this was actually not that easy to achieve. And I needed to meet each teacher individually and explain this. And once they were on board, it was fine. But it was it was is always a difficult route to assign at the pupil level.
Dr Matthew Inglis
Yeah, absolutely. And I think there's several, several barriers you've touched on. I mean one is there's no I mean, I think actually compared to other countries, a big weakness of power in the UK, I'm talking about power teacher training programmes is they're so compressed. There's not really any time to talk about, for example,this kind of research methods issue of how to engage with education research at that kind of level. And so there's no reason why we should expect teachers to be familiar with these sorts of advantages and disadvantages of different, different research designs. Which is a shame, because I think if we want teachers to be research informed practitioners, we do need to provide them with the with the tools to do that. And being able to assess what kind of claims you can draw from different kinds of papers is, is, is an important skill, I think, and one that our teacher training system doesn't value very much. So that's one thing to say. The other thing to say is you could criticise our research design by saying, well, it's an unrealistic design because you were teaching children in very small groups of much smaller groups than you would in a normal class because of this random allocation. So maybe it would be desirable to randomise at the group level so that the class level or even at the school level. So maybe what we could have done is rather than take a class and split them randomly into two, we could have taken two classes and randomly split each into two. Sorry, randomly split the classes into two different groups. And you can do that and people do do these. Such things are called cluster randomised controlled trials. But one of the issues with those is they are you need a mass massively bigger sample size because what you're really doing there is randomly allocating well in a class situation rather than randomly allocating 60 kids from two classes. You're actually really randomly annotating only two things, the classes, so your sample sizes is really effectively massively reduced. So you need many more classes and many more pupils. And this is, I think is a serious issue actually with education research in the UK, particularly because of this big new funder. In my view a lot of educational research randomised controlled trials are what we call underpowered, so they have quite a not enough participants to draw with the kind of conclusions we'd like to make. And actually if you start doing calculations about how many participants you would need to, to detect the kind of plausible differences in the effects of these differences in teaching approach, you do end up running out of children. You know, if everyone in the UK was participating in, in randomised controlled trials, you know, that's not really realistic to expect that all schools are going to be doing multiple trials at once. So there is a genuine issue here about how to design randomised, randomised controlled trials in education if you want strong fidelity to the kind of realistic classroom situation. So yeah, I think there's pros and cons of doing it that way in my view. I think what I would do if I was in charge of the world.
Dr Marie-Josee Bisson
The world according to Matthew, yeah, this is a horrible thought.
Dr Matthew Inglis
But if I what I would do is I would try and encourage many more sort of small scale randomisation studies where randomisation happens at the pupil level.
Dr Marie-Josee Bisson
Yeah.
Dr Matthew Inglis
And then every now and again do one massive, you know, on the things you really want to know at scale in you really want to know if it's, if this intervention should be rolled out to the whole, you know, the whole country, then you can. I think that's where you should be doing these really large scale trials where randomisation is at the class or school level. But I think at the moment we're in this sort of slight hybrid between trying to do very high fidelity trials coupled with the sample sizes to achieve it. And I think that's that's leading to some problems.
Dr Marie-Josee Bisson
So if we go back to the study that we ran, for example, I feel like we had about 200 pupils.
Dr Matthew Inglis
Yeah, that sounds about right. Yeah.
Dr Marie-Josee Bisson
From many different classes, many different schools as well. Yeah.
Dr Matthew Inglis
Yeah. So if we had randomised at the class level I guess we'd have ended up with you know, sort of eight or so?
Dr Marie-Josee Bisson
Yeah eight or so classes.
Dr Matthew Inglis
which isn't really going to be
Dr Marie-Josee Bisson
No, to reflect the same as the people level. You would need about 200 classes.
Dr Matthew Inglis
Well, not as many as that. But yeah, I mean, many, many, many more. I mean, you save a bit because it's not going to the statistics of it.
Dr Marie-Josee Bisson
No let's not. Let's keep it simple. But yeah. So you'd need many more.
Dr Matthew Inglis
s, maybe, maybe, maybe two or:Dr Marie-Josee Bisson
It's a lot of pupils. Definitely. Okay. So if a teacher then is being asked to take part in a research project, what do they need to do?
Dr Matthew Inglis
Well, what I would do if I was a teacher is I would try and understand what the what the interventions about, you know, what what the goal of the study is. Now, obviously, sometimes researchers try and keep that slightly close to their chest because they don't want to influence teachers behaviour too much. You know, you don't want teachers to get super enthusiastic about this new, this new intervention and then any effect you detect is because of the super enthusiasm rather than the the intervention. So there's a slight concern there. But yeah, I would equally there's no point in getting teachers to participate in a research study and asking them to teach in a particular way if they're going to absolutely hate that way of teaching. Because, I mean, you know, a teacher's not going to be a very good teacher if they're really object to what they're being asked to do on some principal or practical level. So it's important to understand about the intervention, I think. I mean, there is this whole line of research actually about how intervention implementation goes wrong. You know, so if the designer of this intervention has a really, you know, really neat idea and sort of tries to roll it out across a large number of schools, if if the sort of, I guess you could call it the philosophy of the intervention is not well explained. Then the person who picks it up and tries to implement it might might misunderstand the point and, you know, implement it in a way that actively works against the purpose, you know, its purpose. So I think it is important for people to understand that teachers do understand. Well firstly, if they have some principled objection to what they're being asked to do, in which case, you know, that's no that's no good at all. And also to have some sense of what it's all about, why is it why they're being asked to teach in this way? So there's a lot yeah, there's a lot to think about, I think, for teachers. But I do. I also think that it's really, really important for teachers to engage in this kind of education research because, you know, I mean, if you think think I mean, maybe a good example is this the recent pandemic. You think about how important we all agreed that the trials of the vaccines were, you know, that really influenced policy and that changed. And they were very, very careful not to let people spend a lot of money and take all these or, you know, not not roll out these treatments or these vaccines without evidence of efficacy. And in contrast, in education, we're pretty bad at that. You know, we do roll out big initiatives across across countries, in educational jurisdictions with little or no evidence of efficacy. And that's that's doing everyone a disservice, really, is it's wasting teachers time, it's harming life chances of pupils. So my view is that, you know, as a as an education community, including researchers and practitioners, we really all should try and get to a point where participating in research is seen as part of the job, really, because that's ultimately going to benefit everybody. It stops us wasting our time doing things that are ineffective, for one thing, and it helps us understand how we can how we can improve learning for the students more generally.
Dr Marie-Josee Bisson
So in a way, we need to educate teachers, but maybe come in early on during their initial teacher training to explain more about research and maybe a little bit of almost like research methods in terms of.
Dr Matthew Inglis
Well it's very difficult. I think, because teacher training is so compressed and, you know, there's so little time in in British teacher training at least. So I think it is quite hard to imagine how the system is currently structured to imagine that that could easily be solved through teacher training. But you know, nevertheless there is this early career framework the UK Government is talking about and we are there is this big emphasis on becoming more research informed as a profession and I yeah, absolutely. I think research methods should be a fundamental part of that because, you know, to be honest with you, it is pretty easy to concoct a dodgy research study to try and prove something looks effective. You know, if, for example, if you just choose your your groups carefully, you know, so that one group is just already better at maths and the other or language or whatever. And the other group is worse. You can, you can make your intervention look quite good if you, if you're motivated to do that and if people are not aware of what kind of features of research study has to have in order to allow a legitimate causal conclusion to be drawn, there's a real danger that that could happen. I think that does happen. You know, I think we're familiar with examples of ineffective practice being publicised because, you know, there's often financial motives and so on. So yeah, I think it's an important thing to happen. Exactly how that could happen, given the constraints of the current system, it's not totally clear to me, but I think it's something we should all be working on.
Dr Marie-Josee Bisson
Great. Okay, so to finish this podcast, what would you like people to remember about what we've discussed today? What's the most important thing do you think?
Dr Matthew Inglis
I. Well I don't know. I guess I would like people to think, I dunno to get be interested in how small details of a research design can have massively different allows massively different conclusions. I think that's a little counterintuitive. So you know, often you can read a paper and a, you know, a research paper and the research to, you know, if you have two papers, one with a random allocation and one without help other than that line, they can look identical. You know, they can have the same statistical analysis, they can have the same graphs, they can have the same sampling strategy, and yet the conclusions that they allow readers to draw a radically different. So I guess I would I would be happy if at least some people who listen to this end up thinking to themselves, all research methods are kind of interesting. Yeah. Should learn a bit more about that.
Dr Marie-Josee Bisson
If they read a research paper, then look for that line that says participants were randomly allocated to each group or condition. And that's only one line in a whole research article. And it's extremely important. Absolutely. And it could have been one that was difficult to implement. So well done to everyone out there.
Dr Matthew Inglis
Obviously very well bitter about it. It's very good.
Dr Marie-Josee Bisson
Um, thank you so much for joining us today, Matthew, and for telling us about randomised controlled trials or simply more simply to randomly allocate participants to partciular conditions. In a study in the next podcast, I will be interviewing my co-hosts, Dr. Kaitlyn Zavaleta, about her research using speech errors. I just want to remind you to take 5 minutes to go into a shownotes and click on the link for our survey. The survey will tell us what you liked about the series and what you would like to hear about next. Thank you for listening and thank you for the British Academy for funding our podcast. I'm Dr. Marie-Josee Bisson and you've been listening to the Language Scientists podcast.
