In this episode, Dr Bisson speaks with Professor Kathy Conklin about eye-tracking in language research. From archaic equipment with bite bar to newer online tools to track people's eyes, Kathy explains what eye-tracking can tell us about language processing and language learning. In particular, you get to hear about her fascinating research on binomials: What is so special about "fish and chips" compared to "chips and fish"? 

Can you help us by completing our survey? We would like to know what you liked about this episode, what you would like to hear about next, and also whether you have encountered any barriers in your language learning journey. Click here for the link to this 5-minute survey. 

Follow Dr Bisson on twitter:  @mjbisson 

Or get in touch via email:  marie-josee.bisson@dmu.ac.uk 

The Language Scientists Podcast website: languagescientists.our.dmu.ac.uk 

Visit Professor Conklin's research webpage: https://www.nottingham.ac.uk/english/people/kathy.conklin 

Or get in touch with her via email: Kathy.Conklin@nottingham.ac.uk 

Link to the eye-tracking book mentioned in the episode:  

CONKLIN, K., PELLICER-SÁNCHEZ, A. and CARROL, G. (2018). Eye-tracking: A guide for applied linguistics research. Cambridge University Press 

Dr Marie-Josee Bisson

If you have a connection to languages, this is the podcast for you. Whether you're a language learner, a language teacher, a language researcher, or anyone who's interested in languages. I'm Dr. Marie-Josee Bisson and alongside Dr. Kaitlyn Zavaleta. We are the language scientists, and this is our podcast. We are senior lecturers in psychology at De Montfort University, and we conduct research in the area of language learning. Throughout this series, we hope to translate the science behind language learning into informative and useful practical advice. So sit back and enjoy. Today we are joined by Dr. Kathy Conklin from the University of Nottingham, who will talk to us about eye tracking in language research. Welcome, Kathy.

Dr Kathy Conklin

Welcome. It's a real pleasure to be here.

Dr Marie-Josee Bisson

Kathy is a professor at the University of Nottingham, but she did an undergraduate degree in English and French at the University of Buffalo, and her Ph.D. was there too. But this time in linguistics and cognitive science. I met Kathy when I did my Ph.D. and University of Nottingham, and she became one of my PhD supervisors. I'm really excited to have you today. It's a real treat. So before we jump into your topic, Kathy, which is eye tracking, I always ask people in these podcasts, to tell us a little bit about their language background. Yeah.

Dr Kathy Conklin

So I'm a native speaker of English or North American English, but in high school I did a study abroad year in France. So I lived with a French family. I went to a French lycee or high school. So I was completely immersed in the French culture and family and school life, and I actually learned French to a really high proficiency, although don't ask me to speak French anymore, as this was a really long time ago.

Dr Marie-Josee Bisson

And this is really interesting because the previous episode was on Incident Learning. So you just were bombarded by an incident learning situation, really and that's how you acquired french. Amazing.

Dr Kathy Conklin

Yeah, absolutely. So it was a very immersive experience. And and absolutely, I learned French very incidentally.

Dr Kathy Conklin

So after coming back from France, I attended a bachelor's and master's, as you said, at the university at Buffalo. And after completing those, I spent two years in the Peace Corps in Nepal. And Peace Corps is kind of like the volunteer service overseas. So the British VSO. And in the first year, I was in a very remote village that was a three day walk from the nearest town. That sounds the most amazing experience. We want to hear all about it. It was so I was kind of in the foothills of Mount Everest, but it took about six days to walk there over very steep mountains or foothills, as they call them, to get to to base camp. And while I was there for the first year, I taught English in this remote village. And the second year I was there, I worked in the capital at the Ministry of Education, helping them to develop their English education language program. Certainly in the first year I was very immersed in Nepali. There was nary another English speaker for three days in any direction. But I never really achieved high proficiency in in Nepali, even though much like the French context, I was immersed. I lived with a family. It's not like there were apartments in the middle of nowhere. So I lived with family. I ate with them. I went and taught in schools, spoke with people in Nepali, but my Nepali never got beyond a really conversational level. And sort of anecdotally, I believe that it was because I never became a fluent reader in Nepali, so it's written in Devanagari script and I never could read very well. And it was always at the level of sounding things out. And we know from research that reading is really important. It helps us build vocabulary, it helps help us exposes to more advanced grammatical structures, reinforces the grammar. And without that, I just never got to be very high level in Nepali. Which is, which is unfortunate. Unfortunate. And that kind of as we'll come to maybe later in the discussion, that kind of difference in my attainment was one of the drivers behind why I wanted to explore a second language acquisition, so why I could achieve high proficiency in one case and not in the other case and understanding why that might be, and now I am exposed to a lot of German. So my husband is German. We're raising our son bilingually. So I speak English to our son. My husband only speaks German to him. So in the home I'm exposed to a lot of spoken German conversation. I hear German TV in the background. German radio, we go several times a year to to Germany. So again, I'm kind of learning German through incidental exposure and my conversational or passive understanding of Germans pretty good, but I can't really say anything. And I haven't tried reading German yet, and I know that if I want to become proficient in German, I'm going to need to put some some effort in it. But again, I'm, I've been able to learn a fairly impressive amount of German not having known a single word in German beforehand, just through this incidental exposure.

Dr Marie-Josee Bisson

That's amazing. And it really similar to my experience as well of living in Germany with the family and having this immersive experience. Like you say, it's it's amazing what you can pick up, but there's just been so much, amazingly interesting information that you've just given us there and your language background. And I feel like we could do a whole podcast exploring, you know, using you as a case study to, you know, learning all of these different languages. And what an amazing experience to live in Nepal and learn Nepali. They're really, really interesting. But I also want to get into eye tracking research. How can we make the jump from this amazingly interesting language background to eye tracking research? I don't know. You told us a little bit about why you were interested in language and becoming a language researcher, kind of based on your own personal experience. Do you want to tell us a little bit more about that?

Dr Kathy Conklin

Yeah. So as I kind of alluded to, this difference in my proficiency in French and and Nepali really spurred me to want to understand how people learn languages and specifically a second language. And so when I came back from the Peace Corps, I decided that I needed to do a Ph.D. in second language acquisition research. And so I went to the university at Buffalo. And while I was there, I learned eye tracking. And this was really in some of the early days of more widespread eye tracking. So the equipment is was very antiquated to what we have these days. So people's heads needed to be completely fixed so that their, their heads wouldn't move and so their eyes would be stable. So we used to have to make dental impressions of people's teeth before doing the experiment. So we'd make these dental impressions. We would screw this dental impression onto a fixed metal bar, and people would have to bite down on the bar so that their head was totally fixed. We would strap them into this and then we would painstakingly track their eye movements because they were just on this dental bar. They would often drool while they were doing this experiments. It was maybe a kind of form of torture. I'm not sure if it would get back.

Dr Marie-Josee Bisson

Well, I'm assuming, you know, from an ethical point of view, that the volunteers knew what they were getting into. I'm sure it had been really well explained to them, but it's amazing that they wanted to go through the experience. But sometimes it's just really interesting as a participant to, to take part in research that is so different or that involves a really new or interesting piece of equipment.

Dr Kathy Conklin

Yeah. So we, we had to entice them with payments to do this and then they didn't know what they were, what they were getting into beforehand. Some people did after a little bit say it was too uncomfortable and then it would drop out of the experiment. But yeah, so this was like kind of the pioneering early days of eye tracking. So that's how I learned to do eye tracking. And then later the new equipment that's come out is so much nicer and user friendly. So we still have a chin rest that people set their chins on and a forehead rest to kind of stabilise people's heads so they don't move around too much while they're doing the language test. But it's certainly moved on from from what I learned in Buffalo.

Dr Marie-Josee Bisson

They don't have to bite on anything and drool all over the equipment. That's definitely an improvement. So tell us a bit more about eye tracking in language research then because you actually wrote a book about this, didn't you? Like the actual amazing expert on eye tracking. I'm so happy that you decided to join us on this podcast. Oh, good.

Dr Kathy Conklin

Oh, good. Thank you. So, just in basic terms, eye tracking equipment provides a means of detecting where people's eyes move when they look and then stop and land at certain points. And we call these fixations and in a lot of language research, we're really interested in the fixation. So we want to see where your eyes stop and we measure how long you fixate at a particular position. And if you fixated a particular position, we know you're spending more effort processing what's at that position. So whatever you're seeing is requiring more processing effort. It might be a complex image that requires more effort for you to decode, or it could be a word, and if you're fixating it for a long time, it's taking you more time to process that word. Either recognise it or kind of integrate it into the, your ongoing understanding of whatever it is that you're processing. So, so the larger discourse maybe be and conversely, if you've got shorter fixations, it means whatever you're seeing is easier to process.

Dr Marie-Josee Bisson

So our eyes don't move fluidly when we read them, made these little jumps.

Dr Kathy Conklin

Absolutely. So we make little jumps from one thing to another. One of the things in reading research we're often interested in is you come to something and your eyes actually jump backwards. So you come to something, you don't quite understand it. So your eyes may need to jump back in a text or a sentence to figure out ooh I don't get this word it doesn't make sense, I think with what I just read. So you might go back and look at something else. We're really interested, maybe where you go back to try and retrieve your understanding of what you've been processing, talking about this and explaining about fixations. There's actually different kinds of eye tracking equipment and some eye tracking equipment offers more precision and accuracy. So when people are reading, I can tell precisely what word or even a set of a cluster of letters that they're they're looking at when they're reading, be that a text or browsing the web on a web page or reading a language exam or engaging in some sort of TV viewing and subtitles. Or if I'm having someone watch TV, I can see if they're looking at someone's face. I can determine whether they're looking at their eyes. Someone's eyes or nose. Then there are other eye tracking systems that are less precise and less accurate. And in that case, I might be able to say, okay, I'm looking at one sort of set of words kind of that are near to each other on a page. But I don't know precisely which one you're looking at, or I know that you're looking at someone's face, but I don't know if you're focussed on their mouth, for example, or if it's their eyes. So eye trackers really differ in how precise they are and how accurately and precisely we can pinpoint where you're looking on a computer screen.

Dr Marie-Josee Bisson

So some eye-tracking is better for particular research design, whereas a less precise one can do the day job for something that you want to know, maybe a little bit less imprecisely. So, for example, if somebody's looking to the left or the right of the screen, you know, if if you could do that, that's fine too for some design. But yeah, so if you want to go to the word level, really, you need the creme de la creme, the expensive equipment. Can we talk about how expensive this is?

Dr Kathy Conklin

Yes, absolutely. If we want the creme de la creme equipment, it is really expensive. It's in the tens of thousands of pounds or tens of thousands of U.S. or Canadian dollars. It's really expensive equipment. So if you want to get into eye tracking, either you have to have access to a lab who has the equipment and that will let you use it. Or often people will have some grant or funding money to buy, buy the equipment for, for a lab. And also this kind of equipment that allows for this precision is going to take up some physical space. So the system I use actually has two computers. One that the participant sits at and one that the experimenter sits at. There's a little camera that sits, in our case near the screen, some of some of the eye tracking systems it's mounted on the screen. But you need space to put that put the computers that's usually in a lab, although we can take our system to schools if we want to test school kids. But again, you're going to need a physical location to put it in. Some of the less precise systems like Google glasses. They're just these glasses with a camera embedded in them. You can obviously take them anywhere. You can track people's eyes when they go to a grocery store. That's not going to give you the precise location of where people are looking. I might know that they're looking at this part of a shelf, but I don't know if they're looking at a particular item on a shelf or if they're reading the label on the item. I can't tell at all what exactly they're reading. So there really is a big difference in the systems in terms of how precise and accurate at least we can pinpoint people's fixations to what they're looking at.

Dr Marie-Josee Bisson

I've also seen recently some software that uses your computer camera to see where you're looking on your screen. What do you think of those?

Dr Kathy Conklin

Yeah. So again, if you're interested in whether people are looking at, you know, kind of a quadrant of the screen and those kinds of systems are usually fine. But if you want the precision to actually drill down, and look at the actual word that someone's looking at, those systems aren't quite good enough yet. Probably in the next 5 to 10 years we'll see that that's that's happening. So you'll just have a little camera that you can put on a screen or embed in glasses and you'll be able to to tell where people are looking, but we're just not quite there yet. And in language research, we often are really interested in, you know, precisely where people are looking. So I want to know if you're taking a language exam, which words you're having trouble with, or if you're watching a movie in your second language and you're using the subtitles, which words are, you know, causing more effort where you might go in the image to help you figure out what the subtitles might mean? So we really want this precise information in language research. So hopefully the kind of thing that you're talking about, just with a camera mounted on a screen or in glasses, will get more precise and accurate. But until then, it's really for language research. Most of us use these more expensive pieces of equipment.

Dr Marie-Josee Bisson

So if anybody's interested in conducting a study using eye tracking, then they'll really have to get in touch with someone like Kathy, for example, who has the equipment and is willing to collaborate with people.

Dr Kathy Conklin

Yeah. Yeah. And we do collaborate with with people, you know, who want to get into eye tracking or have gotten a system and want help with it. So one of the things, these systems take a little effort to learn, so you have to learn how to calibrate the camera so it tracks people's eyes well. You have to learn the dedicated software that goes with it, both in terms of setting up an experiment and to export the data that you want. And doing this takes some time and, and effort. So I think probably the best resources and you could probably speak of your experience as well, Marie, that one of the best resources are other people. So people who've done it before and can give you some tips and clues. But there also now are some, some other resources. Like you mentioned, the book I wrote is is really intended to help people get started with eye tracking. There's some other books that have also come out recently to help people. Some good introductory chapters and articles. But really, I think probably the best resource is having someone show you what to do and how to do it. And then you can kind of supplement with these, these guides that might explain what it is you're you're actually doing.

Dr Marie-Josee Bisson

Yeah, I definitely agree to that. I was very lucky to have Kathy on hand to show me how to use the eye tracker. But you're right, it's everything else that you need to learn around it. And thinking about your research design as well is really important and how you know which measures you can use and what they're going to tell you, thinking about all that before you even start collecting data. And and then there's the whole analysis bit, which we haven't got into, but sometimes you get some imprecision in your data. There are ways around it, there are ways to fix it, but yeah you need to spend a lot of time looking at everything. And yeah, no, I can talk from experience here that it is very time consuming but so interesting. Kathy just mentioned about films with subtitles. This was one of my first research studies when I did my PhD and we were looking at whether people read subtitles in different languages and it was really interesting to look at that with eye tracking. And yeah, we saw that people, even if the subtitles bring in language that they didn't have any prior knowledge of before, they would still exhibit regular reading behaviour. So that was really cool, a really cool study. Okay, so you've told us about generally speaking, you know how to use an eye tracker a little bit in terms of what measures you can look at. So fixations, saccade is the other thing. So that's the jumps, that's the posh name we use for the jumps between those fixations and then the regressions when people go back to a sentence or to a word that were that previously encountered and that can tell us a lot about how they processing information. And can you give us an example of one of your research projects in a bit more detail and kind of what you were interested in looking at and how the eye tracker was really the way forward to to give you information about that.

Dr Kathy Conklin

Yeah. So I'm really interested in formulaic language, which is kind of a fancy way of saying language patterns or word patterns. So these are things like idioms. So kick the bucket meaning to die or spill the beans meaning to reveal a secret. So I've done a lot of research with idioms, but I've also been interested in what are called binomial. And these are two words from the same part of speech. So now two nouns can by a conjunction. So these are like fish and chips, salt and pepper, time and money. And I'm really interested in how people process these. So these are patterns that we encounter repeatedly. So I probably hear fish and chips several times a week. I hear people say it or I walk by a chip shop and see the word fish and chips. So how does my brain process these these kinds of language patterns that I encounter a lot? So one of the early studies I did with a PhD student back then who's now gone on to a successful research career herself, was to look at how people process these binomial expressions. And it was also one of my first experiences using a more modern eye tracker that didn't make you strap people in. So we eye tracked people's eye movements to these binomial expressions compared to the reverse form. So looked at how people read fish and chips in a sentence versus chips and fish or salt and pepper, versus pepper and salt. Okay. And we tested in the lab people who are native speakers of English, as well as people who were non-native speakers of English. So people for whom English was their second language. And we found that people read the binomial. So these things that they encountered a lot in language significantly faster than the reversed form. So things that I encountered a lot, they're really fast at reading. And this was true for the native speakers as well as the high proficiency non-native speakers. But the low proficiency non-native speakers didn't show a difference between the two. And we kind of hypothesise that that's because the the low proficiency speakers didn't have enough exposure to this. They weren't familiar with it. So there was no advantage to that. So more recently, that's what I've set out to really test is the emergence of what we call this formulaic advance.

Dr Marie-Josee Bisson

How we learn those. I think the really hard to learn for myself anyway, is if I make a mistake when I'm when I'm speaking, it's because I'm not using the right conjunction for example in between those two words or one, one of the word is fine, but it's not. I'm using a synonym for the second word and it's kind of interesting errors, I think. Yeah, carry on. Sorry.

Dr Kathy Conklin

You know, and those are the kind of things that really mark you out as a non-native speaker.

Dr Kathy Conklin

So you put things in the wrong order. You say chips and fish and you no one would be forgiven for thinking that the chips are more important to a lot of people than the fish and put the chips first. But so it really marks you, marks someone out so you can, you know, chips and fish is perfectly grammatical. There's nothing wrong with the utterance, but if you say it, it somehow marks your speech. Maybe you're trying to, you know, emphasise something about the chips. But for non-native speakers it marks them as as not, you know, saying things kind of in the canonical way that that a native speaker would. So a lot of my research now has focused on this emergence of this advantage for non-native speakers. So we recently did a study where we made up a bunch of these binomials. So we wrote a story, different stories. One of them was about a couple who bought a new house that needed a lot of work. So it's going to take a lot of time and money. A nice I know how to fix up. And we embedded these new binomial that we made up. So the wires and pipes in the house were awful. The wires and pipes were going to need to be replaced. The joiner had to come in and fix all the wires and pipes. So in these stories we had embedded repeated binomial wires and pipes that were made up, people wouldn't have encountered before. And we found within 4 to 5 times of encountering these in the story, people started to have this canonical processing advantage. So after seeing wires and pipes four times, then seeing pipes and wires to reverse form, there was a bit of a penalties about slow down reading it. We've recently collaborated with some people in Saudi Arabia to repeat this experiment with non-native speakers of English speakers who are native speakers of Arabic to try and see if we see the same sort of pattern emerge. And we do see some similarities so that you get kind of a processing advantage by reading wires and pipes a bunch of times in these stories, but you don't get a disadvantage for the pipes and wires. So it looks like they're starting to associate the two words in memory. So they're fast seeing the two words, but they haven't been able to learn that the order matters so that it's wires and pipes, not pipes and wires. And that might speak to what you just said, that these words might become associated in your brain. You might know that the two words go together, but that what the order is might take more exposure. And maybe it's just something that second language speakers aren't as attuned to.

Dr Marie-Josee Bisson

Really interesting. Yeah. So like you say, we learn to associate the two, but the order doesn't stick. Oh, yeah, I can totally relate to that. I would be the one to say chips and fish. Absolutely.

Dr Kathy Conklin

Chips are more important, right.

Dr Marie-Josee Bisson

But it doesn't sound wrong to me, you know? So I think this is why sometimes I notice that I've said the wrong thing or I'm like, hang on, is this the way to say it? And I'd actually ask people and they'd be like, no, no, it's it's fish and chips. So it's like, it doesn't sound wrong to me. But I wondered about that if there was a difference between the, the spoken form and the written form because all you study is obviously using eye tracking. You're looking at the written form and how quickly people are learning get if they've seen it in writing. Do you think there's a difference if they are just hearing these as opposed to seeing those? Are we kind of faster at learning it in writing?

Dr Kathy Conklin

Yeah. So that's a really a really interesting question. And the student I mentioned for the first study, the Ph.D. student, she's continued to work on formulaic language as as well. And one of the things she's worked on more is the spoken language. So if we encounter these in spoken language, do we get the same sort of processing advantage? And it seems like that, that, that is the case.

But in general, more of our research involves reading simply because we can use tools like eye tracking. But if we use spoken research experiments that are much more difficult and challenging,

So you just see a lot less of it and we understand a lot less about processing of spoken language. And the experiments tend to be less natural. So yeah, eye tracking allows us just to present things on the screen as we normally would. So you can just watch a movie. We don't have to ask you to do any sort of other task, like making judgements. Where as a lot of the experiments involving spoken material usually involve some sort of decision component. So you hear something, you have to make a decision about it and we can measure your accuracy, your response times. So we just have a lot less, less research on spoken language and also production when people produce language. So I might hope that you produced by binomials when I talk to you, but you know, you might never say a binomial an entire conversation. So then I don't have any data to work with. So our understanding of spoken comprehension in production is much less just because it's harder to design experiments that are that are natural.

Dr Marie-Josee Bisson

No, I understand. Yeah. And, and from, from what you said as well earlier, you know, the importance of the written input when we learn another language and how that gives us a big boost in terms of our learning. So I think it's really interesting to look at the written, the written stuff and a couple of extra questions I had about eye tracking in reading because well, I'm assuming that our listeners are potentially not familiar with what is expected in reading. And can you tell us a little bit about the nitty gritty of, you know, when we read and we fixate on certain words? So I know there's a lot of words that we we tend to skip or we see a few letters one way and more the other way. Can you just go through that?

Dr Kathy Conklin

Yeah. So that so that's true. So when we read, we actually fixate probably only around 70% of the word. So we skip about 30%, meaning our eye does not land on about 30% of the words. And that has to do with kind of the second part of your question. So when my eyes fixates on a word, I can actually see a few words or letters to the, for readers of English, Spanish, French, where reading is from left to right. We can actually see quite a few characters forward ahead in our reading to the right. So if I fixate the word dog, I can actually see the next few words. So that means I don't need to fix them. I'm able to decode and tell what they are with, without overtly my eyes landing on them. Interestingly, for speakers of languages like Arabic that are written from right to left, it's the opposite. So when they land on on a word, their eyes can fixate. And when they land on a word and fixated, they can actually see the upcoming words to the left hand side. So if they land on dog, they can see words that are coming after it to the left. So they don't need to then fixate those, especially if words are short. So things like a, the, and we see that those kinds of function words don't get fixated a lot. So we tend to focus more on the content words which are actually the longer words. And then we can usually get a little bit of what we call preview of the upcoming words and then we don't need to to fixate them.

Dr Marie-Josee Bisson

It's amazing, isn't it, when you start to look at it in such detail. What do you want people to remember from this podcast on eye tracking in Language Research?

Dr Kathy Conklin

Yeah, so eye tracking has been a really transformative tool in language research, so it has allowed us to present people with language in a very natural way. So you can just look at anything visual on the screen and we can see where you look and see how much effort you're expending to look at visual objects, language that is presented on the screen. And we don't have to ask you to do unnatural tasks. We also don't no longer have to, you know, have a dental impression and, you know, fasten you to to the, to a, a big pool. So it is these days really a natural sort of experience. You know, we read from a computer all the time, we read books off of iPads. So it's really a transformative tool that has allowed us to look at language processing in a very naturalistic way. But as we've mentioned in the podcast, the equipment is really expensive if you want this high precision equipment. So it's going to take time to learn. Prepare yourself for that if you want to get into eye tracking. So find a lab that has the equipment and find people who are willing to work with you because they are going to be your best resources and learning how to use the equipment, calibrate it, set it up, learn the software. They can also help you with the data that comes out of it that can be quite complex and they can help you maybe with some of the analysis and the statistics. And also these days there are some nice guides. So you mentioned the book and there are other books and, and chapters and, and articles that are really helpful. So I guess be prepared for the effort and look for good people to work with because that will make the, the efforts really worthwhile and fruitful.

Dr Marie-Josee Bisson

Thank you so much, Kathy. Thanks for joining us today. It's been a pleasure to have you. And thank you for all the really interesting information you've told us about eye tracking.

Dr Kathy Conklin

Thanks so much for having me. It's been really nice talking to you.

Dr Marie-Josee Bisson

In the next podcasts, we are going to be expanding on this because we'll be talking specifically about learning vocabulary through reading with Dr. Ana Sanchez. I just want to remind you to take 5 minutes to go into our show notes and click on the link for our survey. The survey will tell us what you've liked about the series and what you would like to hear about next. Thank you for listening and thank you to the British Academy for funding our podcast. I'm Dr. Marie-Josee Bisson,and you've been listening to the Language Scientists podcast.