Peter, Nick and Fraser discuss what the point of averages is, and whether we will need them any more in a world of machine analysis.

For more information on Aleph Insights visit our website https://alephinsights.com or to get in touch about our podcast email [email protected]

Hello and welcome to the Cognitive Engineering podcast produced by Tell Me Studios for Aleph Insights. In this series of podcasts we take a look at interesting topics and discuss what we think they tell us about analysis and decision making. I'm Fraser McGruer and I'm here with Nick Hare and Peter Coghill of Aleph Insights and this week we're discussing averages. So Peter, one definition of an average is a single number that represents a set of numbers. Can you start us off by answering how that function is useful? Sure, so decision makers

are often held distinct from analysts in organisations, particularly in government, that distinction is actually written in policy and it's the decision maker's job to make the call at the end of the day. His job is to take into account information he's given including what the political aims are, what historical examples tell us and decide what is the best thing to do and it's the analyst's job to provide him with some of that information that he needs to do that job. These two roles exist because it's necessary for the decision maker to be separate from the analyst because you need a lot of information to be distilled and aggregated into a smaller set of information and that's really what an average does. It takes a big body of information and puts one number to describe all of that information which is a summary descriptor of everything there. So it aggregates lots of information into a small amount of information, it compresses the information into a smaller space to let the decision maker take on board more data sets than they would have otherwise had capacity for. Now that explains why it's useful but it doesn't explain how to do it and that's where it becomes much more complicated. Now there are many different types of averages and depending on what the data is saying and what the data is about your choice of average will be different and so your arithmetic mean might be misleading compared to say the mode or the median depending on what it is you're looking at. Now the average itself is a model of the information, it's just a very simple way of describing this information and as we know any model will be an incorrect vision of the real world and so it's very important to take that into account. So just choose your average carefully

and to describe your information accurately. Okay, where do we want to go from here? I mean we can talk about the history of averages or... Yeah I mean it was interesting to think about

why averages were first actually used in science and it was to minimise errors so people would, or at least not minimise but to try and sort of get rid of the influence of errors. If people were making observation about for example the position of a planet or the weight of a particular element or something they would repeat that observation several times and take the average, the assumption being that on any one of those observations there might be some kind of error, some sort of human error or something to do with the measuring equipment which would bias that measurement for that particular observation. But if you take lots of them, those errors will sort of cancel out and actually I think that's where we get the term error which has a very specific meaning in statistics. But I'm afraid this in terms of this discussion of averages, the role that averages have sometimes is a kind of instrumental one which is that as a very general formulation of what we're trying to do when we're analysing things is we have some data and what we're trying to do is recover the process that generated that data. We're trying to make inferences from just the data points we have to what the thing was that produced those data in the first place. That's what hypothesis testing is. So we've got some observations about men and women and their heights. And we might want to know what's the probability of some random man being taller than some random woman. And we might try and build a model of the thing that generated those heights. So we take the heights of 100 men and 100 women and they're just a set of numbers. What we're interested in is what's the process that has generated those things. And the process in the real world is going to be what are the things that influence the growth of a human. And we might be interested in the extent to which that human's gender has an impact on their height. And the average is going to be one of the key things we're going to look at there when we're trying to build a model of the data generating process. But it won't be the only thing, of course, because it doesn't tell you about the spread. It just tells you about the average, which has a very specific definition. But we're also interested in the spread. It might be that women are both shorter and more tightly clustered or whatever. So there's another thing you want to know. And then going beyond the spread or the variance, there are other measures of data which will tell you more about the shape of that distribution, how spread out it is, how peaked it is, whether or not it's skewed to one side or the other. And all of those things are tools that you can use to try and pull out from a set of data what the shape of the underlying process is. But I think the question is, whether or not there's something inherently useful about an average. And I think theoretically, at least, there isn't. It doesn't tell you anything that isn't already in the data. In fact, it contains a lot less information than the data itself. So the two questions here, firstly, do we take averages ultimately and talk about them and write about them simply because it's easier for us to understand? So it's circumventing a cognitive limitation. And if so, with the rise of machine inference and big data, will we no longer really need them?

Okay, so that's what I wanted to come on to, is cognitive limitation. Before we go on to that, is there anything you want to add at this point, Peter?

Well, only to flesh out there. I mean, all of these tools that Nick describes, the averages of different types, these different statistical descriptors of data, they're all models in their own right. This is a simplified version of a thing. Now, there are lots of problems with models. And one interesting one is rarification fallacy, where you start believing that the model is the thing. So if you're only given descriptive statistics, say the variance and the average of a dataset, then you would be reasonable for you to assume that it had a normal distribution. And therefore, this normal distribution fully represents the model. And the real world thing that produces this data is essentially a normal distribution random number generator. But that's not the case. It's going to be a much more complicated system if it's a real world system. But it's an interesting fallacy to be aware of, is where you start believing that the model fully represents the real world thing that you're considering. Sure. Okay, that makes sense.

Yeah, the map is not the territory, as it's often quoted.

Yeah. Okay, so we started talking there. So I'm going to summarize, correct me if I'm wrong, but averages are a useful tool to understand data. And one of the reasons why they're useful, or mainly the principal reason why they're useful, is because of cognitive limitations. We can't possibly understand all the data that's out there, and this just helps us to do so. And one thing, and we've talked about the history there of averages. So let's sort of look in the other direction. And one thing you were starting to talk about there, Nick, is, I guess we're talking machines, technology, computers, does that alter the landscape? Does that alter cognitive limitations?

Well, I mean, I think the, what I'm referring to really is the fact that we should expect more and more are statistical testing, if you like, the testing of theories, the test and the finding of theories. So the finding of those, what we're trying to do is essentially find out what the real world process was that produced the data we have. More and more, we will see that being done by machines. So at the moment, if I give you a chart with two massive data sets on them, you've got a big load of red blobs representing one category of person, a big load of blue blobs representing another category of person. If you've got enough data points and they're sufficiently messy, you are not going to be able to tell by eye, you as a human are not going to be able to tell whether there's any interesting differences between those two. But a machine will. I mean, it will say, okay, there is a difference in the average of these two categories, and it's statistically significant, meaning whatever that means. And so will we need it? I mean, the idea is, you know, in future, will we expect to be presenting statistics to humans and saying, here's the numbers, what do you think? Or will we have to take the word of some kind of inference algorithm, which has done the sums, and it just, you know, it's identified that the process probably has these characteristics, but we might not necessarily be able to follow the statistical reasoning, because it'll be too complex for us. And it's something we touched on when we talked about AlphaGo a few weeks ago, the Go playing artificial intelligence, that we should expect to see machines doing things and forming, giving us beliefs that we don't understand ourselves, we're not capable of following exactly why those conclusions have been reached. So yeah, I mean, what I'm saying is, I think in future, the idea of being given descriptive statistics, and expecting to be informed by that may be something which is going to gradually be replaced by simply being given the conclusions of inference algorithms that we trust. I don't think it's a million miles away from what generally

happens already. So you have people conducting analysis on your behalf, if you're a decision maker, often, you're presented with various summary statistics and pieces of information. But do you actually know how that was generated? All the time? Do you know? Do you do you understand the data itself? No, because you've got people who do that for you, and then provide you with advice on what is most likely to happen? Or what is the most risky thing? Or what is the most, what is the highest expected outcome? So it's not that you're just trading one black box for another, one happens to have humans in it, and the other one now has a an AI in it.

Yeah, I mean, you Yeah, I know, there are some people who sort of don't, don't really trust their satnav. I'd like to second guess it, you know, and say, well, they might, you know, that doesn't know, it must not know about these shortcuts, it can't be can't be really optimising, it's not taking account of the traffic on the North Circular and that sort of thing. I mean, if that's true, it won't be true for much longer, right there, they're gonna be they're satnavs are gonna at some point be absolutely optimal at finding the quickest route. And, you know, are we going to ask for the audit trail? Do we expect to be able to understand what an audit trail will look like? You know, it's going to take account of a statistical model of of lengths and times, which is going to be informed by a vast amount of data about observed traffic flows, you know, where it's going to say, look, we, we know that, you know, 14 minutes past two, this time on that stretch of road, it will, on average, take, you know, nine minutes to get from here to there. And, and that will just have to take take his word for it, you know, because because what's the data set going to look like, it's going to be too vast for any human to potentially understand. You will you will be able to ask for the audit trail,

and it would you could ask the computer to present it in such a way that's human readable and understandable, but it might be so vast, it would be an intractable read.

Yeah, yeah. And I feel like the the data would the presenting it in a human readable fashion wouldn't be that much different to simply telling someone the right way to go. You know, it would, it would look pretty much like the route on a sat nav.

Yeah, yeah. What's interesting, though, is you strike me as guys who kind of care about what's going on underneath the hood. Okay. And, and, I mean, and also, it's this question of which we talked about in another podcast of civil servants versus politicians. Because one of the things you said earlier, so first of all, it's about it's a refining of that process. And two, does this decision making actually care beyond the point of this is better information or not? Okay. And because I must admit, a lot of this, I just go, Oh, God, let's just, I don't care, you know, but I guess I'm pleased to be like the Church of England. I'm pleased that there's people that sort of are doing this sort of stuff. I don't know, you know, but I don't want to get involved myself. But I don't know what my point is. But actually, sat navs gives a good way into this for me, because I makes it more understandable for me. Because I think what's interesting is, is I use about three different sat navs at the moment, all at the same time. And one of them is just a sort of an old fashioned 10 years old Tom Tom, right? Which I think is pretty fixed in a way it does stuff. I always also use Google Maps. Okay. And I think it's interesting, often the different things too. I don't know how so Google Maps, I think it uses, but it's more, it uses other inputs, right? So and it will be constantly changing to do with traffic, etc. How it sources it, I don't know, but it sources it from your phone, work out how quickly your phone's moving on. No, no, no, but nobody will talk about traffic ahead as well. And things

like that. So you're not just your phone, but from your other people's phone or in the wider

sense from other people's phones. But how does it know about the traffic? I guess how fast they're

moving or not, right? Yeah, moving and how fast they should be moving for that class. Okay, so and already

that makes me go, wow, that's pretty cool. But I discovered a new app last week, which I don't know if you're familiar with called Waze, which is spelled W A Z E. Okay, which apparently is even more accurate than Google Maps. And what it does, it relies on peer input as well. So it relies on someone going, hey, I'm here and there's really bad traffic or there's been a pigeon just flown under a truck or something. I don't know. Although that probably wouldn't hold up traffic too much. But I think what's interesting is... If it was an elk. Yeah. Pigeon, not so much.

That one's just for information. Yeah. A bee has been sighted in the middle of the A41. Yeah.

But apparently, and I don't know if this is, I don't know if it's true, but I think it's even more sensitive. And it's, I guess it's crowd sourcing, right? And it's going even a little bit further. But I think was the irony here is that it's actual human input into this system, which is making it even more accurate. Well, I'd be interested to see if it is actually more

accurate. If you can expect more efficient journeys with Waze versus Google Maps. Because humans generally, if you're designing a system, you try and look for ways of taking the humans out of the loop because they only really introduce error rather than eliminating it. So if you can source the same information, and I think an automated process of the phone just reporting its speed back to the Google server and the Google server then knowing for that road it's likely on, how fast it should be going. It's probably more use, probably better than people saying, oh, I've been sat here for 10 minutes, I'm in traffic.

But it's interesting here. I mean, this is, I've been struck by a thought really, which is that, you know, we're fairly sceptical of the idea that expertise by itself is going to make you necessarily good at doing analysis, right? Knowing about stuff. But the situation you've described here is a bit like the situation of being presented with the analysis of three different experts, who have different background, perhaps a different record, you know, different methods. And you don't, and I think that's a position a lot of people are in, in, you know, we are in the position of listening to, I mean, you know, people make economic forecasts, and three people might disagree with one another. And we have to sort of decide how much we trust them. And I think it's interesting, the idea that actually we are moving through a situation where we can improve human analysis, make it better, until eventually, we get to the other side, and we are really sort of forced to rely on machine computation to form our beliefs for us.

And that's a good note to wrap up on. But what I want to say, I think what we should do for another podcast is, is I should assign you to a task of using these three different devices, you know, let's actually do it. And you'd have to research, you know, into what the, you know, machinations of each one is. And then the more complicated bit is then actually do it.

But three of us can pick a crosstown objective. And we'll each use a different system to get there.

We go, there we go. I like that. Wouldn't be boring if we all arrived at the same time. But okay. All right. Although we might have had different top gear. Yeah. Okay, as long as you don't punch the producer. Okay, let's stop there. Chaps. So that was averages. Thank you very much. I think I'm clearer. Yeah, I am. I am clearer on some things there. So thank you, as always, to Nick Hare and Peter Coghill of Aleph Insights. And until our next episode. Thank you. Bye bye.