The rise of Donald Trump prompts Nick, Peter and Fraser to discuss the limits of forecasting behaviour.

Links referenced in this episode:

• politifact

Hello and welcome to the Cognitive Engineering Podcast, where we take a look at interesting topics and what we think they tell us about analysis and decision making.

My name is Fraser McGrewer and I'm here with Nick Hare and Peter Coghill of Aleph Insights.

And today we're going to take a look at the American primary elections for their candidate for president.

Specifically, I want to take a look at Donald Trump, who sort of no one thought he had any chance at all.

He's come out of nowhere and now he is a serious contender for becoming the Republican nomination for the presidential election.

Specifically, I think we want to take a look at this in terms of uncertainty and what it tells us about uncertainty.

No one expected this to happen.

So, Nick, can you start us off?

Yeah, I think.

I don't think.

It's probably not true to say nobody expected it to happen, but I think it wasn't something that was totally unconsidered.

I mean, but, you know, it's only in the last couple of years, you know, that I think from what I understand it, that Donald Trump has been interested in the idea and, you know, through a series of events, has ended up in a position where he appears to be leading the Republican primaries.

So that, you know, it's surprising point.

The point about this is it's sort of surprising and the question is to what extent you can predict things like this with better information.

So the elections are a nightmare for predictions because we.

Loads and loads of data, we always have loads of data on polling.

You know, what people say they're going to vote and we're continually surprised.

I mean, in the last UK election, all the polls said it was going to be a hung parliament.

So, in other words, there wasn't going to be a party which had the majority of MPs, and then in the end, the Conservatives got a majority.

It was, I mean, pretty much unforeseen.

And the question is what kinds of uncertainty are in play here?

And I think the trouble is that people confuse two very distinct types of uncertainty.

One is statistical uncertainty and one is model uncertainty or uncertainty about the way the world works.

So the statistical uncertainty is essentially saying is the poll that we've just taken a good and fair sample of the whole population.

You know, we know that if we're only asking a thousand people, we know that there's a chance that actually that we might have picked a thousand people who are kind of more conservative or more Labour.

And you can kind of work out what the.

What that.

How big that uncertainty is.

And I think that encourages people to think that they've sort of, sort of bounded what uncertainty we have.

But actually the really big uncertainty is about the connection between what people say in opinion polls and what they actually do on polling day.

And that seems to be where most of the uncertainty is.

And yet people always cling to opinion polls and think that they have more information than they do.

Sorry.

So that uncertainty you talked about there then, is that uncertainty to do with the model then?

Well, yeah, I mean, in other words, uncertainty about the way the world works.

So what we really don't know is that connection between how people answer opinions polls and what they actually do, whether they vote at all.

And then if they do vote, you know who they vote for.

That's where nearly all of the uncertainty about election forecasting comes in.

And we're inclined to forget about that, to focus on the statistical uncertainty and kind of ignore the fact that actually opinion polls don't give us very good information about behaviour.

Okay, so that being the case then, Peter, so do we just give up then and go, well, there's no point polling.

I think there is a point of polling.

But polls need to be perhaps designed differently and different data can be collected from different sample populations and that would potentially improve polling.

But I think there are all sorts of inherent biases in the way that selections are made.

The people that are asked the questions and the types of questions that are asked often carry either a deliberate or accidental agenda with them.

But I think it's a well defined problem that people's behaviors differ from their opinions.

And so what people say they're going to do is radically different from what they usually end up doing.

I'm guessing that no one is more aware of this issue and this problem than pollsters.

Right?

So I presume that they've got lots of very clever people sitting around working out how to reduce this discrepancy between what people report and what they actually do.

Is that the case?

And do you know of instances what I mean, what are they doing about this?

And it's a famous study in polling because they chose who they were going to ask from lists of, in the phone book and lists of club membership and that sort of thing.

And of course, you know, really only middle class people had phones and were members of clubs and things.

So it showed that Alf Landon was going to win.

And of course, the fact that nobody's heard of him tells you that he didn't.

These days there's all kinds of techniques that pollsters use to make sure that statistical uncertainty is minimized.

So, you know that we've got a good sample that, you know, polls are made, they deliberately randomize, you know, try to make sure that they are asking people in a way that is completely uncorrelated with any of their characteristics.

You know, that's where selection bias comes from.

If the method you're using to choose people is in some way correlated to.

To something about them, then that's where you're going to get selection bias.

And they use all sorts of techniques.

You know, they'll randomize the order in which they'll ask people questions, they'll present the candidates in randomized orders, all sorts of things to try and eliminate that.

But you know, as I said, I think that the problem is that actually a poll, even if we knew perfectly what people say they're going to do, that still only gives us some information about how they're going to behave.

And most of the uncertainty is actually not the statistical kind.

So, Peter, so still we've got that gap.

So why, what is it?

I mean, I'm no sociologist, I don't know.

So why is it that someone reports something then does something else?

I'm not sure.

I think that it's a very well studied subject, but it's just something that we do.

When it comes down to the moment, we often do something different or we, we want to satisfy some desire of the person who's asking the question, so we'll go along in some way.

But I think there's lots of different reasons it happens.

I think potentially better predictors are historical studies and moral development of people's past behaviours.

So an example might be if you could harvest a lot of social media data from Facebook and Twitter and things about what people did, where they went, who they interact with, that would potentially give you much better models of what kind of person they are, which then may correlate well with how they eventually would vote, rather than asking their opinion.

So don't involve their opinion at all.

Cut that out.

Just go on historical data about what they have done.

The big limitation there, of course, I mean, particularly here in the uk, thanks to the pesky Chartists, we don't know what people have voted.

That's the problem.

So we can't link people's characteristics to their actual voting behaviour.

We have to go on what they say.

But what is interesting is that, that exit polls are always phenomenally more reliable than any polls leading up to the election.

I think that says something quite interesting.

People do not lie about what they have done, but they may be mistaken about what they're going to do.

So actually we were speculating on why do people's expressed thoughts differ from their behavior.

And my background is I'm an economist and economists do not trust anything people say.

The people, in order to conduct experiments about how people behave when you change the conditions they're in, we really only look at behavior.

We don't ask people stuff.

You know, we put them in situations where they, they have incentives to behave in certain ways.

But what you can't do when you're studying economics is look at what people say because it just, you know, for all sorts of reasons, people don't even know themselves.

It's not even that people are consciously lying about what they're going to do.

It's, it's also the, the fact that they themselves, their own behavioral preferences are not necessarily transparent to them.

So it's only.

Yeah, and it's interesting that I guess something doesn't happen until it's happened.

Right.

And that's the only, that's the only sort of real test.

Right.

I don't think that was a particularly clever point or question, but thank you.

Thank you.

Okay, well, look, anything else you want to say?

I'm going to round this off by turning it back to Donald Trump, which is where we started.

But before I do that, anything else you want to say about this?

Well, it's a bit, it's off topic from the other thing, but it does relate to decision making.

So consider the decisions of the voters who are voting for Trump.

I think there's a particularly interesting quality to the types of language, the rhetoric that Trump uses.

There's a brilliant website called politifact which breaks down what politicians have said and does fact check to see whether or not how true or how false those statements are.

Trump broadly scores pretty poorly compared to the other runners.

And I think it's sort of looking at the language.

I think it's to do with the sort of rhetorical vagaries in the language he chooses to use, which gives his speeches more, they're generally more attractive to more people because they can mean many more things to lots more people.

They're light on fact, they're light on specifics.

So they're very general hand waving comments.

Whereas if you compare him with Obama or Clinton in the other primaries, they are Very fact driven and very evidence driven because they're talking about things they have done and things that are demonstrably true.

But that's less interesting, it's more boring.

Well, Phil, I'm going to come to you in a moment, Nick, but that reminds me, I recently saw a YouTube video which provided some analysis of one of his short speeches and it was more or less, as you've just said, the way he speaks is very simple, I think.

And you mentioned the word rhetoric.

I mean, he is a study in rhetoric and the way he speaks.

Very short sentences, very simple to understand, lots of repetition and lots of repetition of key words.

Sorry, I forgot to say very short words as well.

And he always seems to end, his sentences often end on a similar word like that.

It's often a negative word as well, which seems to sort of get drilled into the, into the mind of the listener.

And just one of the points it made in this, in this film was that there's maybe you're familiar, there's been a study made of the reading age who could understand a Trump speech, who could understand a Clinton speech and all other politicians.

And his is really, I don't really understand the American grading system, but his was, I think 6 year olds could understand it, whereas with, with Clinton you need to be at least eight, let's say.

So.

Yeah.

Nick, you wanted to come in?

Yeah, I just.

Well, I want to kind of defend that a bit.

I mean, I think, I think, you know, and I'm not, certainly not speaking up for Donald Trump particularly, but, but the question is why we should expect politicians to make truthful claims.

I think sort of the fairest thing to say about Donald Trump is he doesn't really make claims and that isn't his appeal.

It's not based on the fact that he's saying things that are true.

I actually think there's an argument for voting on personality.

It's not one I'd be particularly attached to, but what you can say is the politicians can't foresee all the things that are going to happen, you know, in the next four years.

So voting on personality is a way of saying, well, you know, I actually just want someone in charge whose values I share and who is going, whatever, you know, we're not going to hold them to their manifesto necessarily, but we kind of think that they'll do things that we'll support when they're in, when they're in place.

So I think, you know, there's this assumption, I think that we should hold politicians to account for things that they say and for, you know, not following through on their manifesto.

I don't know if that's necessarily actually the ideal situation.

Well, I'd just like to say two things there.

First of all, I'm glad no one holds me to account for the things that I say publicly and then turns around, says, well, hold on, you didn't do this, because there's probably about, you know, don't tell my wife about this kind of stuff because I'm sure I still haven't done the washing up from this morning.

But that's the first thing.

The second, I'm astonished to hear you say that, actually, because I thought I was looking at myself because this is the kind of thing you often pick me up on.

And I'm saying, oh, it's all just too complicated.

All these decisions, all this analysis, all this data and stuff.

Oh, let's just kind of go with, you know, it's a binary decision.

Let's go with that one.

It'll be fine.

So I'm really surprised to hear you say that, and I'm glad that I've managed to convince you of my approach in life.

So that's good.

I'm going to wrap up this.

Any final things you want to say before we finish?

No.

Okay.

So this has been another episode of the Cognitive Engineering Podcast.

Thank you once again for joining myself, Fraser and Nick and Peter at Aleph Insights.

Goodbye for now.

Thank you for listening, Sam.