In today’s episode of Fibonacci, the Red Olive data podcast, we continue our discussion with Chief Data Officer Robin Hayden. Check out part one if you missed it. With over 20 years of experience in the industry, Robin has seen and done it all. In this part, Robin shares:

• What to consider if you are having a data strategy re-think.
• The approaches and technologies that are better than ELT (Extract, Load, Transform).
• Changing your thinking so that you can gain efficiencies and steal a march on your competitors.
• Thinking about data as something that will make the product experience a lot smarter, rather than simply as something you do in reports.
• Integrating machine learning models with products.
• The benefits of focussing a data team on their core competencies.
• Skills people should think about developing if they want to get ahead in the industry.

Let us know what you think by emailing us at hello@red-olive.co.uk, and please subscribe to the podcast to make sure you get to hear every episode.

- Hello, and welcome to the Red Olive Fibonacci Podcast.

The show all about the brilliant world of data,

covering future trends and topical tech.

We'll be joined by experts in the datasphere

to share their opinions and advice.

I'm your host, Nicky Rudd.

Today, we're going to pick up our conversation

with Robin Hayden, a data expert who spent

the last 20 years at the cutting edge of the industry.

In part one, we chatted about crypto, AI, data ethics,

the best way to use the cloud,

how to run a data product, and much more.

If you haven't heard it yet,

make sure you download it now and subscribe to this podcast.

So you don't miss any future episodes.

In part two, Robin gives the benefit of his vast experience

and talks about how close to the bleeding edge you should be

and the skills that people should work on

if they want to get ahead in the industry.

So let's go.

(energetic music)

Have you got any particular learnings that you can share

for someone thinking about

or rethinking their data strategy?

Before they start out, what should they consider?

- It's quite tough because it's so context-sensitive.

And obviously if you haven't, if you're not on the cloud,

well then you are behind now,

So you need to be on the cloud, so you need to get there.

I would say I'm always of the mindset that you should try

and stay sort of not quite bleeding edge,

but you shouldn't be far behind the bleeding edge.

I think there's a lot of organisations

that are quite happy to wait a long time.

I started in this internet world and was very exciting.

And then I remember at some point getting bought out

by telcos and they were quite conservative

in the sense that at the time there was things

like voiceover IP was a big thing

because it was all these traditional voice networks and,

but they were so hesitant to,

they had great margins on their products.

They had great margins on voice, on texts,

the actual data rates for things like texts

were unbelievable.

And so they were never really incentivized

to cannibalise their own products,

but then they also really bemoaned this thing

that they always used to talk about

these over the top products.

Like all these people are coming in,

like they were stealing their value sort of thing.

And then they were getting pushed down

to just become this utility, but then in some sense,

they were making that happen.

They were making that happen

because they were too conservative.

And I think a lot of businesses make that mistake.

I'm not suggesting people should go out and just be bold.

And just to sort of move fast and break things all the time.

I'm always use this analogy of a fighter pilot,

a fighter plane, and a Boeing.

So fighter planes, they sometimes design them

to be aerodynamically unstable,

because it allows them to move really quickly.

So smaller organisations can be like that,

but you don't build a Boeing,

a big sort of commercial airliner in the same way,

cause they have to be stable in the air with lots of people.

So there is some, a big organisation

needs to be a bit more mindful of the process

and that sort of thing.

But I think you need to not be shy to just be

a few years behind some of the leading edge stuff.

So an example in the data space now

is we're getting event-driven everything,

and for a lot of technologies, that's okay.

Yeah, we should just be event driven.

But a lot of organisations,

a lot of data organisations are still in the past.

They're still talking about things like we had ETL,

which was a very old way of doing things.

And you did all your processing

and then you put it in a database.

And you had ELT, which was better.

So where you take, you say, we've got this big database.

I could take my data, put it in a big database

and use the power of that to do the processing.

A lot of people are still talking about that

as if that's okay, that's the moment, that's the future.

And that's already becoming the past.

We've already got things that now are starting to do

a lot of processing on stream.

And you've got, traffic has got KSQL

and things like Flink over Kinesis.

And oh, you can use things like Spark over streams,

but I think you just need to think differently

and think, okay, well, I'm going to,

if I can count something in a database,

you can usually reframe that problem.

I can count, this is a simple one,

but there's a lot of other more complex examples.

But if you're going to do a daily aggregate

of let's say sales, actually, you could on the stream,

just listen to those as they go by

and update the counter and publish that

whenever it's necessary.

So that's an example of pulling your logic

all the way back onto the stream.

Now, why is that better than ELT or something else?

It's better because you're pulling all your,

we've had people that put logic, say, in reporting systems.

When you do that, you can't reuse that for your products

and you can't reuse that intelligence in other feeds,

in making your actual say, user experience,

more pleasant or something is only people

in your organisation can look at it.

You have to go and replicate that logic

in your product somewhere.

So you have to do twice the work

if you want that same thing to happen.

So you can bring it, in two places,

So you can bring that back

and you can put it in your big database.

And then it's a little bit better

because more people can use it,

but you still have to break out

of your normal development process.

And go and get something from the warehouse,

if all of these products are just producing streams,

and just one of the things that are producing streams

is the thing that's sitting over the top, listening,

producing its outputs that eventually do get pushed

back into a stream and then into a big database somewhere,

then it's available for reporting and all the rest.

But it's also available to anyone who's listening

to that stream to mix and match, to create new products.

So that's one aspect is don't be afraid to,

some of these concepts are not new,

but they're not that many years old.

They still newer, most people aren't 100% there yet.

I think you should adopt those things.

As soon as they look like they're partially stable,

you should adopt them and you should go all in,

because you will be more efficient.

And it'll give you a little bit of an advantage.

You'll achieve that efficiency just before

some of your competitors do.

Other things in the space, of course,

is I think that this is a bit of a harder one to express,

but I think people really do have to get comfortable

with this idea that we have reached the point

where I think the next decade is about intelligent products.

It's not just about intelligent products,

is highly distributed products.

As we see this coming up and things like the crypto space

and other things as well,

but certainly the products we build now,

the noughties was, big data started emerging.

So it was like an infrastructural layer, if you like.

Then over the last decade,

you've had this maturing of the machine learning space

and you've got a lot of high-profile

and Google played Go and beat the world champion

and the first sort of signs of kind of

self-driving cars and things.

There's a lot of kind of hype around that,

but it has really matured over the last 10 years

and these algorithms and stuff,

oh, are you getting AutoML and all these things

that are happening in a lot of these products

that are available just as a service, intelligence services

to do everything from kind of sentiment analysis,

to image recognition and all sorts of other things,

there's all these kinds of intelligence services

starting to emerge, and I think understanding that

and seeing data, not as something which you do in reports,

but that actually the next 10 years is actually about

the whole product experience is going to become

much, much smarter, and you need to think in that way,

think about how you're fully integrating your intelligence

into your whole product flow.

If you're still thinking about intelligence

as something you do in a warehouse and on the lake,

and that goes into reporting somewhere,

you're not going to win this battle.

That's not how this stuff's going to pay you back.

You have to think about how are you integrating

your machine learning models into your products

and just like any other software development process,

You should also continually ask yourself

well, should build it all?

I think there's a subset of things

that will be key core competencies that you're building.

It's a lot of sort of intelligence services

that are starting to arise now

that maybe they're not core competencies

and you should just borrow them.

Or even if it is a core competency,

if somebody is actually just able to do it

much better than you, there's no point holding onto it,

and you have to pick your battles as well.

But that mindset of move things further back,

processing further back,

all the way into streams, if you can.

So more people have access to the data.

The reports will not go away, they're very important.

People will always want to know,

and maybe we'll consume them, as, I don't know

in different ways, it won't just be visual reports,

maybe many other things in the future,

but feeding back that information to people

is always going to be an important thing,

but it's really important not to get stuck in the past

where data has traditionally always been used

as just a sort of a thing that you give to people,

and then they go and write code,

or then they go and do things with it in the business.

I think you have to think now in terms of

how are we driving our, all of our decisions

within our products and all of our experiences

with that intelligence directly,

I think that's just where we are.

And if you don't understand it now,

you're going to fall behind this business.

- It seems to me that say, a whole different way of thinking

perhaps for a future generation coming into this data space.

Are there any particular key skills that you think

anybody entering the industry should really have

as a kind of, will help them out?

Somebody, we've obviously had a really difficult year

and I'm sure there were a lot of people

who haven't managed to do as much kind of work experience

and all the rest of it as they normally would have done.

But if you're trying to get into the data industry now,

what would you recommend?

- There's these different paths.

There's the sorts of data engineering path

there's the machine learning path.

And you do get people who still just become

very good reporting people or analysts or something.

So I think first you have to decide what your ambition is

and how far up the stack,

or you want to learn, and some people do just,

I remember working with a really good architect

who I thought he was just wonderful with people.

Everyone really liked him.

And I asked him if he would be interested

in a sort of management role

and he wasn't interested at all.

So I think actually impressed me when he,

cause he was just like,

he's was very happy with what he's doing.

And I think if you, if you can first understand

what am I aiming for?

Am I trying to be a CEO, I'm gonna try to be a CTO,

am I trying to, where do I want to go with this?

And that to some degree might determine your direction,

but certainly the other bit is what are your interests?

If you just, you know, like analysing the world,

for example, you may not need to be a brilliant engineer

and all the rest.

I think there's still plenty of room

for really good analysts, for people

who are just mathematical mindsets,

but perhaps it just, you know, they just don't,

they don't want to spend all their life doing code

and that sort of thing, they just,

and they may actually end up,

you could start as an analyst

and move into commercial roles eventually.

I think if you're in the data engineering space,

I think all the traditional things,

you have to be aware of warehouses and lakes

and to be very good at SQL and be familiar with ETL and ELT,

really, and that sort of thing.

I think that that's kind of table stakes.

So if you don't know that stuff, it's probably,

it's important to learn it.

And there's tonnes of courses and stuff these days.

So it's not hard to learn, but I would say that's probably,

it is just table stakes.

You can't really call yourself

a good data engineer these days

if all you do is SQL, for example,

I think you have to be good at something else,

at Java, Scala, some functional, it could be Rust,

it could be whatever, things like Java and Scala

are obviously quite big in the data space.

But I think knowing things like functional,

the whole sort of functional paradigm

works really well with data

because I won't go into it now,

but because of the idea of immutable data,

when you're distributing things,

which you do large scale distributed processing of data,

then that functional thinking is very important

to understand because you basically don't have to worry

about things changing in multiple places

in your landscape and trying to coordinate that.

So the whole sort of functional paradigm,

if you're in the data engineering space,

I would say learn some sort of functional language

or learn a language that at least allows you

or something like Python as well.

The engineering skills,

getting to know things like streaming.

I think that's, and understand that sort of stuff.

That's going to be really important as well, going forward.

In the machine learning space, again,

I think they're table stakes now

is you have to know various different ways

to do classifications, tree-based methods.

And you need to know the kind of core sort of neural network

type stuff for the deep learning and all the rest.

So I think that's kind of table stakes,

or you have to be at, you have to,

I think the thing to remember there as well

is you still need to be quite,

you need to be a little bit of a domain expert

in that space too, I suppose, like, a lot of things.

If you know the domain, you still have to do

a lot of analysis to build a good machine learning model,

to understand the space, to figure out

what what's going to work in your models and stuff.

So I think don't just go down the rabbit hole

of being brilliant researcher in the machine learning space.

If you want to make it in business,

you probably need to understand your business

if you're in that space, but things that are coming along

in that space I think that are really interesting,

reinforcement learning is obviously it's been around,

the stuff that people like Google

was doing some impressive stuff with it,

or DeepMind was doing impressive stuff

with reinforcement learning

a good few years back, it's starting to mature now,

I would say really pay attention to reinforcement learning.

A little bit earlier stage I think,

but I think it's going to become very important

to the machine learning community

is things like causal, anything causal,

your kind of causal inference of some sorts.

So we have a bit of a problem with causality.

We don't know often, there's a nuance point.

So in the interest of time, I won't go into a lot of detail.

But what happened is that the machine learning

basically learns from the past and then projects forward.

And lots of what we learn is we learn complex patterns

and some people say that's basically

just very advanced curve fitting.

There's some truth to that.

And some of it's not entirely true,

but there is some truth to that.

And things like the pandemic have made it very obvious

that we had models that were working for a long time,

and then suddenly all the behaviours change

because everyone was suddenly at home,

lots of things change.

A lot of models just stopped working as well.

And so there, when you look at things

like the space where they're trying to learn,

think of it more like science,

they're trying to learn causality,

learn like how does the world actually work,

That's, machine learning models

don't actually learn that well,

but there's a lot of work going into that space.

And there's quite a few good things emerging in that space.

So I think that's probably going to become

more and more important.

And even when I was in the gambling space, for example,

you didn't need a pandemic for that to become important.

You could see regulations coming like a year out.

You knew that they were going to tell you

to stop using credit cards or something of that nature.

But the thing is you didn't have data yet.

Dealing with that can be tricky

in the machine learning space.

So I'd say that preferences, that kind of philosophy

I had earlier, in the commercial world,

you can't afford to be right on the cutting edge.

Because if I think about when Google first started

doing some of the papers around playing Atari games

and stuff is from the reinforcement learning,

you know, many years back, right?

And so you could have fiddled around

with that sort of stuff for years and the people paying you

wouldn't have been very impressed.

So I think there's a few places that might pay you

to do that, but on the whole, in the commercial space,

you still have to be aware of like,

the stuff that's working now that is established.

But I think you always have to stretch yourself

just a little bit further and say, okay,

what's like just on the horizon.

And I think reinforcement learning

is starting to become more prominent now.

And I think the causal stuff is going to become

a bigger and bigger thing over the next five-year period,

so I would definitely pay attention to that.

- I think it's quite interesting you saying

about the fact that, because I think within sort of data

and obviously people have to be good at maths

and IT it you're working in data space

where it's all about fixed rules, if you like,

but actually now it's having an understanding of that,

but also having that flexibility to let your mind go

and actually be brave enough to take those next steps.

I don't know necessarily if those two kind of sides

of the brain, I don't know, will work.

Yeah, a future data scientist

is thumbs up for both sides. - Well, I you do,

touch on a interesting point there.

And that is that the other thing

which is emerging a lot more now

is that there's a lot more AutoML.

This idea that you would automate

the training process and all the rest.

Now I still don't think we're at the point

where it's fully mature and you can just throw

AutoML at everything, we used to build hierarchical models

and things that they sometimes

still don't do well in those AutoML systems.

If you look at the history computing,

you go back to fifties, sixties,

a lot of the people that were working in early computing,

perhaps did need to understand a lot more the detail

and perhaps have to be mathematicians in the early stage.

And then gradually were at least certainly much more aware

of the inner workings of processes and things.

And then it's got to the point where there's tonnes of people.

I actually have been doing some React stuff,

some front-end stuff it's completely outside

of the world of data.

There's tonnes of those kinds of React front-end developers,

for example, who they're not mathematicians at all now,

that's what I mean.

They're just people who enjoy coding front-end type stuff.

Get good computer scientists who I think

still understand quite a range of things.

But I would say the average developer now

actually doesn't need to be,

doesn't need to be the sort of almost mathematician

of the early world of computing.

And I think eventually things like machine learning will,

and it's obvious that everything,

for things to be adopted at the widest possible scale,

they have to be available to a very wide community

and that's where things like AutoML come along.

So I do think we will see over time layers of this strategy.

We'll have just like you have the real sort of experts.

You don't have a lot of people

that will gradually start working in this space

that may just know the business well,

maybe some of those AutoML tools and things

are good enough and they don't necessarily

have to be mathematicians.

So yeah, it's a bit of a nuanced thing.

Right now, if I was hiring a team

of machine learning people, I think I still expect

a little bit of a mathematical head,

because I think there's still enough stuff in there

that now, you get caught up.

People just make mistakes within,

like causality versus correlation and that sort of thing.

There's the little things.

And it just don't understand some of the concepts

around statistics like IID and stuff like that,

which I think sometimes you make really critical mistakes.

But I do think I'm actually always pushing my ML teams now

to say, "why aren't we using more AutoML?"

And they've always got reasons why it doesn't do this right,

it doesn't do this right.

There is a part of me that thinks yeah, but, hold on.

There is a class of problems.

that doesn't, where that stuff works.

and I think the trick is making sure

you don't get caught either in this idea that actually,

it's back to not do I need to build it all myself.

Do I need to understand, does everybody who programmes

need to understand how registers work in a processor?

Absolutely not, and in fact,

sometimes it's probably a handicap.

So yeah, that's going to be the,

I think the challenge is managing the sort of move

from the one phase to the next, in that space.

- Yeah, it seems to me that there is a real sort of process

for moving forward in this sort of data space

of collaboration across technologies

and the storytelling or the reporting side of it.

Being able to actually understand what the data

is actually telling you and then asking it questions

of, you know, it being a bit more challenging,

you know, challenging the data to tell you more,

but also then as an organisation,

having the priorities and the trust in your data team

to actually make that a reality, rather than just a,

this is something that we'd like to be doing,

but we're all a bit too scared to take the jump.

Do you feel like that has changed in that people,

particularly with cloud, that because they don't have to do

that enormous sort of CapEx spend right upfront

that actually that move is a bit more acceptable

for larger organisations?

- I think it's, few execs now,

would brush aside this idea that you need to be data-driven

and more intelligent or something.

If an analyst was asking if you were a public company

and an analyst was asking you

what you're doing with your data,

I think few people would be comfortable saying

nah, we don't think that stuff is important.

So I think that they've gotten a hub cycle type thing.

I think we there's, this there's a portion of the hub cycle

we've got over already.

But yeah, I do think it's still presently working

in a situation where you know,

and I won't, I won't mention names and all the rest, yeah,

but where it amazed me just how much time

the exec suite was focusing on the app.

On what the app looked like

and the colours of the app and all that.

And this is not to downplay this right,

like branding and that sort of thing is quite important.

And having something that people just like the feel of

and stuff is quite important, but I think it's also,

I think sometimes it's also easy to do that.

It's very easy because everyone understands it.

I think a lot of execs, I just, you know

they're people as well, they make those mistakes.

They have strong opinions and they carry their opinions

into things, And that's why they might focus

on something like an app or something,

because it's easy for them to kind of state that opinion.

But even when they're stating that opinion, very often,

unless you go out and test it and all the rest,

you realise what you think is good might be terrible.

And in fact, what you think is good is probably good

for a subset of the population that is like you,

which is why you should be building personalization,

so that the subset of the organisation,

that you can identify the subset of the population

that is like you, and you can present it

to them in that way.

But it's way more important to think in that way,

to think actually we need an intelligent product.

We need something that adapts

to all these different viewpoints.

If you have a room of people disagreeing,

then that means that they sample different attitudes

and perspectives that you need to cater to.

And the only way you're going to do that

is by being very sort of intelligent centric.

So I'd say that's where the, probably the challenge is,

that people may say they want to be smart.

And they liked the idea of, yeah,

we're doing all the smart stuff with our product,

but I think people still fall back

into just doing the things they understand.

So that's where I think it takes a little bit of bravery,

probably because you can't be good at everything.

If you're a CEO, you've got an extremely wide remit

and you've got so many things to look after,

you can't possibly know all of this stuff.

And so I think it's developing faith in the right people

and saying, okay, show me what intelligence looks like

and putting enough resources behind that.

I think if there's a core message,

it's really taking that to heart, saying, you know what?

Over the next five, ten years,

we're going to have a smarter product.

It's not just that it looks better, it's not just,

it's going to have to be much smarter than everyone else.

And if you really internalise that,

then you'll end up putting some budget that way,

and you'll, you know, you'll do a lot of things wrong.

And the teams will, you'll get traditional teams

that just go and build a big warehouse.

It is a difficult journey, but you're going to have to

persist on that journey with this idea that the question

you should be asking yourself, is my product smarter?

Is that product smarter than the other guy?

How is it smarter?

If you keep asking that question,

then even if you don't know the field,

even if you're a CEO, who's got a huge organisation

and can't possibly be good at,

you know, understand every little bit of it,

but just asking that question over and over again,

you'll draw people in the right direction

and you'll funnel some of the resources in that direction

and you might succeed.

- That was our conversation with Robin Hayden.

Thanks very much to him for taking the time to talk to us.

That's all for this episode,

but we have many more fascinating interviews coming up.

Make sure you subscribe so you don't miss anything.

Thanks for listening, catch up next time.