We need to reform research evaluation because "It's not just which people are we excluding, which profiles are we excluding, which ideas are we excluding, but also whose problems are we not solving" says Dr Karen Stroobants.

Karen is a researcher, policy adviser and consultant on research policy and strategy, with a focus on research culture. Her research background and PhD is in chemistry. She now manages a portfolio of activities, combining roles as (part-time) lead policy advisor on research landscape & economy for the Royal Society of Chemistry in the UK and as freelance consultant, focusing most recently on contributing to the drafting of a European agreement on research assessment reform. 

This agreement on reforming research assessment from CoARA, the Coalition for Advancing Research Assessment, is the reason I wanted to speak with Karen. We cover: concerns of current research assessment approaches; the need for both top-down and bottom up buy in to create research culture change and what that change might look like; how we can navigate career choices while the system is still in transition; the importance of reflection for research culture change and embracing a diversity of people ideas and research problems; the core commitments of the CoARA agreement; and the move to more qualitative assessments at both individual and institutional and national levels. We finish with Karen reflecting on her own career choices driven by values and what is important. 

• [00:00:29] Episode introduction
• [00:02:39] Welcome & Introduction
• [00:04:39] Concerns around research system
• [00:13:05] Research culture change needs top-down and bottom-up buy-in
• [00:20:12] Negotiating choices while the system is undergoing transition
• [00:23:25] Importance of reflection for research culture change
• [00:30:21] Diversity of people, ideas and research problems
• [00:34:17] CoARA Agreement on reforming research assessment
• [00:40:04] Signing up to the CoARA agreement
• [00:50:10] Narrative CVs - for inividuals, organisations and national level
• [00:54:02] Other ways of brining a qualitative lens
• [00:56:15] Karen's career path - setting boundaries, choosing values

Related links:

CoARA: Coalition for Advancing Research Assessment https://coara.eu

Prof Frank Miedema, UMC Utrecht, https://www.umcutrecht.nl/en/research/researchers/miedema-frank-f#

Marie Sklodowska-Curie Actions MSCA https://marie-sklodowska-curie-action; San Francisco Declaration on Research Assessment s.ec.europa.eu

INORMS: International Network of Research Management Societies https://inorms.net

DORA: San Francisco Declaration on Research Assessment https://sfdora.org/dora-community-engagement-grants-supporting-academic-assessment-reform/

The Metric Tide: Review of metrics in research assessment https://www.ukri.org/publications/review-of-metrics-in-research-assessment-and-management/

Book:

Caroline Criado Perez, Invisible Women, Vintage Books, 2019

https://carolinecriadoperez.com/book/invisible-women/ 

Related podcasts:

Sarah Davies: Part 1 on mobility, precarity and notions of excellence https://www.changingacademiclife.com/blog/2023/cal81-sarah-davies-part-1 and part 2 on luck, disrupting excellence, and cultures of care https://www.changingacademiclife.com/blog/2023/cal82-sarah-davies-part-2

Tanita Casci and Elizabeth Adams on supporting, rewarding and celebrating a positive collegial research culture https://www.changingacademiclife.com/blog/2021/5/21/tanita-casci-elizabeth-adams

James Wilsden on metrics and responsible research evaluation https://www.changingacademiclife.com/blog/2023/cal77-james-wilsdon-replay

This podcast uses the following third-party services for analysis:

Chartable - https://chartable.com/privacy

Geri: 00:00:05

Welcome to Changing Academic Life.

Geri: 00:00:08

I'm Geraldine Fitzpatrick, and this is a podcast series where academics and

Geri: 00:00:12

others share their stories, provide ideas, and provoke discussions about what

Geri: 00:00:17

we can do individually and collectively to change academic life for the better.

Geri: 00:00:30

I'm really excited to kick off this new season with what I think is a

Geri: 00:00:34

really important conversation with Dr.

Geri: 00:00:36

Karen Stroobants.

Geri: 00:00:38

it's about reforming research assessment.

Geri: 00:00:42

Karen has a research background and PhD in chemistry.

Geri: 00:00:46

And has transitioned to be a researcher, policy, advisor, and consultant

Geri: 00:00:50

on research, policy and strategy.

Geri: 00:00:52

She now manages a portfolio of activities, combining roles as

Geri: 00:00:56

lead policy advisor for the Royal society of chemistry in the UK.

Geri: 00:01:00

And as a freelance consultant focusing most recently on contributing to the

Geri: 00:01:06

drafting of a European level agreement on research assessment reform.

Geri: 00:01:11

And this agreement on reforming research assessment from CoARA the coalition

Geri: 00:01:16

for advancing research assessment.

Geri: 00:01:18

That's the reason I wanted to speak with Karen.

Geri: 00:01:22

In our conversation, we cover a range of important topics.

Geri: 00:01:27

We talk about concerns of the current research assessment approaches.

Geri: 00:01:31

And the need for both top-down and bottom-up buy-in to create

Geri: 00:01:35

research, culture change, and what that change might look like.

Geri: 00:01:41

And she also gives an overview of the core commitments of the

Geri: 00:01:44

CoARA agreement, reflecting much of what we have talked about.

Geri: 00:01:48

And this is around more qualitative assessments at individual

Geri: 00:01:52

institutional and national levels.

Geri: 00:01:54

And also about enabling a greater diversity of people career profiles,

Geri: 00:01:58

research problems, and methodologies.

Geri: 00:02:01

We finished off with Karen reflecting on her own career choices and

Geri: 00:02:05

really interesting transitions.

Geri: 00:02:06

And how these continue to be driven by knowing what her values

Geri: 00:02:11

are and what's important to her.

Geri: 00:02:14

I just need to note here that we recorded this interview at the end of may.

Geri: 00:02:19

And as she says, the work on the CoARA agreement has likely

Geri: 00:02:22

progressed on a lot since then.

Geri: 00:02:24

Especially as the working groups were just starting to get moving.

Geri: 00:02:28

So I'd really encourage you to go to the CoARA website.

Geri: 00:02:31

And check out where it's at now.

Geri: 00:02:35

And to get involved yourself.

Geri: 00:02:40

Thank you, Karen, for joining me today.

Geri: 00:02:42

I really wanted to talk to you because I saw that you had a key role in

Geri: 00:02:47

developing an agreement on reforming research assessment that was put

Geri: 00:02:51

out by the EU in July last year.

Geri: 00:02:55

But by way of introduction, do you want to give us a brief intro to your background?

Karen: 00:03:00

Yeah, definitely.

Karen: 00:03:02

Thanks, and thanks for inviting me.

Karen: 00:03:04

So my name is Karen Stroobants, and thinking about where I'm coming

Karen: 00:03:08

from, I definitely still identify as a researcher and that's also my background.

Karen: 00:03:13

I have a PhD in chemistry and I also did a postdoc after that.

Karen: 00:03:19

But then I actually started to diverge away from the academic environment and

Karen: 00:03:25

I've been for the last years really involved in research innovation policy and

Karen: 00:03:31

I've taken on a few roles over the years, but at the moment I would say I have

Karen: 00:03:35

a portfolio where I have a few hats on.

Karen: 00:03:38

The first one of those is that I'm the lead policy advisor on research innovation

Karen: 00:03:43

for Royal Society of Chemistry, so that get some of my time in the week, but

Karen: 00:03:48

then I'm also filling some of my time as a consultant on anything related to, to

Karen: 00:03:54

research culture and research assessment.

Karen: 00:03:57

And then in voluntary capacity, I'm one of the vice chairs of the Coalition

Karen: 00:04:01

for Advancing Research Assessment.

Karen: 00:04:03

That's the initiative I think Geraldine, you wanted to talk about today a lot.

Karen: 00:04:07

And I'm also a governing board member of Euroscience.

Karen: 00:04:11

So I think that kind of captures briefly, where I come from

Karen: 00:04:14

and briefly, where I am now.

Geri: 00:04:15

Yeah.

Geri: 00:04:16

I want to come back if we have time at the end to just explore how you got

Geri: 00:04:20

to shift, transition to those roles.

Geri: 00:04:24

But to focus on the research culture and research policy issues.

Geri: 00:04:30

What are the big challenges do you see at the moment that makes this

Geri: 00:04:35

really critical that you're prepared to make this the focus of your work.

Karen: 00:04:39

Yeah, it's a good question.

Karen: 00:04:41

And it, of course, all interleaves and we can talk a bit about the

Karen: 00:04:45

kind of journey after, but I did already have concerns around how the

Karen: 00:04:50

research system is working early on.

Karen: 00:04:52

I think in my first, second year of my PhD, I started to be concerned around

Karen: 00:04:56

some of the things I was seeing around me.

Karen: 00:04:59

And I really had a feeling that there is so much potential in what research

Karen: 00:05:03

as an endeavour can do for humanity and the planet, but we are really not

Karen: 00:05:08

reaching that potential at the moment.

Karen: 00:05:10

And I started to observe a lot of different issues.

Karen: 00:05:14

And it's not until later, because one of my first roles in policy

Karen: 00:05:17

was working on the research culture program for the Royal Society.

Karen: 00:05:22

That I started to feel that all those things really hang together.

Karen: 00:05:25

There's a lot of different things that are causing problems, but somehow

Karen: 00:05:30

they are all related to each other.

Karen: 00:05:32

And I had a realization, and it was also one of the outcomes of that

Karen: 00:05:35

program at the time, that the biggest lever to not change one of them, but to

Karen: 00:05:40

influence all of them is really around the incentive system and how we assess.

Karen: 00:05:44

And that's really set me up to, to pursue a path where that is the

Karen: 00:05:48

main focus of what I try to do.

Karen: 00:05:50

So all my positions all the different hats that I'm wearing at the moment, they all

Karen: 00:05:55

center around the questions of research culture and specifically using the lever

Karen: 00:06:00

of assessment to solve some of them.

Geri: 00:06:03

So important and that's a theme that comes up again and again

Geri: 00:06:06

in discussions is the impact of the current incentives systems that we have.

Geri: 00:06:12

What are you seeing in the people or hearing about.

Geri: 00:06:15

From the people you're talking to, working with, or what your understanding

Geri: 00:06:20

is of the current research on this, about the current incentive system.

Karen: 00:06:26

Yeah, and there's a lot of ground to cover here.

Karen: 00:06:29

But I think, at the moment, I think there is really a common almost agreement.

Karen: 00:06:36

I mean, not everyone agrees, but I think there is momentum around

Karen: 00:06:40

a common understanding on the one hand from the research community.

Karen: 00:06:45

I think those who really do research on assessment are really in agreement

Karen: 00:06:49

that the way that especially journal based metrics and also article

Karen: 00:06:56

based metrics are used that it's really not always appropriate.

Karen: 00:07:00

And that there's a lot of kind of trickle down effects on not just

Karen: 00:07:05

researchers, but also the research.

Karen: 00:07:07

So I think both.

Karen: 00:07:08

But the research suffers that the scientific record suffers,

Karen: 00:07:12

but also the people behind it.

Karen: 00:07:13

We're not really getting the most out of those people.

Karen: 00:07:17

And I think something that I'm also hearing more and more is that there is

Karen: 00:07:21

this focus on outputs really kind of this very narrow end of the journey of

Karen: 00:07:27

it's not even the end it's supposedly somewhere in the middle you should

Karen: 00:07:30

be done building on those to achieve impact but there's this very strong

Karen: 00:07:34

focus on the outputs, with very little attention for the things that come

Karen: 00:07:39

after although impact is picking up.

Karen: 00:07:42

But also the things that come before, more importantly, the

Karen: 00:07:44

whole process of research.

Karen: 00:07:46

And I think also with moving outside academia, I started to realize

Karen: 00:07:51

how little focus there is on the outputs in other environments.

Karen: 00:07:55

There's much more focus on how do people behave?

Karen: 00:07:58

What are the skills they acquired?

Karen: 00:08:00

And these are things that I've just not visibly seen within an academic setting.

Karen: 00:08:05

And it's not something people talk to me about from that setting

Karen: 00:08:07

because they're not aware How important these things are outside.

Karen: 00:08:11

So I think there's on the one hand, the issues that people talk about, but

Karen: 00:08:15

there's also really importantly, certain things they don't talk about because

Karen: 00:08:19

most people who are in academia have been there, they've grown up there,

Karen: 00:08:23

they've been there their entire life.

Karen: 00:08:25

And there are certain things that I think would be interesting to

Karen: 00:08:28

them that they don't talk about because they don't know about it.

Karen: 00:08:31

So I think those are maybe the two sides to think about really.

Geri: 00:08:35

Yeah.

Geri: 00:08:36

And there's also that tension, isn't there, of Even if people might be

Geri: 00:08:41

aware when you're in the system and on this treadmill chasing these incentive

Geri: 00:08:46

based quantitative metrics you're moving, doing that sort of thing.

Geri: 00:08:50

And that there's space to actually stand back and reflect and make deliberate

Geri: 00:08:54

changes or to talk about this.

Geri: 00:08:56

It's a bit hard when there are ongoing evaluation cycles within

Geri: 00:09:00

our universities or within our grant funding programs and so on.

Karen: 00:09:06

Definitely.

Karen: 00:09:07

And I mean, it's a question we always get because lots of universities

Karen: 00:09:10

have started to not explore necessarily how to assess differently.

Karen: 00:09:15

But at least they've started to think about how do we offer more training?

Karen: 00:09:19

How do we offer training on all the kind of things people are struggling with?

Karen: 00:09:22

How do we offer mental health groups?

Karen: 00:09:25

How do we offer all the kind of things to solve some of these issues?

Karen: 00:09:28

And not to discredit any of these initiatives, I think they're important.

Karen: 00:09:33

But sometimes...

Karen: 00:09:35

what I'm seeing is, and it's also just coming from my own background

Karen: 00:09:38

as a postdoc, people sign up to these things, but then their experiment

Karen: 00:09:43

in the lab fails, and the priority is repeating the experiment.

Karen: 00:09:47

They will just not show up for these things because they need to publish

Karen: 00:09:51

that paper for their career progression.

Karen: 00:09:53

So it gets the absolute priority.

Karen: 00:09:55

And despite all the efforts.

Karen: 00:09:58

To have trainings, to have support events.

Karen: 00:10:01

It's just the thing that drops off the list most often.

Karen: 00:10:04

So I think unless we can embed those things in the incentive system and in

Karen: 00:10:08

what supports the career progression, it's very hard to convince people to

Karen: 00:10:12

make the time and space for these things.

Karen: 00:10:15

And I think that's one of my ambitions is.

Karen: 00:10:18

really to contribute to an assessment system that can allow people space

Karen: 00:10:23

for this without being punished in their career progression.

Geri: 00:10:27

And in fact, rewarded, if it's actually recognized in the incentive

Geri: 00:10:32

system , valued and rewarded.

Karen: 00:10:34

Exactly.

Karen: 00:10:35

Exactly.

Karen: 00:10:35

And we can see how in other sectors, this works, people are rewarded to develop

Karen: 00:10:40

certain skills that are not output driven.

Karen: 00:10:43

They're more around behavior.

Karen: 00:10:45

They're more around the way they do things.

Karen: 00:10:48

You know, there's really big questions around how to embed

Karen: 00:10:50

something that like that in a sector and in an environment where it's

Karen: 00:10:55

almost a big unknown at the moment.

Geri: 00:10:57

Yeah.

Geri: 00:10:59

I had a little smile to myself when you said about people, prioritizing, fixing

Geri: 00:11:03

up their broken experiment, then going to the course that they signed up for.

Geri: 00:11:07

Because I ran a course recently, it was around saying yes, no, around setting

Geri: 00:11:12

boundaries and, setting priorities.

Geri: 00:11:14

And I think we had.

Geri: 00:11:16

12 people signed up and three turned up on the day and these were all postdocs

Geri: 00:11:22

and for similar sort of reasons and it was just ironic that the course that

Geri: 00:11:26

could help them manage this in a longer term got reshuffled at the last minute

Geri: 00:11:33

because of other shorter term priorities.

Karen: 00:11:36

Yeah, and it's usually really not, it's not even that they don't want to be

Karen: 00:11:41

there, or it's really just this complete drive towards this one very narrow set of

Karen: 00:11:48

things that will help them move forward.

Geri: 00:11:50

Yeah.

Karen: 00:11:52

And I think we're putting them in a difficult position because many

Karen: 00:11:55

of them know they need these things.

Karen: 00:11:58

And we are putting them constantly in these dilemmas between what they know

Karen: 00:12:03

they need, but what helps their career progression, or what they know is best

Karen: 00:12:06

for science versus what is best for their career progression, because there's quite

Karen: 00:12:10

a big gap at the moment between the two.

Karen: 00:12:12

So we're constantly really...

Karen: 00:12:15

In a way, in an unfair way, posing dilemmas to people.

Karen: 00:12:18

And in the UK as well, there's lots of conversations now

Karen: 00:12:21

around integrity and misconduct.

Karen: 00:12:24

And I think, of course, everyone has a personal responsibility, but

Karen: 00:12:28

we're not making it easy for people.

Karen: 00:12:30

Because we're really setting this gap between what are the things to do?

Karen: 00:12:35

What is the space to take, to think, to develop?

Karen: 00:12:39

Versus the kind of speed we're expecting them to produce results with.

Karen: 00:12:42

The two are really incompatible.

Karen: 00:12:44

So we're really asking people to do something very difficult to constantly,

Karen: 00:12:48

almost on a daily basis, make choices between these two, between what's best

Karen: 00:12:52

for science, what's best for career progression, but also between what they

Karen: 00:12:56

need for their personal development versus what they need for career progression.

Karen: 00:13:00

So it's just not an easy position.

Karen: 00:13:02

I think we put researchers in at the moment.

Geri: 00:13:05

And what you also say there points to it's great that there is an

Geri: 00:13:11

increased awareness around mental health.

Geri: 00:13:13

And I'm definitely seeing that far more in the last couple of years.

Geri: 00:13:19

I think COVID has been helpful as well as bad in that respect, it's made

Geri: 00:13:24

it normal to discuss much more now.

Geri: 00:13:27

And I also see the value in individual initiatives and people

Geri: 00:13:32

taking courses and doing training, but they are useless strategies if

Geri: 00:13:38

we don't get the systemic change.

Geri: 00:13:40

Like you talked about the lever being the incentive systems and

Geri: 00:13:43

the whole research culture and just changing behaviors more generally.

Geri: 00:13:47

What's the training, what's the process for creating that sort of change?

Karen: 00:13:53

That's a good question.

Karen: 00:13:54

It's not an easy one to answer.

Karen: 00:13:56

And I think the reason why the kind of the initiatives more at the ends

Karen: 00:14:01

of the process, let's say, are so important is because the systemic

Karen: 00:14:05

ones are going to take time.

Karen: 00:14:07

I think we're on a good trajectory in terms of already gaining

Karen: 00:14:11

understanding that there are problems that really needs to be looked at.

Karen: 00:14:15

And I think the things that we will need to really get there is, on the

Karen: 00:14:20

one hand, senior management buy in at every organization and institution

Karen: 00:14:24

in the sector, because I think it's very important to have researchers on

Karen: 00:14:28

board, but I also feel we have many of them on board, especially early career.

Karen: 00:14:33

I think the barrier at the moment is more at management levels.

Karen: 00:14:37

We really need to get buy in because if we want to get systemic change, it will

Karen: 00:14:42

need to really be organizational change.

Karen: 00:14:44

And that will really meet the drive from people at the top.

Karen: 00:14:48

We have seen the very big examples where.

Karen: 00:14:51

Change has happened, for example, in the Netherlands, I think Frank Miedema

Karen: 00:14:55

is one of those people who in a senior position has really just changed so much

Karen: 00:15:00

in his institution and he will always say, how important it was to be at that

Karen: 00:15:04

level to be able to make that change.

Karen: 00:15:06

So I definitely think we need also the bottom up, but I think without

Karen: 00:15:10

the top down and without that senior management inputs, we just won't get

Karen: 00:15:14

there because it is a systemic issue.

Karen: 00:15:18

And I also think we will need to go further than what we're

Karen: 00:15:23

thinking about now because I think.

Karen: 00:15:25

Research assessment is a very important lever to think about, but there's also

Karen: 00:15:31

the issue around precarity of careers.

Karen: 00:15:33

There's the funding cycles, and there's the amount of evaluation that goes on.

Karen: 00:15:38

Researchers are probably the most evaluated professionals I

Karen: 00:15:42

would imagine, across sectors.

Karen: 00:15:45

There's every output is evaluated separately.

Karen: 00:15:49

They are evaluated by their institution, by funders to get funding.

Karen: 00:15:53

I do think with the current reform we'll be able to do a lot, but I do think we'll

Karen: 00:15:58

still bounce on some boundaries around the life cycle the funding length of

Karen: 00:16:04

funding schemes, those kinds of things.

Karen: 00:16:05

So I do think there will be a few other issues that we will need to start to think

Karen: 00:16:10

about and start to solve to, to really get to some of the achievements we want to

Karen: 00:16:14

get to in terms of changing the culture.

Geri: 00:16:16

Yeah, totally agree.

Geri: 00:16:20

I'm hearing as well there's in terms of the spectrum of

Geri: 00:16:25

where the changes are needed.

Geri: 00:16:27

So there's, what we can do as individuals to better skill

Geri: 00:16:30

or to cope in the day to day.

Geri: 00:16:32

There's what we could do as an organization in creating our

Geri: 00:16:37

evolving, changing different local research cultures and practices.

Geri: 00:16:42

And that's embedded in the bigger institutional policy level systems around

Geri: 00:16:47

government policies that set the funding programs or set the evaluation criteria.

Geri: 00:16:52

And that's complicated as well, isn't it?

Geri: 00:16:54

Because you've got national initiatives, priorities, and you've got International

Geri: 00:16:59

practices and disciplinary practices crossing over as well, and it's a

Geri: 00:17:05

really hard space to think about when you start unpacking all those layers.

Karen: 00:17:11

Definitely, and I mean, the real answer is that we will

Karen: 00:17:14

need change at all those levels.

Karen: 00:17:17

But also we'll need to ensure that those levels are communicating to each other.

Karen: 00:17:21

Because one of the issues we're seeing also when we're talking about the

Karen: 00:17:25

setup of the coalition is that there are certain levels, like for example,

Karen: 00:17:31

maybe institutional levels who want to move forward, but they're then

Karen: 00:17:34

bound by rules set by the level above that they just can't get around.

Karen: 00:17:41

So unless change happens elsewhere, they are actually kind of stuck in that space.

Karen: 00:17:47

So one of the things that I see as the kind of big things that the

Karen: 00:17:51

coalition will bring is a space to actually bring all these levels

Karen: 00:17:54

together and to enable discussions.

Karen: 00:17:58

So not to take away any autonomy, I think each organization

Karen: 00:18:02

should be able to think about.

Karen: 00:18:03

What it is they can do to progress.

Karen: 00:18:06

And of course, there is the agreement that sets a very common direction

Karen: 00:18:09

for everyone, but there's a lot of space within that to explore and to

Karen: 00:18:13

think about how to implement that.

Karen: 00:18:15

But at the same time, there needs to be constant exchange of information so that

Karen: 00:18:19

we Ensure we're not creating different things at different levels or in different

Karen: 00:18:25

organizations that contradict each other.

Karen: 00:18:28

So I think that will be really the value of the coalition to

Karen: 00:18:30

create that space for discussion.

Karen: 00:18:34

But you're right.

Karen: 00:18:34

It's immensely complex and, you know, sometimes I refer to it

Karen: 00:18:38

as one of those wicked problems.

Karen: 00:18:40

And we will need dialogue at a lot of different levels and, including

Karen: 00:18:44

also there's, of course, this big discussion around the quantitative

Karen: 00:18:49

versus qualitative approaches, where some say the qualitative is definitely

Karen: 00:18:55

the most important at individual level.

Karen: 00:18:57

But when we go to more aggregate levels, we need some of that

Karen: 00:19:00

quantitative for it to be manageable.

Karen: 00:19:03

And I think, in a sense, I think as humanity almost we've become obsessed with

Karen: 00:19:12

measuring things, not just in academia, I think, it's also well understood at the

Karen: 00:19:16

moment that GDP is not the best measure for success at a very different level.

Karen: 00:19:22

So I do think there are bigger questions, not just in academia,

Karen: 00:19:26

but in more general terms about how we've become as a society to define

Karen: 00:19:31

success and how that shapes behaviors.

Karen: 00:19:36

I sometimes hope that we can set an example in academia too, and in the

Karen: 00:19:41

research space more broadly because it's not just academia there's research

Karen: 00:19:45

going on in other sectors as well that we need to involve in those

Karen: 00:19:49

discussions, but we can hopefully really reflect on how it can be done

Karen: 00:19:55

in a way that we don't cause those bad behaviors and set those bad incentives.

Karen: 00:20:01

But it's not an easy challenge and it will need really buy in and thinking

Karen: 00:20:06

and support from all those different stakeholders that you describe.

Geri: 00:20:12

One of the tensions that I experienced just in a day to day sense is

Geri: 00:20:18

being passionate about the need for these sort of changes and having these more

Geri: 00:20:23

holistic measures and valuing research culture and all of the other things that

Geri: 00:20:28

we've talked about and recognizing that my current PhD students or postdocs are

Geri: 00:20:34

still going to be working or playing in the system as it is now while it's

Geri: 00:20:39

still undergoing this transition.

Geri: 00:20:42

Any hints or suggestions about how to navigate that and how to help them

Geri: 00:20:47

make choices that both start to play out the behaviors that we want to see,

Geri: 00:20:54

but not jeopardize their own careers.

Karen: 00:20:57

It's a difficult, it's very difficult.

Karen: 00:21:00

I think it's really about, as you say, it's finding a balance.

Karen: 00:21:03

It's exploring with them how much can I go in the right direction without

Karen: 00:21:08

losing out on any opportunities.

Karen: 00:21:11

And I think there's already many kind of ways to participate in things

Karen: 00:21:16

like open science in thinking about at the team level, how to talk about

Karen: 00:21:21

inclusion and diversity, how to incorporate some of these elements,

Karen: 00:21:26

but sometimes also being frank with someone in the team and say, look, in

Karen: 00:21:29

an ideal world, this is what I would do.

Karen: 00:21:32

But thinking about your career progression, you might need

Karen: 00:21:35

to compromise in X or Y ways.

Karen: 00:21:39

I think we need to be honest about that and be able to really

Karen: 00:21:42

have honest conversations.

Karen: 00:21:45

But I also think there are small things we can do to start really

Karen: 00:21:48

changing the narrative, like one example I often give is, I remember

Karen: 00:21:52

here in the lab in Cambridge when someone would say that they happily

Karen: 00:21:55

announced they've published a paper.

Karen: 00:21:58

No matter who asked the first question, it was always, which journal was it in?

Karen: 00:22:04

You know, that doesn't need to be the first question we can easily ask a

Karen: 00:22:07

different question about the content about maybe who it's important for.

Karen: 00:22:13

So I think, while we don't have the means to immediately change the big things,

Karen: 00:22:17

and there will need to be compromises.

Karen: 00:22:20

There are many examples like this where we can start to change the

Karen: 00:22:23

small things on a day to day basis.

Karen: 00:22:25

And I do think that those will also already have an impact also just on how

Karen: 00:22:29

those people grow up and are shaped as a professional, whether that's within

Karen: 00:22:33

academia or going to other sectors.

Karen: 00:22:36

So I think there's things to do around that.

Karen: 00:22:39

Also, can we, I remember my own professional development conversations in

Karen: 00:22:43

academia, they were well non existing or at least not during the postdoc where they

Karen: 00:22:49

were maybe most important for me because I was trying to kind of change track.

Karen: 00:22:54

They were really based on the research.

Karen: 00:22:56

They were just about the experiments.

Karen: 00:22:58

I think one thing PIs can do is start to really have development conversations

Karen: 00:23:04

that really talk about the things, the behaviors, the competencies,

Karen: 00:23:07

the things that are important.

Karen: 00:23:09

So I think there are definitely things we can start to do to signal that we're

Karen: 00:23:15

moving in a different direction without jeopardizing the career progression,

Karen: 00:23:20

but it's a balance and it's not going to always be an easy conversation.

Geri: 00:23:25

I like just the small things that we can start to do in what we talk about.

Geri: 00:23:30

And we've also just tried celebrating when people submit a paper, because that's

Geri: 00:23:34

really what they're totally in control of, and whether it gets accepted or not

Geri: 00:23:37

depends upon reviewer two, as we know.

Geri: 00:23:40

So yeah, also just trying to celebrate, getting something done.

Geri: 00:23:44

Having written it up.

Geri: 00:23:45

When you talk about behavior and competencies, this is getting to the nitty

Geri: 00:23:50

gritty of our everyday research cultures.

Geri: 00:23:53

What other sort of things would you be wanting to highlight there for us?

Karen: 00:23:59

I mean, and now I'm really speaking in personal capacity because

Karen: 00:24:02

this is an idea that I've developed over the last few years and I know

Karen: 00:24:05

there's others, especially in the research and research space who are keen

Karen: 00:24:09

to explore this further but it's not something that's been tested much yet.

Karen: 00:24:13

One of the approaches that I've seen used outside when I went into a different

Karen: 00:24:18

career direction is for example in interviews or in performance reviews,

Karen: 00:24:25

the use of kind of competency based approaches where people really ask

Karen: 00:24:29

you to reflect on how you do things.

Karen: 00:24:30

So ask questions about, giving an example of a really difficult management,

Karen: 00:24:35

people management situation and how you've handled that, whether there's

Karen: 00:24:38

anything you would be differently if, in that situation again.

Karen: 00:24:43

And I really would encourage to start to put at least a reflection behind

Karen: 00:24:48

that in academia, because one of the things, and now I'm kind of touching

Karen: 00:24:52

a bit maybe on the journey elements.

Karen: 00:24:54

One of the things that I really had to learn when I transitioned

Karen: 00:24:59

was to reflect on how I do things.

Karen: 00:25:01

I had never had to do that, and I was around, I don't know, I was

Karen: 00:25:06

probably almost nearing my early 30s.

Karen: 00:25:09

I had never really had to, in a conversation within my previous

Karen: 00:25:14

environments in different universities I was in, No one had really pushed me

Karen: 00:25:18

to reflect on how I did certain things on how I approached something and how I

Karen: 00:25:23

put a strategy in place for something.

Karen: 00:25:26

And this is one of the things I really had to learn, and it's so valuable

Karen: 00:25:29

to do that, to make time for that.

Karen: 00:25:32

So I would say that is, I think, one of the things that for me will be

Karen: 00:25:36

important to somehow embed in the sector.

Karen: 00:25:40

And I think assessment is a good way to do that because it means

Karen: 00:25:43

everyone will need to get involved.

Karen: 00:25:45

It's not optional.

Karen: 00:25:47

And it's not clear maybe yet how or where it would be fitting because

Karen: 00:25:51

it might not fit in every assessment process, but it's something that

Karen: 00:25:54

I think can bring a lot of value.

Karen: 00:25:57

To help people to really reflect on the way they do things.

Karen: 00:26:00

And it relates to a lot of culture themes, inclusivity open science,

Karen: 00:26:05

integrity, for all those areas.

Karen: 00:26:08

You could really think of questions where people need to think about have I been in

Karen: 00:26:12

a situation where I felt that my research or someone else's research, there was

Karen: 00:26:18

maybe an issue or there was a gray zone.

Karen: 00:26:21

And how did I deal with that?

Karen: 00:26:23

How would I do it differently next time?

Karen: 00:26:25

So it's really almost embedding some of those difficult

Karen: 00:26:28

conversations into what we're doing.

Karen: 00:26:32

I feel that's an approach that's not talked about so much now.

Karen: 00:26:35

It's more about how can we measure open science?

Karen: 00:26:38

Can we measure how many open science, how many open access

Karen: 00:26:41

publications or how many data sets?

Karen: 00:26:44

I would really encourage that.

Karen: 00:26:46

I'm not saying we shouldn't do any of that and especially at aggregate level

Karen: 00:26:49

to see where we're making progress, but on an individual level, I believe

Karen: 00:26:53

it will be more valuable to be able to have narratives and conversations

Karen: 00:26:59

around how to deal with some of this.

Karen: 00:27:01

And that can also come back a bit to the previous point, Geraldine,

Karen: 00:27:04

around, could you in a conversation talk with people who are now in that

Karen: 00:27:09

transition phase in their career?

Karen: 00:27:11

How do you balance the two?

Karen: 00:27:13

If you have to make a decision on a publication, Will you give more

Karen: 00:27:16

weight to publishing open access or to publishing in a certain journal?

Karen: 00:27:21

How do you deal with those kind of dilemmas, you know?

Geri: 00:27:25

So talking about what we need to talk about research and how we do it.

Geri: 00:27:30

And what You're saying also requires managers, leaders, senior people to be

Geri: 00:27:39

trained in how to ask good questions and how to engage in these conversations

Geri: 00:27:43

because, you said before about we haven't been socialized into this as

Geri: 00:27:47

a way of being and doing research.

Karen: 00:27:52

Yes, definitely.

Karen: 00:27:53

And the leadership aspect is very important there.

Karen: 00:27:56

I think this is where the moment we experience a lot of the barriers.

Karen: 00:28:03

Because we are now, we are in a system where leaders are really not

Karen: 00:28:06

accustomed to any of those things.

Karen: 00:28:09

And many of them also don't see a need for them.

Karen: 00:28:11

I really noticed that in conversations.

Karen: 00:28:12

They have a feeling that they have been used to doing something a certain way.

Karen: 00:28:15

And there's absolutely nothing wrong with doing it that way.

Karen: 00:28:19

So there's a bit of a.

Karen: 00:28:21

disconnect there.

Karen: 00:28:22

So there's, of course, a question around, can we already enthusiast,

Karen: 00:28:26

make enthusiast some of those coming into kind of, say, the next generation?

Karen: 00:28:32

Are we going to be able to convince those that are there currently?

Karen: 00:28:34

I'm honestly not so sure what the answer to that is, but also one of the things I'm

Karen: 00:28:39

really Sometimes breaking my head around is, how can we increase the mobility

Karen: 00:28:45

between sectors, because for me that would be I mean it's almost a chicken

Karen: 00:28:48

and egg situation because I think if we can have more focus on process competence

Karen: 00:28:52

skill, it will be less difficult for people from outside to come in because

Karen: 00:28:58

they'll have things to talk about.

Karen: 00:29:00

They don't have to just have a publication record.

Karen: 00:29:02

They can talk about all those other things in the same way.

Karen: 00:29:05

But at the same time, and then when you bring those in.

Karen: 00:29:09

That will also shift how things are done because you'll have people who bring

Karen: 00:29:12

experiences from other environments and who will be much more open even at

Karen: 00:29:16

that senior level to some, you know, they'll be more socialized with some of

Karen: 00:29:20

these approaches, but of course at the moment we have neither we don't have the

Karen: 00:29:25

instream of those people, and we don't really have the routes to bring them

Karen: 00:29:28

in because the assessment system is a barrier for them to move into those areas.

Karen: 00:29:33

You know, we need to get either the in Instream first or change the assessment

Karen: 00:29:37

system so we can bring them in because there's a lot of appetite to have more.

Karen: 00:29:42

Mobility, but I do feel the assessment is one of the barriers

Karen: 00:29:46

to be able to achieve that.

Karen: 00:29:48

And I do think it would bring a lot of richness to all sectors, if we could have

Karen: 00:29:54

more of a stream of people in and out and also put less pressure, because now a lot

Karen: 00:30:00

of people who leave academia, they still feel some kind of failure because they

Karen: 00:30:05

think there's no way to ever come back.

Karen: 00:30:07

Whereas if you would have a more dynamic system where people can go in

Karen: 00:30:10

and out, I think you would, to some extent, solve some of those issues.

Karen: 00:30:16

Because people wouldn't feel if I leave now, there's never a way

Karen: 00:30:19

to come back to that environment.

Geri: 00:30:21

Yeah.

Geri: 00:30:21

Absolutely.

Geri: 00:30:22

And also solve some of the big societal challenges that again you raised at the

Geri: 00:30:26

beginning about one of the motivators for why this is so critical to address.

Geri: 00:30:30

So is that an example of diversity and inclusion that you just talked about,

Geri: 00:30:35

like the diversity of backgrounds that we might be enabling in our processes.

Karen: 00:30:43

Definitely, because I do think at the moment with the narrowness of what

Karen: 00:30:47

we're assessing, there's really almost this kind of one type of profile that

Karen: 00:30:53

kind of makes it all the way to the end.

Karen: 00:30:54

I mean, I'm being a bit black and white now.

Karen: 00:30:58

It's probably a bit more gray than that.

Karen: 00:31:01

But I do think we're missing out on a lot of different talents.

Karen: 00:31:05

And then also the question is, what does that mean for the

Karen: 00:31:08

research we're missing out on?

Karen: 00:31:09

Because I really enjoyed the Invisible Women, the book by Caroline Criado

Karen: 00:31:14

Perez, and she gave this example.

Karen: 00:31:17

It's more on the innovation side about the first Apple health app, not including

Karen: 00:31:23

anything on the menstrual cycle because they didn't have a woman in the room.

Karen: 00:31:26

So no one thought about it.

Karen: 00:31:28

I mean, they missed out on something huge, absolutely huge.

Karen: 00:31:32

And then there's also the stories around how, you know, endometriosis

Karen: 00:31:36

as a disease has not received that much attention because it's just

Karen: 00:31:40

not a problem that those maybe in charge mostly male had recognized.

Karen: 00:31:46

So I do think and these are just examples on the gender side.

Karen: 00:31:48

I'm sure there's examples in all types of diversity direction.

Karen: 00:31:53

So I think It's not just which people are we excluding, which

Karen: 00:31:56

profiles are we excluding, which ideas are we excluding, but also

Karen: 00:32:00

whose problems are we not solving.

Karen: 00:32:02

So I think that's both diversity of people, ideas and also

Karen: 00:32:06

the research problems that we really need to think about.

Geri: 00:32:10

One of the things that gets talked about in the coalition agreement

Geri: 00:32:15

that we'll get to in a tick is also diversity of research methods and

Geri: 00:32:20

approaches and disciplinary perspectives and theoretical perspectives, which

Geri: 00:32:24

is often a dimension of diversity that doesn't always come to mind first.

Karen: 00:32:30

Yeah, and I think that's very important to also be appreciative of the

Karen: 00:32:34

methods that are used across sectors, so indeed bringing in maybe some of those

Karen: 00:32:39

people with experiences elsewhere, but also disciplines to recognize each other's

Karen: 00:32:44

methods, because one thing that's often coming up also in conversations around

Karen: 00:32:49

the quantitative versus qualitative is subjective, quantitative is objective,

Karen: 00:32:54

which I think anyone who's been industry a bit deeper involved in these

Karen: 00:32:57

conversations really knows it's not true.

Karen: 00:33:00

But to some extent, I also sometimes see some people in the harder sciences

Karen: 00:33:05

kind of almost discredit some of the methods used in the softer sciences And,

Karen: 00:33:10

often unknowingly because they often really don't know that much about those

Karen: 00:33:14

subject areas, and it's actually causing problems, not just for the methodologies

Karen: 00:33:18

used in assessment itself, I would say.

Karen: 00:33:22

So I do think that this diversity of recognizing methods that have

Karen: 00:33:27

been established, and, they've been established for many years,

Karen: 00:33:29

is important, not just for the scientific endeavor, but even just

Karen: 00:33:33

for progressing assessment itself.

Geri: 00:33:36

Yeah, because we know from the research that interdisciplinary research

Geri: 00:33:41

is harder to get funded because people often find it difficult to review from

Geri: 00:33:47

different methodological perspectives or disciplinary perspectives.

Karen: 00:33:54

Yes, and I think sometimes also don't have that much confidence maybe

Karen: 00:33:58

methods they're not so familiar with and I do think there's also an element of

Karen: 00:34:02

trust there in, you know, where there is recognized established methodologies.

Karen: 00:34:08

I think that also does need to be included and recognized, but it's a challenge

Karen: 00:34:13

interdisciplinary one definitely.

Geri: 00:34:17

So you talked about the coalition before.

Geri: 00:34:20

Do you want to explain what the coalition is and what work it's done?

Karen: 00:34:25

Yeah, definitely.

Karen: 00:34:26

In a few, I think it's now two years ago, there was a process that started with a

Karen: 00:34:31

consultation where there was a realization coming from a number of organizations

Karen: 00:34:37

who had been working on this for a while.

Karen: 00:34:40

That there are issues with assessment and the commission wanted to

Karen: 00:34:44

understand better the problem.

Karen: 00:34:46

The European Commission.

Karen: 00:34:47

Yeah, sorry.

Karen: 00:34:48

So the European Commission wanted to really better understand views of

Karen: 00:34:52

different stakeholders in the landscape.

Karen: 00:34:54

So they first started a consultation.

Karen: 00:34:57

And there was a report really with the conclusions of that, but also

Karen: 00:35:01

some principles to think about how to move forward and how to really start

Karen: 00:35:06

to change the way assessment works.

Karen: 00:35:09

And so this is where I came into the process.

Karen: 00:35:11

I was invited to be part of drafting an agreement and the agreement was

Karen: 00:35:17

building on the principles from that big consultation with lots of stakeholders.

Karen: 00:35:21

But the idea was really to put down on paper in very specific

Karen: 00:35:25

commitments, a common direction for the sector to move in.

Karen: 00:35:29

And alongside it also a time frame to say, okay, we'll do

Karen: 00:35:32

these things by a certain time.

Karen: 00:35:35

And this agreement was published in July 2022.

Karen: 00:35:39

And so the commitments, some of them, I mean, are very close to what

Karen: 00:35:43

we've already been talked about.

Karen: 00:35:44

The first one's really around recognizing diversity in contributions

Karen: 00:35:48

to research, but also all the different roles that people play.

Karen: 00:35:52

And then the other core commitments are really around.

Karen: 00:35:56

primarily using qualitative assessments, but also avoiding any inappropriate uses

Karen: 00:36:03

of metrics and the use of rankings in of institutions in research assessment.

Karen: 00:36:08

And then there's a range of kind of supporting commitments to enable

Karen: 00:36:11

more kind of the how we do it.

Karen: 00:36:12

So this is the, these four are like the what's.

Karen: 00:36:16

And the time frame put to it was for organizations to say, okay, I commit

Karen: 00:36:20

to this, to have an action plan with very concrete milestones after one

Karen: 00:36:24

year of signing, and to then also give a more elaborate progress, which is

Karen: 00:36:30

based on just self assessing themselves after about five years of signing.

Karen: 00:36:35

And so this agreement is really what this coalition is coming together around.

Karen: 00:36:39

So after the agreement was published, there was some time that was

Karen: 00:36:44

taken really to discuss how the coalition should function, which

Karen: 00:36:47

governance structure it should have.

Karen: 00:36:49

But so in December last year, there was a constitutive assembly bringing

Karen: 00:36:54

together all the organizations that want to be part of the coalition.

Karen: 00:36:59

And I have to check the exact numbers, but we're well over 450 members

Karen: 00:37:04

member organizations, because it's open to organizations at the moment.

Karen: 00:37:08

And so the mission of the coalition is really To achieve this systemic

Karen: 00:37:14

reform that we've already spoken about and the vision is really around the

Karen: 00:37:19

direction that the agreement set.

Karen: 00:37:21

So that's why I wanted to first touch on the agreement.

Karen: 00:37:24

And of course, this is just to start.

Karen: 00:37:26

We just really started end of last year, but at the moment we are forming

Karen: 00:37:32

working groups for all our members.

Karen: 00:37:34

Are able to be involved in proposals to set up working groups to really

Karen: 00:37:38

start to move forward the agenda and really kind of think about how can we

Karen: 00:37:42

achieve the commitments and how can we achieve the mission of the coalition.

Karen: 00:37:46

So a lot of the work is still coming is ahead of us.

Karen: 00:37:52

But we've been really encouraged by.

Karen: 00:37:55

the momentum, the enthusiasm, the amount of people who are really like, okay,

Karen: 00:37:58

we really need to, we really need to do something we've talked about this for

Karen: 00:38:02

a long time, but it's really time for action and for seeing change implemented.

Karen: 00:38:08

So I'm quite yeah, positive about what will come in the next few months.

Karen: 00:38:11

Once these working groups are approved and can really start the work of the coalition

Karen: 00:38:16

and can really start collaborations.

Karen: 00:38:18

with organizations in a lot of countries, not just in Europe, because the initiative

Karen: 00:38:23

now is really globally focused and we have quite a bit of work to do to achieve

Karen: 00:38:27

some of the international strategy that we're setting out at the moment.

Karen: 00:38:32

But we are starting to see also organizations from

Karen: 00:38:34

outside Europe joined slowly.

Geri: 00:38:37

That's very exciting that there's 450 so far signed up.

Geri: 00:38:42

If in signing up and putting in agreements, like you talk about

Geri: 00:38:47

organizational levels, and I'll come to that in a tick, is the EU itself?

Geri: 00:38:52

Looking at how to implement I'm holding up a copy of the paper that no one can

Geri: 00:38:56

see because we're going to be on audio.

Geri: 00:38:58

And is the EU itself also looking at the agreement to review say the funding

Geri: 00:39:05

criteria for the horizon 2020 or whatever upcoming funding programs they have.

Karen: 00:39:13

Yes, so they are a signatory the European Commission as a funder

Karen: 00:39:17

is a signatory of the agreement.

Karen: 00:39:19

And so they will, in the same way that everyone else will they will

Karen: 00:39:23

put in place an action plan to think about how they will implement the

Karen: 00:39:28

change, and they will release it at the same time as other signatories.

Karen: 00:39:33

And I've also been speaking for example specifically to The MSCA unit who are

Karen: 00:39:38

working on the Marie Sklodowska Curie fellowships, and they also are really

Karen: 00:39:43

asking questions about, okay, what do we need to change in our evaluation

Karen: 00:39:48

processes to be completely aligned?

Karen: 00:39:50

And I mean, they are already quite ahead, I think they already align with

Karen: 00:39:54

a lot what's in the commitments, but there's definitely also thinking across

Karen: 00:39:58

the European Commission around what their action plans need to look like

Karen: 00:40:02

to fully align with the initiative.

Geri: 00:40:04

So for people listening , what would they need to do to talk to or convince

Geri: 00:40:10

their organization to sign up to it?

Geri: 00:40:12

Do you have any suggestions or any good practices that could serve as role models

Geri: 00:40:16

for how to increase that membership?

Karen: 00:40:19

definitely.

Karen: 00:40:20

And obviously I'm a lot also in conversations where the audience

Karen: 00:40:23

are more individual researchers.

Karen: 00:40:26

So usually what I say is try to find at least already in your department

Karen: 00:40:31

who would be responsible for being involved in some of these conversations.

Karen: 00:40:35

And that's usually not too difficult.

Karen: 00:40:37

Sometimes finding the right person at the kind of central university level

Karen: 00:40:41

can be a bit more difficult, but there should be people on the department.

Karen: 00:40:44

departmental level who can at least point to the right direction.

Karen: 00:40:48

And then, yeah, really have the conversations also within your own team

Karen: 00:40:52

with your supervisor within your own department around why you think this

Karen: 00:40:57

is important, because we definitely see that where there's appetite from

Karen: 00:41:01

a lot of the kind of researchers.

Karen: 00:41:04

It does, it takes some time, but it does tend to really reach the right levels

Karen: 00:41:07

and at least cause some kind of discussion and initial conversations around whether

Karen: 00:41:13

it makes sense to sign up or not.

Karen: 00:41:16

Also, we have a secretariat, all the information is on the coara.

Karen: 00:41:20

eu website that are really responsive to any questions that people might have

Karen: 00:41:24

about what it means to become part what the, maybe the requirements are, but also

Karen: 00:41:31

the many opportunities that it presents, the benefits for the organization.

Karen: 00:41:36

So there's also definitely easy ways to gain information if people

Karen: 00:41:39

want to first inform themselves before having the conversations.

Karen: 00:41:44

But it's really partly trying to really find the people who have a

Karen: 00:41:48

bit of gravitas to to to put these kinds of things on the agenda of the

Karen: 00:41:52

conversations where they need to be held.

Karen: 00:41:55

And that can really just start at the departmental level usually.

Geri: 00:41:59

Yeah.

Geri: 00:42:02

And then as you said, there's all the work to say, what does this mean?

Geri: 00:42:07

How do we operationalize these principles and how do we put them into practice?

Karen: 00:42:12

Yes, definitely.

Karen: 00:42:13

And one thing that we found very important in putting in the commitment.

Karen: 00:42:16

So one of these supporting commitments that I very briefly touched on is to

Karen: 00:42:20

ensure that the resources are there.

Karen: 00:42:23

To actually then follow through and that doesn't necessarily mean

Karen: 00:42:27

financially can also be staff capacity.

Karen: 00:42:29

Maybe there's someone who's working on a related issue who could

Karen: 00:42:33

make space in their portfolio to really dedicate time to this.

Karen: 00:42:37

But just to indeed ensure that when an organization signs up, they have thought

Karen: 00:42:41

about how to accommodate the change and And of course, this will differ

Karen: 00:42:46

for different institutions, how much resources, but I think even if there

Karen: 00:42:50

is just already a bit of staff capacity to take care of this to follow the

Karen: 00:42:54

discussions, and to be in those exchanges.

Karen: 00:42:59

There can be differences in how active people participate there will be people

Karen: 00:43:02

who lead a working group but equally, there will be organizations that feel

Karen: 00:43:06

okay all we have capacity for is.

Karen: 00:43:08

To focus on our own action plan and to listen into some of the conversation.

Karen: 00:43:12

So we know we're going roughly in the same direction and that's absolutely fine.

Karen: 00:43:17

You know, there's space for all those that diversity within the coalition, but we

Karen: 00:43:22

do ask to have kind of a minimum resource allocation to ensure that the commitments

Karen: 00:43:28

can be really followed through.

Geri: 00:43:31

you talked about the four main principles of the agreement and one

Geri: 00:43:37

of them you talked about was, not relying on rankings, for example.

Geri: 00:43:43

. To me, that could be one of the hardest things, apart from the obvious sort of

Geri: 00:43:48

journal metrics up, but, the ranking system is just so toxic and people are

Geri: 00:43:55

happy to be critical of it when they're not ranked highly, but as soon as they're

Geri: 00:43:59

ranked highly, they do all the press releases, and I know that there are

Geri: 00:44:03

some institutions that have their tenure track or promotion agreements that require

Geri: 00:44:08

evaluators to assess them against other researchers at equally ranked institutions

Geri: 00:44:14

and all of this sort of language.

Geri: 00:44:16

I think that's going to be one of the hardest things to change.

Geri: 00:44:19

What's your opinion around the ranking system and how we can really shift that?

Karen: 00:44:25

I agree with you.

Karen: 00:44:26

It is the most difficult one and I'm going to be very honest.

Karen: 00:44:28

It's been the one that's been most kind of contentious and most discussed

Karen: 00:44:32

when we were drafting the agreement.

Karen: 00:44:34

And in the end, we very specifically worded it that research assessment

Karen: 00:44:41

of institutions should not be used in research assessment specifically

Karen: 00:44:46

because saying anything in more general terms was causing a lot of discussion.

Karen: 00:44:50

I'm really happy it's in there in that form, I'm going to be honest, because I

Karen: 00:44:54

do think it's important it's in there.

Karen: 00:44:56

But I think the key thing that was always coming back is it can trickle down.

Karen: 00:45:00

So some of the language that you're referring to, hopefully at

Karen: 00:45:04

least that will already decrease.

Karen: 00:45:06

So that's not seeing a complete ban of rankings in all the kind of uses

Karen: 00:45:10

they have in attracting students and so on, but at least start there.

Karen: 00:45:14

Of course, at the same time, there's really appetite to to really think about,

Karen: 00:45:17

do we really need this as the format for some of the other uses they have.

Karen: 00:45:22

And I really iNorms.

Karen: 00:45:24

I don't know whether you're familiar with iNorms more than our rank initiative.

Karen: 00:45:27

So they have provided this kind of alternative where universities can

Karen: 00:45:33

talk in a more narrative way about the things that they feel they're

Karen: 00:45:37

uniquely delivering to students.

Karen: 00:45:39

So I do think there's exploration also of how can we do things

Karen: 00:45:43

differently in that space.

Karen: 00:45:44

But as you say, These rankings are very ingrained in the system at the moment

Karen: 00:45:49

in how universities talk about their achievements, but also their strengths.

Karen: 00:45:55

And that will be something very difficult to to move away from.

Karen: 00:45:59

I do think that there will be work within the coalition to start to

Karen: 00:46:03

think about some of this, but it's one of, I think, as you rightly point

Karen: 00:46:07

out, one of the harder problems to think about how to move on that.

Karen: 00:46:14

But at the same time, I think there are things that can be done already in

Karen: 00:46:18

assessment specifically around really thinking about where is this language.

Karen: 00:46:23

Inappropriate and it will be an in most cases.

Karen: 00:46:26

And at the same time.

Karen: 00:46:27

Yeah, there are these initiatives really looking at the broader use of rankings in

Karen: 00:46:34

setting out what the strengths and the unique selling points of universities are.

Karen: 00:46:39

And interestingly, I was also in a conversation a while ago giving a

Karen: 00:46:43

workshop with all kind of younger universities who were more maybe focused

Karen: 00:46:48

also on, on kind of knowledge transfer.

Karen: 00:46:51

And we were, as part of a conversation on assessment, the starting points.

Karen: 00:46:55

that I often try to get people to think about is if you have to

Karen: 00:47:00

really think about the things you're proud of just as an institution what

Karen: 00:47:04

are the things that come to mind?

Karen: 00:47:06

And one of the things that came out of this specific conversation was

Karen: 00:47:10

regional development, attracting people who maybe We're first generation

Karen: 00:47:14

students really changing the really making a local impact in that sense.

Karen: 00:47:21

And then the conversation went on, , is this something that's recognized that

Karen: 00:47:24

you feel is recognized that's taken into account, for example, in these rankings.

Karen: 00:47:28

And, you know, it was a bit of an eye opener because All these institutions are

Karen: 00:47:33

the kind of the representatives of these institutions were realizing, these are

Karen: 00:47:37

not really things, the things we're most proud about are not things we're really

Karen: 00:47:42

rewarded or recognized for at the moment.

Karen: 00:47:44

So it did give an incentive to think about why changes is so important.

Karen: 00:47:50

And so I do think that.

Karen: 00:47:51

We can have the kind of conversations that make it very clear why these

Karen: 00:47:56

rankings are not probably the right way of bringing out all the diverse

Karen: 00:48:00

contributions that institutions make.

Karen: 00:48:02

So now we're looking at a different level.

Karen: 00:48:05

But how to practically move away from them.

Karen: 00:48:07

I think we're at the start of that conversation rather than at the end

Karen: 00:48:10

where we already see clear solutions.

Geri: 00:48:12

But it's still so encouraging that the conversation is starting, and

Geri: 00:48:17

I see that the coalition agreement builds on things like DORA and, The

Geri: 00:48:22

Metric Tide work in the UK and so on.

Geri: 00:48:24

And I'm also seeing things at the moment.

Geri: 00:48:26

I know that in Austria, there's the funding that the government department

Geri: 00:48:31

is starting to have discussions about.

Geri: 00:48:34

Revising renewing research evaluation.

Geri: 00:48:36

I know that Australia is currently running a survey asking for input on this.

Geri: 00:48:42

So , it is an encouraging time.

Geri: 00:48:45

And, yeah, will take time.

Karen: 00:48:48

Yeah, definitely.

Karen: 00:48:49

And I'm just also thinking when I had the first kind of experiences in

Karen: 00:48:54

advocating for this 10 years ago during my PhD, I was involved in this action

Karen: 00:48:59

group in Flanders where we're trying to talk to all the different political

Karen: 00:49:05

parties in Flanders about changing specifically the key that was used to

Karen: 00:49:09

allocate money between the different universities in Flanders, because there

Karen: 00:49:14

as well, they were using certain metrics that were then trickling down to the

Karen: 00:49:19

assessment of individual researchers.

Karen: 00:49:22

And it was an interesting experience because I was quite young and it

Karen: 00:49:25

was something I hadn't done before.

Karen: 00:49:27

I learned a lot from it, but we really hit a wall.

Karen: 00:49:30

It was very clear that there were a few people who are taking the decisions

Karen: 00:49:33

that we weren't really getting anywhere.

Karen: 00:49:35

So if I compare that experience with where we are now 10 years later, it is

Karen: 00:49:39

encouraging you know there's a lot of work ahead of us and there's many issues to

Karen: 00:49:43

still be resolved, but at the same time.

Karen: 00:49:46

I think those who have been consistent in working in the space

Karen: 00:49:50

for a number of years can also see how much progress has been made.

Karen: 00:49:53

Just even in terms of the number of people who are talking about this, the

Karen: 00:49:57

number of organizations who actually are recognizing that there is an issue,

Karen: 00:50:01

we're in a very different place already.

Karen: 00:50:02

So I think if we can make the same amount of progress or ideally even

Karen: 00:50:06

more in the next 10 years, you know, I'm quite optimistic about that.

Geri: 00:50:10

Yes.

Geri: 00:50:12

I really am, because it has to change.

Geri: 00:50:14

It's not sustainable the way it is at the moment.

Geri: 00:50:18

I'm just going back to something you said before, I know that there's a

Geri: 00:50:22

whole shift to narrative CVs as part of the emphasis on quality and people

Geri: 00:50:26

being able to argue their own value and I know that you were developing one

Geri: 00:50:30

of the first templates for that when you were working at the Royal Society.

Geri: 00:50:34

But I had never thought about it at an institutional level.

Geri: 00:50:36

You know, like an institution can develop.

Geri: 00:50:38

Its narrative cv, which is interesting.

Geri: 00:50:43

And even country level, country levels too.

Karen: 00:50:46

Exactly.

Karen: 00:50:47

I think we don't need to pour down everything in numbers.

Karen: 00:50:49

I mean, of course, there are, I'm often conversations about where they have

Karen: 00:50:54

value and I understand that there are situations where they have value, but

Karen: 00:50:58

I do think there's a lot we can do with narratives and with giving people

Karen: 00:51:02

space to focus on the things that they find important because the numbers

Karen: 00:51:06

often narrows down to a few things that, have organically been chosen or

Karen: 00:51:10

historically been chosen to be important.

Karen: 00:51:13

And they exclude a lot of other things.

Karen: 00:51:15

Indeed the more than our rank is an institutional level where there is space

Karen: 00:51:20

for for organizations to really think about what it is they bring us strengths.

Karen: 00:51:25

And I think we can do that at national level as well, too.

Karen: 00:51:28

And I think there are some countries that are very in regards.

Karen: 00:51:31

What are their research strength?

Karen: 00:51:32

So I definitely see that we can do that at different levels.

Karen: 00:51:37

But at the same time, it is probably the most important at the individual

Karen: 00:51:42

level, because there we are really dealing with people in different

Karen: 00:51:46

circumstances and contexts and not giving space to explain some of that I think

Karen: 00:51:50

is really important to me suppressing diversity, suppressing inclusivity.

Karen: 00:51:55

So I think we can do it at all levels, but for me, it is the most

Karen: 00:51:58

important that we, that is the primary method at the individual level.

Geri: 00:52:02

And that really needs breaking of the superstar researcher

Geri: 00:52:07

and that competitive culture.

Geri: 00:52:09

I also recognize, that if you've got 100 applications for a position and

Geri: 00:52:16

you only want to send 10 applications out for review and you've got to do

Geri: 00:52:20

that sifting process, it may not, I don't, I think there's still lots of

Geri: 00:52:24

tensions we need to work out about how do we bring a qualitative eye to doing

Geri: 00:52:30

this larger scale stuff and how do people still argue for their own quality

Geri: 00:52:36

when they're so used to the metrics.

Geri: 00:52:38

But I feel like there's, there'll be progress and we'll work it out.

Karen: 00:52:43

Yeah, and I think, there's also, there's been also conversations

Karen: 00:52:47

I've been where someone says, okay, we need the metrics because we can peer

Karen: 00:52:50

review every article of that person, but those are not the only two options.

Karen: 00:52:55

There are lighter touch qualitative elements you can look at that don't

Karen: 00:52:59

require you to return research publications back to front.

Karen: 00:53:04

So I think we haven't explored all the options yet, frankly.

Karen: 00:53:08

And I think there's also a question about how often we evaluate, because there's

Karen: 00:53:12

qualitative assessment, all good and but it takes a lot more resource and time.

Karen: 00:53:18

But then my question is, do we need to evaluate as much as we do?

Karen: 00:53:21

Could we evaluate less, but do it better, you know?

Karen: 00:53:24

So I think there are a few aspects that we still need to really think about.

Karen: 00:53:28

And sometimes there's a lot of focus on one element, like the

Karen: 00:53:31

narrative CV is a great instrument.

Karen: 00:53:34

And I am very enthusiastic about it, but it's not the kind of golden bullet.

Karen: 00:53:38

We need to look at the entire process and ensure that it's embedded in a

Karen: 00:53:42

process where there's changes that other stages and in different ways as well.

Karen: 00:53:47

So I think there is a lot of opportunities still to look at.

Karen: 00:53:50

Other parts of the process to explore methods that we just aren't used to

Karen: 00:53:54

yet and haven't used yet, but also to really think about frequency and

Karen: 00:53:59

whether we need to evaluate so often.

Geri: 00:54:02

Do you have a, just a brief sort of one or two examples of other

Geri: 00:54:06

methods or other light touch ways of bringing that sort of qualitative lens?

Karen: 00:54:11

Yeah, I mean, and here I'm drawing again a bit more on

Karen: 00:54:14

my experience in other sectors.

Karen: 00:54:17

With what I often also see with academic positions is the role

Karen: 00:54:22

descriptions are very broad.

Karen: 00:54:24

People feel like they need to send in a 10 page CV because.

Karen: 00:54:28

they need to cover all the grounds.

Karen: 00:54:30

Whereas in other sectors, Descriptions are often much narrower.

Karen: 00:54:35

And I think there is a question also around do academics need to

Karen: 00:54:38

do everything because they start to have a broader and broader portfolio.

Karen: 00:54:43

Or would it be beneficial to find a compromise between thinking, for example,

Karen: 00:54:48

about institutional and then departmental strategy and thinking about, okay, we

Karen: 00:54:52

have this interdisciplinary team, but we really need someone who focuses more on

Karen: 00:54:55

the outreach element or policy engagement.

Karen: 00:54:58

Can we.

Karen: 00:54:59

Hire someone who has that in their profile and balance that still with

Karen: 00:55:03

ensuring there's independence for people to develop in the ways they want to.

Karen: 00:55:06

So it could be that we don't go, of course, all the way to what other sectors

Karen: 00:55:10

do, but there's maybe a profile that gives some flexibility, but also gives

Karen: 00:55:15

a few more specific requirements as to what we're looking for in a person.

Karen: 00:55:19

And then you can really start to write very short paragraphs These are the

Karen: 00:55:25

things that I've done that's address the exact requirements you're looking for.

Karen: 00:55:30

If you're looking for a person with this type of more specific profile, you can

Karen: 00:55:35

explain usually in quite a concise way how you fit that without having to give

Karen: 00:55:40

the whole list of all your experiences that you have over your career.

Karen: 00:55:45

So that's one example where you would need a bit of, see a bit of change

Karen: 00:55:49

in the way that jobs are positioned.

Karen: 00:55:52

You'd need a bit more work on from institutions and departments to

Karen: 00:55:56

set out what are the skills they need to complement the teams that

Karen: 00:56:00

they are trying to build, and the achievements the mission, the

Karen: 00:56:03

vision they're trying to achieve.

Karen: 00:56:05

But that then would make it easier.

Karen: 00:56:07

To get to much more concise formats, which would need much less time to review.

Karen: 00:56:12

So that's just one example.

Geri: 00:56:15

That's great.

Geri: 00:56:16

I think we probably need to head to wrapping up to let you go and I had

Geri: 00:56:21

said I wanted to talk a little bit about your journey and you've already

Geri: 00:56:24

reflected on that, at different points.

Geri: 00:56:27

One thing I do want to reflect back from what you've said is any of us,

Geri: 00:56:34

like what you've illustrated is any of us at any stage of our career can

Geri: 00:56:40

be part of changing the conversation of, Arguing for change because you

Geri: 00:56:45

talked about in the Flanders example, even, 10 years ago, you were still a

Geri: 00:56:50

PhD student, yet you were there, in the mix of it, making these arguments.

Karen: 00:56:57

Yes.

Karen: 00:56:57

And I think it's partly also about setting priorities because I saw, I don't know

Karen: 00:57:05

why, maybe it's, maybe I have to thank my parents, but I had the confidence to do

Karen: 00:57:10

other things, so I was not only focusing on my research, and that's been really

Karen: 00:57:14

a red, thread throughout my career.

Karen: 00:57:16

I've always had the confidence to make space for the things that I found

Karen: 00:57:19

important and I must say it didn't, I didn't really see any suffering

Karen: 00:57:25

in my career as a researcher, as a consequence of it, so I do think it is

Karen: 00:57:29

possible But it was also because I was able to set boundaries very easily.

Karen: 00:57:35

So I think it is possible.

Karen: 00:57:38

But it also really depends on what you prioritize and what's important to you.

Karen: 00:57:42

And I think the key thing there is to understand what your values are.

Karen: 00:57:47

And the thing I've tried to do is really Live and work in line with those

Karen: 00:57:53

values and it's driven a lot of the choices that I've made in my career in

Karen: 00:57:57

the volunteering that I've done in the things I've dedicated my time to and

Karen: 00:58:03

for me it's worked out because it's got me to a point where I'm really enjoying

Karen: 00:58:07

all the things I'm doing but That is something I think, in a way where I've

Karen: 00:58:15

been lucky because that was instilled in me and it's not something everyone

Karen: 00:58:19

has an easy time with, I think, setting boundaries, saying no, choosing for your

Karen: 00:58:24

values is not something that's easy and for some people will require support and

Karen: 00:58:30

training that's maybe not always there.

Karen: 00:58:32

So I do recognize that's a challenge in itself.

Karen: 00:58:35

But I do feel that's what's been helping me to do some of these things.

Karen: 00:58:41

And as you see we all have a voice and I did refer to this

Karen: 00:58:46

early example where I used it.

Karen: 00:58:48

It wasn't very effective, but I learned from it and, it helped my

Karen: 00:58:51

personal development and I do feel it's contributed to where I am now.

Karen: 00:58:56

So I think we also sometimes need to recognize that we're Things haven't

Karen: 00:58:59

quite worked out in that moment.

Karen: 00:59:01

They do still contribute to to, to our learnings and to who we become as a person

Karen: 00:59:06

and what we can maybe achieve later on.

Geri: 00:59:09

That's really brilliant, Karen, I think that's just lovely.

Geri: 00:59:13

Are there any final things that you would want to add or say before we wrap?

Karen: 00:59:20

Yeah, I think maybe just one thing to still refer to is,

Karen: 00:59:23

you know, I knew I was gonna change direction quite early in my postdoc.

Karen: 00:59:27

So I started my postdoc in Cambridge in 2015.

Karen: 00:59:32

And I think in the first few weeks, I already was talking to the career service

Karen: 00:59:36

because I just felt at that moment, I want to finish this postdoc, but that's

Karen: 00:59:40

going to be, there's going to be a turning point that I'm going to need

Karen: 00:59:44

to do something where I'm just working with people more and with samples less.

Karen: 00:59:47

That was one of the motivations.

Geri: 00:59:49

So people is one of your core values.

Karen: 00:59:52

Yes, definitely.

Karen: 00:59:54

And even knowing that, so I transitioned in the end in 2018.

Karen: 00:59:58

So three years later.

Karen: 00:59:59

It was still so hard.

Karen: 01:00:01

I still felt that failure that I talked about earlier of leaving, although I had

Karen: 01:00:06

made that decision three years earlier.

Karen: 01:00:08

So you have to imagine the stigma around it to still feel that way at that point.

Karen: 01:00:12

But what I did realize is that a lot of the things that I thought that I would

Karen: 01:00:16

miss from the academic environment, the freedom, the independence, the

Karen: 01:00:21

kind of the space for thinking.

Karen: 01:00:24

Those aren't things I've lost.

Karen: 01:00:25

I still have been able to build those in my career.

Karen: 01:00:28

And I've also changed a lot in terms of kind of roles and jobs in the last few

Karen: 01:00:35

years, and I have realized that decision shouldn't have been so difficult because,

Karen: 01:00:41

it's really the world is changing.

Karen: 01:00:43

People are trying different things , in my sector, people move around a lot.

Karen: 01:00:48

And, one decision is really not setting your career direction

Karen: 01:00:51

for the rest of your life.

Karen: 01:00:53

So I would encourage people to really think about what it is they value and

Karen: 01:00:57

whether that's what they're getting at the moment in their work environment.

Karen: 01:01:00

And if it's not, maybe it's time for a change.

Karen: 01:01:03

And although we talked about how difficult it is at the moment to have that to sector

Karen: 01:01:06

mobility, I don't think it's impossible to go back to the academic sector.

Karen: 01:01:11

I've already had questions actually from people in the research on research

Karen: 01:01:14

space, whether I would, what I would ever considering going into that space.

Karen: 01:01:19

So it would be a different discipline.

Karen: 01:01:21

But it's all becoming more dynamic.

Karen: 01:01:23

And I think people should really think about whether they're getting out of

Karen: 01:01:28

their work and their work environment, what is important to them, and if

Karen: 01:01:32

not, to not be too afraid of change, because change can be a good thing.

Karen: 01:01:37

And I think there's a lot of opportunity to move around, especially now.

Geri: 01:01:44

That's great.

Geri: 01:01:46

Karen, I thank you so much for your contributions today and the

Geri: 01:01:51

really interesting discussions and I will provide links on the web

Geri: 01:01:55

page to the many of the names and papers and that you've referenced.

Geri: 01:02:01

Thank you for your work in being part of making that change happen and glad that

Geri: 01:02:06

you did make the decision to leave and shift directions to contribute to this.

Geri: 01:02:12

And hopefully this will encourage lots of other people to get involved as well.

Geri: 01:02:17

Whether it's in the small conversations in their workplaces or being part of trying

Geri: 01:02:22

to push that the agreement or whatever.

Geri: 01:02:25

So thank you very much for your time.

Karen: 01:02:28

Thanks for inviting me.

Karen: 01:02:29

And of course, I also hope many people will get involved in all the

Karen: 01:02:32

different ways, because we really need at all those different levels

Karen: 01:02:36

from the small conversations to the senior management team signing up.

Karen: 01:02:40

So yeah, hopefully we'll get a lot more engagements over

Karen: 01:02:43

the coming months and years.

Karen: 01:02:45

Thanks very much for inviting me.

Karen: 01:02:48

So glad to have been able to bring you this conversation with Karen

Karen: 01:02:51

about such an important topic.

Karen: 01:02:54

And I'd repeat the call for each of us to be part of making this change happen.

Karen: 01:02:59

From the small conversations we can have every day.

Karen: 01:03:02

To encouraging our senior management and organizations to sign up to the agreement.

Karen: 01:03:07

And then there's the work of how to put the agreement to

Karen: 01:03:10

work in a practical sense.

Karen: 01:03:13

And also just remind you that as this conversation was recorded

Karen: 01:03:17

in May, you may want to make sure to check out the CoARA website.

Karen: 01:03:20

coara.eu.

Karen: 01:03:24

To see where it's at now.

Karen: 01:03:27

And if you liked this conversation, you might also be interested in

Karen: 01:03:29

listening to other changing academic life conversations on similar themes.

Karen: 01:03:33

And I'm thinking here of the more recent conversations with Sarah Davies around

Karen: 01:03:38

research researchers, research cultures.

Karen: 01:03:41

With Tanita Casci and Elizabeth Adams from Glasgow uni around.

Karen: 01:03:46

Including collegiality as part of research evaluation and the conversation

Karen: 01:03:50

with James Wilsden on metrics and responsible research evaluation.

Geri: 01:03:59

You can find the summary notes, a transcript and related

Geri: 01:04:02

links for this podcast on www.

Geri: 01:04:05

changingacademiclife.

Geri: 01:04:07

com.

Geri: 01:04:08

You can also subscribe to Changing Academic Life on iTunes,

Geri: 01:04:13

Spotify and Google Podcasts.

Geri: 01:04:14

And you can follow Change Acad Life on Twitter.

Geri: 01:04:19

And I'm really hoping that we can widen the conversation about how

Geri: 01:04:22

we can do academia differently.

Geri: 01:04:24

And you can contribute to this by rating the podcast and also giving feedback.

Geri: 01:04:29

And if something connected with you, please consider sharing this

Geri: 01:04:32

podcast with your colleagues.

Geri: 01:04:35

Together, we can make change happen.