We need to reform research evaluation because "It's not just which people are we excluding, which profiles are we excluding, which ideas are we excluding, but also whose problems are we not solving" says Dr Karen Stroobants.

Karen is a researcher, policy adviser and consultant on research policy and strategy, with a focus on research culture. Her research background and PhD is in chemistry. She now manages a portfolio of activities, combining roles as (part-time) lead policy advisor on research landscape & economy for the Royal Society of Chemistry in the UK and as freelance consultant, focusing most recently on contributing to the drafting of a European agreement on research assessment reform. 

This agreement on reforming research assessment from CoARA, the Coalition for Advancing Research Assessment, is the reason I wanted to speak with Karen. We cover: concerns of current research assessment approaches; the need for both top-down and bottom up buy in to create research culture change and what that change might look like; how we can navigate career choices while the system is still in transition; the importance of reflection for research culture change and embracing a diversity of people ideas and research problems; the core commitments of the CoARA agreement; and the move to more qualitative assessments at both individual and institutional and national levels. We finish with Karen reflecting on her own career choices driven by values and what is important. 

• [00:00:29] Episode introduction
• [00:02:39] Welcome & Introduction
• [00:04:39] Concerns around research system
• [00:13:05] Research culture change needs top-down and bottom-up buy-in
• [00:20:12] Negotiating choices while the system is undergoing transition
• [00:23:25] Importance of reflection for research culture change
• [00:30:21] Diversity of people, ideas and research problems
• [00:34:17] CoARA Agreement on reforming research assessment
• [00:40:04] Signing up to the CoARA agreement
• [00:50:10] Narrative CVs - for inividuals, organisations and national level
• [00:54:02] Other ways of brining a qualitative lens
• [00:56:15] Karen's career path - setting boundaries, choosing values

Related links:

CoARA: Coalition for Advancing Research Assessment https://coara.eu

Prof Frank Miedema, UMC Utrecht, https://www.umcutrecht.nl/en/research/researchers/miedema-frank-f#

Marie Sklodowska-Curie Actions MSCA https://marie-sklodowska-curie-action; San Francisco Declaration on Research Assessment s.ec.europa.eu

INORMS: International Network of Research Management Societies https://inorms.net

DORA: San Francisco Declaration on Research Assessment https://sfdora.org/dora-community-engagement-grants-supporting-academic-assessment-reform/

The Metric Tide: Review of metrics in research assessment https://www.ukri.org/publications/review-of-metrics-in-research-assessment-and-management/

Book:

Caroline Criado Perez, Invisible Women, Vintage Books, 2019

https://carolinecriadoperez.com/book/invisible-women/ 

Related podcasts:

Sarah Davies: Part 1 on mobility, precarity and notions of excellence https://www.changingacademiclife.com/blog/2023/cal81-sarah-davies-part-1 and part 2 on luck, disrupting excellence, and cultures of care https://www.changingacademiclife.com/blog/2023/cal82-sarah-davies-part-2

Tanita Casci and Elizabeth Adams on supporting, rewarding and celebrating a positive collegial research culture https://www.changingacademiclife.com/blog/2021/5/21/tanita-casci-elizabeth-adams

James Wilsden on metrics and responsible research evaluation https://www.changingacademiclife.com/blog/2023/cal77-james-wilsdon-replay

Welcome to Changing Academic Life.

I'm Geraldine Fitzpatrick, and this is a podcast series where academics and

others share their stories, provide ideas, and provoke discussions about what

we can do individually and collectively to change academic life for the better.

I'm really excited to kick off this new season with what I think is a

really important conversation with Dr.

Karen Stroobants.

it's about reforming research assessment.

Karen has a research background and PhD in chemistry.

And has transitioned to be a researcher, policy, advisor, and consultant

on research, policy and strategy.

She now manages a portfolio of activities, combining roles as

lead policy advisor for the Royal society of chemistry in the UK.

And as a freelance consultant focusing most recently on contributing to the

drafting of a European level agreement on research assessment reform.

And this agreement on reforming research assessment from CoARA the coalition

for advancing research assessment.

That's the reason I wanted to speak with Karen.

In our conversation, we cover a range of important topics.

We talk about concerns of the current research assessment approaches.

And the need for both top-down and bottom-up buy-in to create

research, culture change, and what that change might look like.

And she also gives an overview of the core commitments of the

CoARA agreement, reflecting much of what we have talked about.

And this is around more qualitative assessments at individual

institutional and national levels.

And also about enabling a greater diversity of people career profiles,

research problems, and methodologies.

We finished off with Karen reflecting on her own career choices and

really interesting transitions.

And how these continue to be driven by knowing what her values

are and what's important to her.

I just need to note here that we recorded this interview at the end of may.

And as she says, the work on the CoARA agreement has likely

progressed on a lot since then.

Especially as the working groups were just starting to get moving.

So I'd really encourage you to go to the CoARA website.

And check out where it's at now.

And to get involved yourself.

Thank you, Karen, for joining me today.

I really wanted to talk to you because I saw that you had a key role in

developing an agreement on reforming research assessment that was put

out by the EU in July last year.

But by way of introduction, do you want to give us a brief intro to your background?

Yeah, definitely.

Thanks, and thanks for inviting me.

So my name is Karen Stroobants, and thinking about where I'm coming

from, I definitely still identify as a researcher and that's also my background.

I have a PhD in chemistry and I also did a postdoc after that.

But then I actually started to diverge away from the academic environment and

I've been for the last years really involved in research innovation policy and

I've taken on a few roles over the years, but at the moment I would say I have

a portfolio where I have a few hats on.

The first one of those is that I'm the lead policy advisor on research innovation

for Royal Society of Chemistry, so that get some of my time in the week, but

then I'm also filling some of my time as a consultant on anything related to, to

research culture and research assessment.

And then in voluntary capacity, I'm one of the vice chairs of the Coalition

for Advancing Research Assessment.

That's the initiative I think Geraldine, you wanted to talk about today a lot.

And I'm also a governing board member of Euroscience.

So I think that kind of captures briefly, where I come from

and briefly, where I am now.

Yeah.

I want to come back if we have time at the end to just explore how you got

to shift, transition to those roles.

But to focus on the research culture and research policy issues.

What are the big challenges do you see at the moment that makes this

really critical that you're prepared to make this the focus of your work.

Yeah, it's a good question.

And it, of course, all interleaves and we can talk a bit about the

kind of journey after, but I did already have concerns around how the

research system is working early on.

I think in my first, second year of my PhD, I started to be concerned around

some of the things I was seeing around me.

And I really had a feeling that there is so much potential in what research

as an endeavour can do for humanity and the planet, but we are really not

reaching that potential at the moment.

And I started to observe a lot of different issues.

And it's not until later, because one of my first roles in policy

was working on the research culture program for the Royal Society.

That I started to feel that all those things really hang together.

There's a lot of different things that are causing problems, but somehow

they are all related to each other.

And I had a realization, and it was also one of the outcomes of that

program at the time, that the biggest lever to not change one of them, but to

influence all of them is really around the incentive system and how we assess.

And that's really set me up to, to pursue a path where that is the

main focus of what I try to do.

So all my positions all the different hats that I'm wearing at the moment, they all

center around the questions of research culture and specifically using the lever

of assessment to solve some of them.

So important and that's a theme that comes up again and again

in discussions is the impact of the current incentives systems that we have.

What are you seeing in the people or hearing about.

From the people you're talking to, working with, or what your understanding

is of the current research on this, about the current incentive system.

Yeah, and there's a lot of ground to cover here.

But I think, at the moment, I think there is really a common almost agreement.

I mean, not everyone agrees, but I think there is momentum around

a common understanding on the one hand from the research community.

I think those who really do research on assessment are really in agreement

that the way that especially journal based metrics and also article

based metrics are used that it's really not always appropriate.

And that there's a lot of kind of trickle down effects on not just

researchers, but also the research.

So I think both.

But the research suffers that the scientific record suffers,

but also the people behind it.

We're not really getting the most out of those people.

And I think something that I'm also hearing more and more is that there is

this focus on outputs really kind of this very narrow end of the journey of

it's not even the end it's supposedly somewhere in the middle you should

be done building on those to achieve impact but there's this very strong

focus on the outputs, with very little attention for the things that come

after although impact is picking up.

But also the things that come before, more importantly, the

whole process of research.

And I think also with moving outside academia, I started to realize

how little focus there is on the outputs in other environments.

There's much more focus on how do people behave?

What are the skills they acquired?

And these are things that I've just not visibly seen within an academic setting.

And it's not something people talk to me about from that setting

because they're not aware How important these things are outside.

So I think there's on the one hand, the issues that people talk about, but

there's also really importantly, certain things they don't talk about because

most people who are in academia have been there, they've grown up there,

they've been there their entire life.

And there are certain things that I think would be interesting to

them that they don't talk about because they don't know about it.

So I think those are maybe the two sides to think about really.

Yeah.

And there's also that tension, isn't there, of Even if people might be

aware when you're in the system and on this treadmill chasing these incentive

based quantitative metrics you're moving, doing that sort of thing.

And that there's space to actually stand back and reflect and make deliberate

changes or to talk about this.

It's a bit hard when there are ongoing evaluation cycles within

our universities or within our grant funding programs and so on.

Definitely.

And I mean, it's a question we always get because lots of universities

have started to not explore necessarily how to assess differently.

But at least they've started to think about how do we offer more training?

How do we offer training on all the kind of things people are struggling with?

How do we offer mental health groups?

How do we offer all the kind of things to solve some of these issues?

And not to discredit any of these initiatives, I think they're important.

But sometimes...

what I'm seeing is, and it's also just coming from my own background

as a postdoc, people sign up to these things, but then their experiment

in the lab fails, and the priority is repeating the experiment.

They will just not show up for these things because they need to publish

that paper for their career progression.

So it gets the absolute priority.

And despite all the efforts.

To have trainings, to have support events.

It's just the thing that drops off the list most often.

So I think unless we can embed those things in the incentive system and in

what supports the career progression, it's very hard to convince people to

make the time and space for these things.

And I think that's one of my ambitions is.

really to contribute to an assessment system that can allow people space

for this without being punished in their career progression.

And in fact, rewarded, if it's actually recognized in the incentive

system , valued and rewarded.

Exactly.

Exactly.

And we can see how in other sectors, this works, people are rewarded to develop

certain skills that are not output driven.

They're more around behavior.

They're more around the way they do things.

You know, there's really big questions around how to embed

something that like that in a sector and in an environment where it's

almost a big unknown at the moment.

Yeah.

I had a little smile to myself when you said about people, prioritizing, fixing

up their broken experiment, then going to the course that they signed up for.

Because I ran a course recently, it was around saying yes, no, around setting

boundaries and, setting priorities.

And I think we had.

12 people signed up and three turned up on the day and these were all postdocs

and for similar sort of reasons and it was just ironic that the course that

could help them manage this in a longer term got reshuffled at the last minute

because of other shorter term priorities.

Yeah, and it's usually really not, it's not even that they don't want to be

there, or it's really just this complete drive towards this one very narrow set of

things that will help them move forward.

Yeah.

And I think we're putting them in a difficult position because many

of them know they need these things.

And we are putting them constantly in these dilemmas between what they know

they need, but what helps their career progression, or what they know is best

for science versus what is best for their career progression, because there's quite

a big gap at the moment between the two.

So we're constantly really...

In a way, in an unfair way, posing dilemmas to people.

And in the UK as well, there's lots of conversations now

around integrity and misconduct.

And I think, of course, everyone has a personal responsibility, but

we're not making it easy for people.

Because we're really setting this gap between what are the things to do?

What is the space to take, to think, to develop?

Versus the kind of speed we're expecting them to produce results with.

The two are really incompatible.

So we're really asking people to do something very difficult to constantly,

almost on a daily basis, make choices between these two, between what's best

for science, what's best for career progression, but also between what they

need for their personal development versus what they need for career progression.

So it's just not an easy position.

I think we put researchers in at the moment.

And what you also say there points to it's great that there is an

increased awareness around mental health.

And I'm definitely seeing that far more in the last couple of years.

I think COVID has been helpful as well as bad in that respect, it's made

it normal to discuss much more now.

And I also see the value in individual initiatives and people

taking courses and doing training, but they are useless strategies if

we don't get the systemic change.

Like you talked about the lever being the incentive systems and

the whole research culture and just changing behaviors more generally.

What's the training, what's the process for creating that sort of change?

That's a good question.

It's not an easy one to answer.

And I think the reason why the kind of the initiatives more at the ends

of the process, let's say, are so important is because the systemic

ones are going to take time.

I think we're on a good trajectory in terms of already gaining

understanding that there are problems that really needs to be looked at.

And I think the things that we will need to really get there is, on the

one hand, senior management buy in at every organization and institution

in the sector, because I think it's very important to have researchers on

board, but I also feel we have many of them on board, especially early career.

I think the barrier at the moment is more at management levels.

We really need to get buy in because if we want to get systemic change, it will

need to really be organizational change.

And that will really meet the drive from people at the top.

We have seen the very big examples where.

Change has happened, for example, in the Netherlands, I think Frank Miedema

is one of those people who in a senior position has really just changed so much

in his institution and he will always say, how important it was to be at that

level to be able to make that change.

So I definitely think we need also the bottom up, but I think without

the top down and without that senior management inputs, we just won't get

there because it is a systemic issue.

And I also think we will need to go further than what we're

thinking about now because I think.

Research assessment is a very important lever to think about, but there's also

the issue around precarity of careers.

There's the funding cycles, and there's the amount of evaluation that goes on.

Researchers are probably the most evaluated professionals I

would imagine, across sectors.

There's every output is evaluated separately.

They are evaluated by their institution, by funders to get funding.

I do think with the current reform we'll be able to do a lot, but I do think we'll

still bounce on some boundaries around the life cycle the funding length of

funding schemes, those kinds of things.

So I do think there will be a few other issues that we will need to start to think

about and start to solve to, to really get to some of the achievements we want to

get to in terms of changing the culture.

Yeah, totally agree.

I'm hearing as well there's in terms of the spectrum of

where the changes are needed.

So there's, what we can do as individuals to better skill

or to cope in the day to day.

There's what we could do as an organization in creating our

evolving, changing different local research cultures and practices.

And that's embedded in the bigger institutional policy level systems around

government policies that set the funding programs or set the evaluation criteria.

And that's complicated as well, isn't it?

Because you've got national initiatives, priorities, and you've got International

practices and disciplinary practices crossing over as well, and it's a

really hard space to think about when you start unpacking all those layers.

Definitely, and I mean, the real answer is that we will

need change at all those levels.

But also we'll need to ensure that those levels are communicating to each other.

Because one of the issues we're seeing also when we're talking about the

setup of the coalition is that there are certain levels, like for example,

maybe institutional levels who want to move forward, but they're then

bound by rules set by the level above that they just can't get around.

So unless change happens elsewhere, they are actually kind of stuck in that space.

So one of the things that I see as the kind of big things that the

coalition will bring is a space to actually bring all these levels

together and to enable discussions.

So not to take away any autonomy, I think each organization

should be able to think about.

What it is they can do to progress.

And of course, there is the agreement that sets a very common direction

for everyone, but there's a lot of space within that to explore and to

think about how to implement that.

But at the same time, there needs to be constant exchange of information so that

we Ensure we're not creating different things at different levels or in different

organizations that contradict each other.

So I think that will be really the value of the coalition to

create that space for discussion.

But you're right.

It's immensely complex and, you know, sometimes I refer to it

as one of those wicked problems.

And we will need dialogue at a lot of different levels and, including

also there's, of course, this big discussion around the quantitative

versus qualitative approaches, where some say the qualitative is definitely

the most important at individual level.

But when we go to more aggregate levels, we need some of that

quantitative for it to be manageable.

And I think, in a sense, I think as humanity almost we've become obsessed with

measuring things, not just in academia, I think, it's also well understood at the

moment that GDP is not the best measure for success at a very different level.

So I do think there are bigger questions, not just in academia,

but in more general terms about how we've become as a society to define

success and how that shapes behaviors.

I sometimes hope that we can set an example in academia too, and in the

research space more broadly because it's not just academia there's research

going on in other sectors as well that we need to involve in those

discussions, but we can hopefully really reflect on how it can be done

in a way that we don't cause those bad behaviors and set those bad incentives.

But it's not an easy challenge and it will need really buy in and thinking

and support from all those different stakeholders that you describe.

One of the tensions that I experienced just in a day to day sense is

being passionate about the need for these sort of changes and having these more

holistic measures and valuing research culture and all of the other things that

we've talked about and recognizing that my current PhD students or postdocs are

still going to be working or playing in the system as it is now while it's

still undergoing this transition.

Any hints or suggestions about how to navigate that and how to help them

make choices that both start to play out the behaviors that we want to see,

but not jeopardize their own careers.

It's a difficult, it's very difficult.

I think it's really about, as you say, it's finding a balance.

It's exploring with them how much can I go in the right direction without

losing out on any opportunities.

And I think there's already many kind of ways to participate in things

like open science in thinking about at the team level, how to talk about

inclusion and diversity, how to incorporate some of these elements,

but sometimes also being frank with someone in the team and say, look, in

an ideal world, this is what I would do.

But thinking about your career progression, you might need

to compromise in X or Y ways.

I think we need to be honest about that and be able to really

have honest conversations.

But I also think there are small things we can do to start really

changing the narrative, like one example I often give is, I remember

here in the lab in Cambridge when someone would say that they happily

announced they've published a paper.

No matter who asked the first question, it was always, which journal was it in?

You know, that doesn't need to be the first question we can easily ask a

different question about the content about maybe who it's important for.

So I think, while we don't have the means to immediately change the big things,

and there will need to be compromises.

There are many examples like this where we can start to change the

small things on a day to day basis.

And I do think that those will also already have an impact also just on how

those people grow up and are shaped as a professional, whether that's within

academia or going to other sectors.

So I think there's things to do around that.

Also, can we, I remember my own professional development conversations in

academia, they were well non existing or at least not during the postdoc where they

were maybe most important for me because I was trying to kind of change track.

They were really based on the research.

They were just about the experiments.

I think one thing PIs can do is start to really have development conversations

that really talk about the things, the behaviors, the competencies,

the things that are important.

So I think there are definitely things we can start to do to signal that we're

moving in a different direction without jeopardizing the career progression,

but it's a balance and it's not going to always be an easy conversation.

I like just the small things that we can start to do in what we talk about.

And we've also just tried celebrating when people submit a paper, because that's

really what they're totally in control of, and whether it gets accepted or not

depends upon reviewer two, as we know.

So yeah, also just trying to celebrate, getting something done.

Having written it up.

When you talk about behavior and competencies, this is getting to the nitty

gritty of our everyday research cultures.

What other sort of things would you be wanting to highlight there for us?

I mean, and now I'm really speaking in personal capacity because

this is an idea that I've developed over the last few years and I know

there's others, especially in the research and research space who are keen

to explore this further but it's not something that's been tested much yet.

One of the approaches that I've seen used outside when I went into a different

career direction is for example in interviews or in performance reviews,

the use of kind of competency based approaches where people really ask

you to reflect on how you do things.

So ask questions about, giving an example of a really difficult management,

people management situation and how you've handled that, whether there's

anything you would be differently if, in that situation again.

And I really would encourage to start to put at least a reflection behind

that in academia, because one of the things, and now I'm kind of touching

a bit maybe on the journey elements.

One of the things that I really had to learn when I transitioned

was to reflect on how I do things.

I had never had to do that, and I was around, I don't know, I was

probably almost nearing my early 30s.

I had never really had to, in a conversation within my previous

environments in different universities I was in, No one had really pushed me

to reflect on how I did certain things on how I approached something and how I

put a strategy in place for something.

And this is one of the things I really had to learn, and it's so valuable

to do that, to make time for that.

So I would say that is, I think, one of the things that for me will be

important to somehow embed in the sector.

And I think assessment is a good way to do that because it means

everyone will need to get involved.

It's not optional.

And it's not clear maybe yet how or where it would be fitting because

it might not fit in every assessment process, but it's something that

I think can bring a lot of value.

To help people to really reflect on the way they do things.

And it relates to a lot of culture themes, inclusivity open science,

integrity, for all those areas.

You could really think of questions where people need to think about have I been in

a situation where I felt that my research or someone else's research, there was

maybe an issue or there was a gray zone.

And how did I deal with that?

How would I do it differently next time?

So it's really almost embedding some of those difficult

conversations into what we're doing.

I feel that's an approach that's not talked about so much now.

It's more about how can we measure open science?

Can we measure how many open science, how many open access

publications or how many data sets?

I would really encourage that.

I'm not saying we shouldn't do any of that and especially at aggregate level

to see where we're making progress, but on an individual level, I believe

it will be more valuable to be able to have narratives and conversations

around how to deal with some of this.

And that can also come back a bit to the previous point, Geraldine,

around, could you in a conversation talk with people who are now in that

transition phase in their career?

How do you balance the two?

If you have to make a decision on a publication, Will you give more

weight to publishing open access or to publishing in a certain journal?

How do you deal with those kind of dilemmas, you know?

So talking about what we need to talk about research and how we do it.

And what You're saying also requires managers, leaders, senior people to be

trained in how to ask good questions and how to engage in these conversations

because, you said before about we haven't been socialized into this as

a way of being and doing research.

Yes, definitely.

And the leadership aspect is very important there.

I think this is where the moment we experience a lot of the barriers.

Because we are now, we are in a system where leaders are really not

accustomed to any of those things.

And many of them also don't see a need for them.

I really noticed that in conversations.

They have a feeling that they have been used to doing something a certain way.

And there's absolutely nothing wrong with doing it that way.

So there's a bit of a.

disconnect there.

So there's, of course, a question around, can we already enthusiast,

make enthusiast some of those coming into kind of, say, the next generation?

Are we going to be able to convince those that are there currently?

I'm honestly not so sure what the answer to that is, but also one of the things I'm

really Sometimes breaking my head around is, how can we increase the mobility

between sectors, because for me that would be I mean it's almost a chicken

and egg situation because I think if we can have more focus on process competence

skill, it will be less difficult for people from outside to come in because

they'll have things to talk about.

They don't have to just have a publication record.

They can talk about all those other things in the same way.

But at the same time, and then when you bring those in.

That will also shift how things are done because you'll have people who bring

experiences from other environments and who will be much more open even at

that senior level to some, you know, they'll be more socialized with some of

these approaches, but of course at the moment we have neither we don't have the

instream of those people, and we don't really have the routes to bring them

in because the assessment system is a barrier for them to move into those areas.

You know, we need to get either the in Instream first or change the assessment

system so we can bring them in because there's a lot of appetite to have more.

Mobility, but I do feel the assessment is one of the barriers

to be able to achieve that.

And I do think it would bring a lot of richness to all sectors, if we could have

more of a stream of people in and out and also put less pressure, because now a lot

of people who leave academia, they still feel some kind of failure because they

think there's no way to ever come back.

Whereas if you would have a more dynamic system where people can go in

and out, I think you would, to some extent, solve some of those issues.

Because people wouldn't feel if I leave now, there's never a way

to come back to that environment.

Yeah.

Absolutely.

And also solve some of the big societal challenges that again you raised at the

beginning about one of the motivators for why this is so critical to address.

So is that an example of diversity and inclusion that you just talked about,

like the diversity of backgrounds that we might be enabling in our processes.

Definitely, because I do think at the moment with the narrowness of what

we're assessing, there's really almost this kind of one type of profile that

kind of makes it all the way to the end.

I mean, I'm being a bit black and white now.

It's probably a bit more gray than that.

But I do think we're missing out on a lot of different talents.

And then also the question is, what does that mean for the

research we're missing out on?

Because I really enjoyed the Invisible Women, the book by Caroline Criado

Perez, and she gave this example.

It's more on the innovation side about the first Apple health app, not including

anything on the menstrual cycle because they didn't have a woman in the room.

So no one thought about it.

I mean, they missed out on something huge, absolutely huge.

And then there's also the stories around how, you know, endometriosis

as a disease has not received that much attention because it's just

not a problem that those maybe in charge mostly male had recognized.

So I do think and these are just examples on the gender side.

I'm sure there's examples in all types of diversity direction.

So I think It's not just which people are we excluding, which

profiles are we excluding, which ideas are we excluding, but also

whose problems are we not solving.

So I think that's both diversity of people, ideas and also

the research problems that we really need to think about.

One of the things that gets talked about in the coalition agreement

that we'll get to in a tick is also diversity of research methods and

approaches and disciplinary perspectives and theoretical perspectives, which

is often a dimension of diversity that doesn't always come to mind first.

Yeah, and I think that's very important to also be appreciative of the

methods that are used across sectors, so indeed bringing in maybe some of those

people with experiences elsewhere, but also disciplines to recognize each other's

methods, because one thing that's often coming up also in conversations around

the quantitative versus qualitative is subjective, quantitative is objective,

which I think anyone who's been industry a bit deeper involved in these

conversations really knows it's not true.

But to some extent, I also sometimes see some people in the harder sciences

kind of almost discredit some of the methods used in the softer sciences And,

often unknowingly because they often really don't know that much about those

subject areas, and it's actually causing problems, not just for the methodologies

used in assessment itself, I would say.

So I do think that this diversity of recognizing methods that have

been established, and, they've been established for many years,

is important, not just for the scientific endeavor, but even just

for progressing assessment itself.

Yeah, because we know from the research that interdisciplinary research

is harder to get funded because people often find it difficult to review from

different methodological perspectives or disciplinary perspectives.

Yes, and I think sometimes also don't have that much confidence maybe

methods they're not so familiar with and I do think there's also an element of

trust there in, you know, where there is recognized established methodologies.

I think that also does need to be included and recognized, but it's a challenge

interdisciplinary one definitely.

So you talked about the coalition before.

Do you want to explain what the coalition is and what work it's done?

Yeah, definitely.

In a few, I think it's now two years ago, there was a process that started with a

consultation where there was a realization coming from a number of organizations

who had been working on this for a while.

That there are issues with assessment and the commission wanted to

understand better the problem.

The European Commission.

Yeah, sorry.

So the European Commission wanted to really better understand views of

different stakeholders in the landscape.

So they first started a consultation.

And there was a report really with the conclusions of that, but also

some principles to think about how to move forward and how to really start

to change the way assessment works.

And so this is where I came into the process.

I was invited to be part of drafting an agreement and the agreement was

building on the principles from that big consultation with lots of stakeholders.

But the idea was really to put down on paper in very specific

commitments, a common direction for the sector to move in.

And alongside it also a time frame to say, okay, we'll do

these things by a certain time.

And this agreement was published in July 2022.

And so the commitments, some of them, I mean, are very close to what

we've already been talked about.

The first one's really around recognizing diversity in contributions

to research, but also all the different roles that people play.

And then the other core commitments are really around.

primarily using qualitative assessments, but also avoiding any inappropriate uses

of metrics and the use of rankings in of institutions in research assessment.

And then there's a range of kind of supporting commitments to enable

more kind of the how we do it.

So this is the, these four are like the what's.

And the time frame put to it was for organizations to say, okay, I commit

to this, to have an action plan with very concrete milestones after one

year of signing, and to then also give a more elaborate progress, which is

based on just self assessing themselves after about five years of signing.

And so this agreement is really what this coalition is coming together around.

So after the agreement was published, there was some time that was

taken really to discuss how the coalition should function, which

governance structure it should have.

But so in December last year, there was a constitutive assembly bringing

together all the organizations that want to be part of the coalition.

And I have to check the exact numbers, but we're well over 450 members

member organizations, because it's open to organizations at the moment.

And so the mission of the coalition is really To achieve this systemic

reform that we've already spoken about and the vision is really around the

direction that the agreement set.

So that's why I wanted to first touch on the agreement.

And of course, this is just to start.

We just really started end of last year, but at the moment we are forming

working groups for all our members.

Are able to be involved in proposals to set up working groups to really

start to move forward the agenda and really kind of think about how can we

achieve the commitments and how can we achieve the mission of the coalition.

So a lot of the work is still coming is ahead of us.

But we've been really encouraged by.

the momentum, the enthusiasm, the amount of people who are really like, okay,

we really need to, we really need to do something we've talked about this for

a long time, but it's really time for action and for seeing change implemented.

So I'm quite yeah, positive about what will come in the next few months.

Once these working groups are approved and can really start the work of the coalition

and can really start collaborations.

with organizations in a lot of countries, not just in Europe, because the initiative

now is really globally focused and we have quite a bit of work to do to achieve

some of the international strategy that we're setting out at the moment.

But we are starting to see also organizations from

outside Europe joined slowly.

That's very exciting that there's 450 so far signed up.

If in signing up and putting in agreements, like you talk about

organizational levels, and I'll come to that in a tick, is the EU itself?

Looking at how to implement I'm holding up a copy of the paper that no one can

see because we're going to be on audio.

And is the EU itself also looking at the agreement to review say the funding

criteria for the horizon 2020 or whatever upcoming funding programs they have.

Yes, so they are a signatory the European Commission as a funder

is a signatory of the agreement.

And so they will, in the same way that everyone else will they will

put in place an action plan to think about how they will implement the

change, and they will release it at the same time as other signatories.

And I've also been speaking for example specifically to The MSCA unit who are

working on the Marie Sklodowska Curie fellowships, and they also are really

asking questions about, okay, what do we need to change in our evaluation

processes to be completely aligned?

And I mean, they are already quite ahead, I think they already align with

a lot what's in the commitments, but there's definitely also thinking across

the European Commission around what their action plans need to look like

to fully align with the initiative.

So for people listening , what would they need to do to talk to or convince

their organization to sign up to it?

Do you have any suggestions or any good practices that could serve as role models

for how to increase that membership?

definitely.

And obviously I'm a lot also in conversations where the audience

are more individual researchers.

So usually what I say is try to find at least already in your department

who would be responsible for being involved in some of these conversations.

And that's usually not too difficult.

Sometimes finding the right person at the kind of central university level

can be a bit more difficult, but there should be people on the department.

departmental level who can at least point to the right direction.

And then, yeah, really have the conversations also within your own team

with your supervisor within your own department around why you think this

is important, because we definitely see that where there's appetite from

a lot of the kind of researchers.

It does, it takes some time, but it does tend to really reach the right levels

and at least cause some kind of discussion and initial conversations around whether

it makes sense to sign up or not.

Also, we have a secretariat, all the information is on the coara.

eu website that are really responsive to any questions that people might have

about what it means to become part what the, maybe the requirements are, but also

the many opportunities that it presents, the benefits for the organization.

So there's also definitely easy ways to gain information if people

want to first inform themselves before having the conversations.

But it's really partly trying to really find the people who have a

bit of gravitas to to to put these kinds of things on the agenda of the

conversations where they need to be held.

And that can really just start at the departmental level usually.

Yeah.

And then as you said, there's all the work to say, what does this mean?

How do we operationalize these principles and how do we put them into practice?

Yes, definitely.

And one thing that we found very important in putting in the commitment.

So one of these supporting commitments that I very briefly touched on is to

ensure that the resources are there.

To actually then follow through and that doesn't necessarily mean

financially can also be staff capacity.

Maybe there's someone who's working on a related issue who could

make space in their portfolio to really dedicate time to this.

But just to indeed ensure that when an organization signs up, they have thought

about how to accommodate the change and And of course, this will differ

for different institutions, how much resources, but I think even if there

is just already a bit of staff capacity to take care of this to follow the

discussions, and to be in those exchanges.

There can be differences in how active people participate there will be people

who lead a working group but equally, there will be organizations that feel

okay all we have capacity for is.

To focus on our own action plan and to listen into some of the conversation.

So we know we're going roughly in the same direction and that's absolutely fine.

You know, there's space for all those that diversity within the coalition, but we

do ask to have kind of a minimum resource allocation to ensure that the commitments

can be really followed through.

you talked about the four main principles of the agreement and one

of them you talked about was, not relying on rankings, for example.

. To me, that could be one of the hardest things, apart from the obvious sort of

journal metrics up, but, the ranking system is just so toxic and people are

happy to be critical of it when they're not ranked highly, but as soon as they're

ranked highly, they do all the press releases, and I know that there are

some institutions that have their tenure track or promotion agreements that require

evaluators to assess them against other researchers at equally ranked institutions

and all of this sort of language.

I think that's going to be one of the hardest things to change.

What's your opinion around the ranking system and how we can really shift that?

I agree with you.

It is the most difficult one and I'm going to be very honest.

It's been the one that's been most kind of contentious and most discussed

when we were drafting the agreement.

And in the end, we very specifically worded it that research assessment

of institutions should not be used in research assessment specifically

because saying anything in more general terms was causing a lot of discussion.

I'm really happy it's in there in that form, I'm going to be honest, because I

do think it's important it's in there.

But I think the key thing that was always coming back is it can trickle down.

So some of the language that you're referring to, hopefully at

least that will already decrease.

So that's not seeing a complete ban of rankings in all the kind of uses

they have in attracting students and so on, but at least start there.

Of course, at the same time, there's really appetite to to really think about,

do we really need this as the format for some of the other uses they have.

And I really iNorms.

I don't know whether you're familiar with iNorms more than our rank initiative.

So they have provided this kind of alternative where universities can

talk in a more narrative way about the things that they feel they're

uniquely delivering to students.

So I do think there's exploration also of how can we do things

differently in that space.

But as you say, These rankings are very ingrained in the system at the moment

in how universities talk about their achievements, but also their strengths.

And that will be something very difficult to to move away from.

I do think that there will be work within the coalition to start to

think about some of this, but it's one of, I think, as you rightly point

out, one of the harder problems to think about how to move on that.

But at the same time, I think there are things that can be done already in

assessment specifically around really thinking about where is this language.

Inappropriate and it will be an in most cases.

And at the same time.

Yeah, there are these initiatives really looking at the broader use of rankings in

setting out what the strengths and the unique selling points of universities are.

And interestingly, I was also in a conversation a while ago giving a

workshop with all kind of younger universities who were more maybe focused

also on, on kind of knowledge transfer.

And we were, as part of a conversation on assessment, the starting points.

that I often try to get people to think about is if you have to

really think about the things you're proud of just as an institution what

are the things that come to mind?

And one of the things that came out of this specific conversation was

regional development, attracting people who maybe We're first generation

students really changing the really making a local impact in that sense.

And then the conversation went on, , is this something that's recognized that

you feel is recognized that's taken into account, for example, in these rankings.

And, you know, it was a bit of an eye opener because All these institutions are

the kind of the representatives of these institutions were realizing, these are

not really things, the things we're most proud about are not things we're really

rewarded or recognized for at the moment.

So it did give an incentive to think about why changes is so important.

And so I do think that.

We can have the kind of conversations that make it very clear why these

rankings are not probably the right way of bringing out all the diverse

contributions that institutions make.

So now we're looking at a different level.

But how to practically move away from them.

I think we're at the start of that conversation rather than at the end

where we already see clear solutions.

But it's still so encouraging that the conversation is starting, and

I see that the coalition agreement builds on things like DORA and, The

Metric Tide work in the UK and so on.

And I'm also seeing things at the moment.

I know that in Austria, there's the funding that the government department

is starting to have discussions about.

Revising renewing research evaluation.

I know that Australia is currently running a survey asking for input on this.

So , it is an encouraging time.

And, yeah, will take time.

Yeah, definitely.

And I'm just also thinking when I had the first kind of experiences in

advocating for this 10 years ago during my PhD, I was involved in this action

group in Flanders where we're trying to talk to all the different political

parties in Flanders about changing specifically the key that was used to

allocate money between the different universities in Flanders, because there

as well, they were using certain metrics that were then trickling down to the

assessment of individual researchers.

And it was an interesting experience because I was quite young and it

was something I hadn't done before.

I learned a lot from it, but we really hit a wall.

It was very clear that there were a few people who are taking the decisions

that we weren't really getting anywhere.

So if I compare that experience with where we are now 10 years later, it is

encouraging you know there's a lot of work ahead of us and there's many issues to

still be resolved, but at the same time.

I think those who have been consistent in working in the space

for a number of years can also see how much progress has been made.

Just even in terms of the number of people who are talking about this, the

number of organizations who actually are recognizing that there is an issue,

we're in a very different place already.

So I think if we can make the same amount of progress or ideally even

more in the next 10 years, you know, I'm quite optimistic about that.

Yes.

I really am, because it has to change.

It's not sustainable the way it is at the moment.

I'm just going back to something you said before, I know that there's a

whole shift to narrative CVs as part of the emphasis on quality and people

being able to argue their own value and I know that you were developing one

of the first templates for that when you were working at the Royal Society.

But I had never thought about it at an institutional level.

You know, like an institution can develop.

Its narrative cv, which is interesting.

And even country level, country levels too.

Exactly.

I think we don't need to pour down everything in numbers.

I mean, of course, there are, I'm often conversations about where they have

value and I understand that there are situations where they have value, but

I do think there's a lot we can do with narratives and with giving people

space to focus on the things that they find important because the numbers

often narrows down to a few things that, have organically been chosen or

historically been chosen to be important.

And they exclude a lot of other things.

Indeed the more than our rank is an institutional level where there is space

for for organizations to really think about what it is they bring us strengths.

And I think we can do that at national level as well, too.

And I think there are some countries that are very in regards.

What are their research strength?

So I definitely see that we can do that at different levels.

But at the same time, it is probably the most important at the individual

level, because there we are really dealing with people in different

circumstances and contexts and not giving space to explain some of that I think

is really important to me suppressing diversity, suppressing inclusivity.

So I think we can do it at all levels, but for me, it is the most

important that we, that is the primary method at the individual level.

And that really needs breaking of the superstar researcher

and that competitive culture.

I also recognize, that if you've got 100 applications for a position and

you only want to send 10 applications out for review and you've got to do

that sifting process, it may not, I don't, I think there's still lots of

tensions we need to work out about how do we bring a qualitative eye to doing

this larger scale stuff and how do people still argue for their own quality

when they're so used to the metrics.

But I feel like there's, there'll be progress and we'll work it out.

Yeah, and I think, there's also, there's been also conversations

I've been where someone says, okay, we need the metrics because we can peer

review every article of that person, but those are not the only two options.

There are lighter touch qualitative elements you can look at that don't

require you to return research publications back to front.

So I think we haven't explored all the options yet, frankly.

And I think there's also a question about how often we evaluate, because there's

qualitative assessment, all good and but it takes a lot more resource and time.

But then my question is, do we need to evaluate as much as we do?

Could we evaluate less, but do it better, you know?

So I think there are a few aspects that we still need to really think about.

And sometimes there's a lot of focus on one element, like the

narrative CV is a great instrument.

And I am very enthusiastic about it, but it's not the kind of golden bullet.

We need to look at the entire process and ensure that it's embedded in a

process where there's changes that other stages and in different ways as well.

So I think there is a lot of opportunities still to look at.

Other parts of the process to explore methods that we just aren't used to

yet and haven't used yet, but also to really think about frequency and

whether we need to evaluate so often.

Do you have a, just a brief sort of one or two examples of other

methods or other light touch ways of bringing that sort of qualitative lens?

Yeah, I mean, and here I'm drawing again a bit more on

my experience in other sectors.

With what I often also see with academic positions is the role

descriptions are very broad.

People feel like they need to send in a 10 page CV because.

they need to cover all the grounds.

Whereas in other sectors, Descriptions are often much narrower.

And I think there is a question also around do academics need to

do everything because they start to have a broader and broader portfolio.

Or would it be beneficial to find a compromise between thinking, for example,

about institutional and then departmental strategy and thinking about, okay, we

have this interdisciplinary team, but we really need someone who focuses more on

the outreach element or policy engagement.

Can we.

Hire someone who has that in their profile and balance that still with

ensuring there's independence for people to develop in the ways they want to.

So it could be that we don't go, of course, all the way to what other sectors

do, but there's maybe a profile that gives some flexibility, but also gives

a few more specific requirements as to what we're looking for in a person.

And then you can really start to write very short paragraphs These are the

things that I've done that's address the exact requirements you're looking for.

If you're looking for a person with this type of more specific profile, you can

explain usually in quite a concise way how you fit that without having to give

the whole list of all your experiences that you have over your career.

So that's one example where you would need a bit of, see a bit of change

in the way that jobs are positioned.

You'd need a bit more work on from institutions and departments to

set out what are the skills they need to complement the teams that

they are trying to build, and the achievements the mission, the

vision they're trying to achieve.

But that then would make it easier.

To get to much more concise formats, which would need much less time to review.

So that's just one example.

That's great.

I think we probably need to head to wrapping up to let you go and I had

said I wanted to talk a little bit about your journey and you've already

reflected on that, at different points.

One thing I do want to reflect back from what you've said is any of us,

like what you've illustrated is any of us at any stage of our career can

be part of changing the conversation of, Arguing for change because you

talked about in the Flanders example, even, 10 years ago, you were still a

PhD student, yet you were there, in the mix of it, making these arguments.

Yes.

And I think it's partly also about setting priorities because I saw, I don't know

why, maybe it's, maybe I have to thank my parents, but I had the confidence to do

other things, so I was not only focusing on my research, and that's been really

a red, thread throughout my career.

I've always had the confidence to make space for the things that I found

important and I must say it didn't, I didn't really see any suffering

in my career as a researcher, as a consequence of it, so I do think it is

possible But it was also because I was able to set boundaries very easily.

So I think it is possible.

But it also really depends on what you prioritize and what's important to you.

And I think the key thing there is to understand what your values are.

And the thing I've tried to do is really Live and work in line with those

values and it's driven a lot of the choices that I've made in my career in

the volunteering that I've done in the things I've dedicated my time to and

for me it's worked out because it's got me to a point where I'm really enjoying

all the things I'm doing but That is something I think, in a way where I've

been lucky because that was instilled in me and it's not something everyone

has an easy time with, I think, setting boundaries, saying no, choosing for your

values is not something that's easy and for some people will require support and

training that's maybe not always there.

So I do recognize that's a challenge in itself.

But I do feel that's what's been helping me to do some of these things.

And as you see we all have a voice and I did refer to this

early example where I used it.

It wasn't very effective, but I learned from it and, it helped my

personal development and I do feel it's contributed to where I am now.

So I think we also sometimes need to recognize that we're Things haven't

quite worked out in that moment.

They do still contribute to to, to our learnings and to who we become as a person

and what we can maybe achieve later on.

That's really brilliant, Karen, I think that's just lovely.

Are there any final things that you would want to add or say before we wrap?

Yeah, I think maybe just one thing to still refer to is,

you know, I knew I was gonna change direction quite early in my postdoc.

So I started my postdoc in Cambridge in 2015.

And I think in the first few weeks, I already was talking to the career service

because I just felt at that moment, I want to finish this postdoc, but that's

going to be, there's going to be a turning point that I'm going to need

to do something where I'm just working with people more and with samples less.

That was one of the motivations.

So people is one of your core values.

Yes, definitely.

And even knowing that, so I transitioned in the end in 2018.

So three years later.

It was still so hard.

I still felt that failure that I talked about earlier of leaving, although I had

made that decision three years earlier.

So you have to imagine the stigma around it to still feel that way at that point.

But what I did realize is that a lot of the things that I thought that I would

miss from the academic environment, the freedom, the independence, the

kind of the space for thinking.

Those aren't things I've lost.

I still have been able to build those in my career.

And I've also changed a lot in terms of kind of roles and jobs in the last few

years, and I have realized that decision shouldn't have been so difficult because,

it's really the world is changing.

People are trying different things , in my sector, people move around a lot.

And, one decision is really not setting your career direction

for the rest of your life.

So I would encourage people to really think about what it is they value and

whether that's what they're getting at the moment in their work environment.

And if it's not, maybe it's time for a change.

And although we talked about how difficult it is at the moment to have that to sector

mobility, I don't think it's impossible to go back to the academic sector.

I've already had questions actually from people in the research on research

space, whether I would, what I would ever considering going into that space.

So it would be a different discipline.

But it's all becoming more dynamic.

And I think people should really think about whether they're getting out of

their work and their work environment, what is important to them, and if

not, to not be too afraid of change, because change can be a good thing.

And I think there's a lot of opportunity to move around, especially now.

That's great.

Karen, I thank you so much for your contributions today and the

really interesting discussions and I will provide links on the web

page to the many of the names and papers and that you've referenced.

Thank you for your work in being part of making that change happen and glad that

you did make the decision to leave and shift directions to contribute to this.

And hopefully this will encourage lots of other people to get involved as well.

Whether it's in the small conversations in their workplaces or being part of trying

to push that the agreement or whatever.

So thank you very much for your time.

Thanks for inviting me.

And of course, I also hope many people will get involved in all the

different ways, because we really need at all those different levels

from the small conversations to the senior management team signing up.

So yeah, hopefully we'll get a lot more engagements over

the coming months and years.

Thanks very much for inviting me.

So glad to have been able to bring you this conversation with Karen

about such an important topic.

And I'd repeat the call for each of us to be part of making this change happen.

From the small conversations we can have every day.

To encouraging our senior management and organizations to sign up to the agreement.

And then there's the work of how to put the agreement to

work in a practical sense.

And also just remind you that as this conversation was recorded

in May, you may want to make sure to check out the CoARA website.

coara.eu.

To see where it's at now.

And if you liked this conversation, you might also be interested in

listening to other changing academic life conversations on similar themes.

And I'm thinking here of the more recent conversations with Sarah Davies around

research researchers, research cultures.

With Tanita Casci and Elizabeth Adams from Glasgow uni around.

Including collegiality as part of research evaluation and the conversation

with James Wilsden on metrics and responsible research evaluation.

You can find the summary notes, a transcript and related

links for this podcast on www.

changingacademiclife.

com.

You can also subscribe to Changing Academic Life on iTunes,

Spotify and Google Podcasts.

And you can follow Change Acad Life on Twitter.

And I'm really hoping that we can widen the conversation about how

we can do academia differently.

And you can contribute to this by rating the podcast and also giving feedback.

And if something connected with you, please consider sharing this

podcast with your colleagues.

Together, we can make change happen.