Artwork for podcast Changing Academic Life
Reforming research assessment top-down bottom-up middle-out (solo)
Episode 4 • 27th September 2023 • Changing Academic Life • Geraldine Fitzpatrick
00:00:00 00:41:33

Share Episode

Shownotes

Following the conversation with Dr Karen Stroobants on the EU CoARA agreement, I discuss some other top-down initiatives at international and national levels for reforming research assessment. I also share some of my own bottom-up experiences trying to put these principles to work eg in writing references and being part of evaluation panels. It is also my hope that these actions can also have some middle-out influence.

Overview [41:33 mins] - full transcript available:

[00:00:29] Episode introduction

[00:02:10] The 10 Commitments of CoARA

[00:05:02] Other international initiatives

[00:10:34] Netherlands as example of national initiatives

[00:17:20] Some of my 'bottom up' examples

[00:34:18] Middle out strategies

[00:38:35] Wrapping up

Related links:

CoARA: Coalition for Advancing Research Assessment and CoARA Commitments

LERU: League of European Research Universities 

LERU Publication: A Pathway towards Multidimensional Academic Careers 2022

DORA: San Francisco Declaration on Research Assessment 2013

Leiden Manifesto for Research Metrics 2015 

The Metric Tide: Review of metrics in research assessment 

Strategy Evaluation Protocol 2021-2027 

Room for Everyone’s Talent 

Related podcasts:

Karen Stroobants on changing research culture and reforming research assessment

Sarah Davies: Part 1 on mobility, precarity and notions of excellence  and part 2 on luck, disrupting excellence, and cultures of care

Tanita Casci and Elizabeth Adams on supporting, rewarding and celebrating a positive collegial research culture 

James Wilsden on metrics and responsible research evaluation 

Keywords:

Academia, CoARA, Diversity, Governance and policy, Research culture, Research Evaluation



This podcast uses the following third-party services for analysis:

Chartable - https://chartable.com/privacy

Transcripts

Geri:

Welcome to Changing Academic Life.

2

:

I'm Geraldine Fitzpatrick, and this is

a podcast series where academics and

3

:

others share their stories, provide

ideas, and provoke discussions about what

4

:

we can do individually and collectively

to change academic life for the better.

5

:

Finally, I'm getting to the episode that

I wanted to do about following on from

6

:

my conversation with Karen Stroobants.

7

:

If you remember, Karen is a

researcher and policy advisor and

8

:

consultant, and in our conversation in

particular, she talked about her work

9

:

contributing to the development of

the European initiative called CoARA.

10

:

, C O A R A.

11

:

The coalition for advancing

research assessment.

12

:

And the agreement that comes out of that.

13

:

If you haven't already listened

to that episode, I encourage you

14

:

to stop listening to me now and

go and listen to that one first.

15

:

As always with these sorts of agreements.

16

:

There's the challenge of moving on

from actually having words on paper.

17

:

And signatories to agreements to actually

putting these agreements in place.

18

:

and that means.

19

:

Really significant change initiatives.

20

:

And in that episode, Karen.

21

:

talked quite compellingly about the

need for both top down and bottom up.

22

:

Buy-in.

23

:

If we're going to create

such research culture change.

24

:

And she also encouraged us to have

both the small conversations in

25

:

our every day situations as well as

at more senior management levels.

26

:

And about the systemic issues as well.

27

:

That need much more coordinated action.

28

:

What I'd thought I'd like to do in this

episode is to complement what Karen talked

29

:

about and flesh it out a little bit more.

30

:

, by providing a little bit

more detail about CoARA.

31

:

First of all, I thought it could

be really useful just to read out.

32

:

And name the 10 commitments of the CoARA

agreement, just because I think they do a

33

:

really nice job of summarizing the core.

34

:

Well, the core commitments, just

exactly as they say on the tin.

35

:

So I'm not going to read every item

fully, but just so you get the sense.

36

:

Number one commitment is to recognize

the diversity of contributions to and

37

:

careers in research, in accordance with

the needs and nature of the research.

38

:

Two.

39

:

To base research assessment

primarily on qualitative evaluation.

40

:

, supported by responsible use

of quantitative indicators.

41

:

Three abandon inappropriate uses

in research assessment of journal

42

:

and publication based metrics.

43

:

Like the general impact factor in H index.

44

:

Four.

45

:

Avoid the use of rankings of research

organizations in research assessment.

46

:

Five.

47

:

Commit resources to reforming

research assessment to achieve

48

:

the organizational changes.

49

:

Six.

50

:

To review and develop research assessment,

criteria, tools, and processes.

51

:

Seven to raise awareness of research,

assessment reform, and provide transparent

52

:

communication, guidance and training.

53

:

Eight to exchange practices and

experiences to enable mutual learning.

54

:

Nine to communicate progress made

on adherence to the principles and

55

:

commitments And ten, to evaluate

practices, criteria, and tools, and so on.

56

:

I imagine that these are all

commitments that we would be

57

:

really keen to sign up to.

58

:

And indeed.

59

:

As at the 31st of July is

reported on the CoARA webpage.

60

:

They've had 608 organizations that

have signed up to these 10 commitments.

61

:

And , as I mentioned around Karen's

conversation last week, they're currently

62

:

setting up lots of the working groups

that are trying to do that work of

63

:

putting the commitments into action.

64

:

And we know of course that this is going

to be a challenge at a European level.

65

:

Since Europe, isn't a homogeneous.

66

:

Region.

67

:

It has lots of different countries

and institutions and cultures.

68

:

And so there is a lot of work, a lot

of work to practically translate.

69

:

, the principles of the

commitments into action.

70

:

But still, I find it really

encouraging to see this happening in.

71

:

And I'm excited to see what I think

is a real momentum for change.

72

:

So it's not just COARA , I can, we

can also point to some evidence of

73

:

more general moves for change that now

are showing up in other initiatives.

74

:

So another initiative at the European

level is something called LERU L.

75

:

E.

76

:

R.

77

:

U.

78

:

The league of European

research universities?

79

:

And that's an initiative that involves

23, what they call themselves as

80

:

leading universities and to paraphrase

directly from their document.

81

:

This league was based upon an exchange of

current practices and recent developments

82

:

at the universities regarding the

assessment of researchers in the context

83

:

of hiring promotion and evaluation.

84

:

in particular, the document I am

looking at is a LERU framework

85

:

for the assessment of researchers

and a position paper from:

86

:

And they talk about developing a common

framework that can inspire and support

87

:

universities in the, in the context of

their hiring promotion and evaluation.

88

:

With an issue.

89

:

With an objective to reward and recognize

again that, that argument about the

90

:

diversity of profiles and contributions

and recognizing that we need all of those

91

:

in order to be a success, whether it's

in research education or in service.

92

:

And in this particular framework

document, they also talk about three

93

:

different perspectives of assessment

that I think are really interesting.

94

:

The first one is the

multi-dimensional perspective.

95

:

And here they're drawing attention

to the diversity of contributions.

96

:

They also talk about the

developmental perspective.

97

:

And here they draw attention

to issues around leadership and

98

:

innovation and collaboration.

99

:

And.

100

:

Point to the necessary transpersonal

skills that are needed to engage in

101

:

those activities and therefore the

need for us to develop ourselves.

102

:

As people in those interpersonal skills.

103

:

And the third perspective

that they draw attention to

104

:

is the contextual perspective.

105

:

And here they're pointing to.

106

:

Both professional circumstances, as

well as personal circumstances and

107

:

contextualizing peoples research.

108

:

In those ways.

109

:

And that this in particular being

very important for inclusivity.

110

:

So this particular research paper then

continues with examples and may also,

111

:

encourage experimentation of different

approaches to assessment and may also

112

:

call on policymakers and funders to

explicitly support such experimentation.

113

:

So just as another

example, similar to CoARA.

114

:

It's again, this thing of pointing to

recognizing and rewarding diversity

115

:

of profiles and contributions, and,

, Recognizing that we need all different

116

:

types for the overall success of

research and in the service of society.

117

:

And there are many other examples

that I won't go into in any detail.

118

:

And we can point to the Leiden

manifesto for research metrics from

119

:

2015, that's been quite influential.

120

:

, I can also point back to the episode

with James Wilsden, where he talks

121

:

about the metric tide report in the UK.

122

:

And.

123

:

Um, we can see that there are lots

of these discussions and initiatives

124

:

happening at national and cross national

levels, which points to some of the

125

:

systemic changes that we might need.

126

:

And of course, many of these

initiatives can also be traced back

127

:

to DORA the San Francisco declaration

on research assessment from:

128

:

And I think DORA was particularly

influential in starting the discussion

129

:

about moving away from the use of

journal based metrics , or at least

130

:

driving the discussion a bit further.

131

:

, And looking at.

132

:

, how we could assess research on its

own merits rather than just on the

133

:

journal in which it was published.

134

:

And even more impressive

is the sign up to DORA.

135

:

So again, on their web page, they

report that since:

136

:

23,936.

137

:

Individuals and organizations in

164 countries that have signed up.

138

:

So these are really indicating some big.

139

:

Initiatives internationally.

140

:

At least on paper for these

top down systemic initiatives.

141

:

And we can also see some top

down initiatives happening.

142

:

At more national levels as well.

143

:

Where at national level, they're starting

to actually think about putting some

144

:

of these initiatives into practice, at

least in terms of policies and funding.

145

:

By funding councils or relevant

government departments.

146

:

I know that the Australian research

councils have recently had a survey

147

:

looking for input on different

forms of research assessment.

148

:

Um, as we've said, the UK has

been doing a lot of work on this.

149

:

In Austria, where I'm

currently located there.

150

:

I've also been involved in discussions

with the relevant ministries in Austrian,

151

:

involved in different workshops.

152

:

Again, looking at how we might

reform research assessment.

153

:

And bouncing off a lot of the work that's

going on in these other initiatives.

154

:

One country though, that I'd

particularly like to draw attention to.

155

:

Is the Netherlands.

156

:

Because I think they're doing

particularly well, at least

157

:

looking in from the outside.

158

:

In setting in place, um, some of

the supporting processes for new

159

:

forms of research assessment.

160

:

A very simple document that I can

point you to is a:

161

:

that involved universities and funding

councils and government agencies.

162

:

Called room for everyone's talent.

163

:

And it's short and sharp

and visually compelling.

164

:

And it's focus.

165

:

Is on how to rethink academic reward

and recognition systems, as it says,

166

:

And again, what I like about it is the

way that it promotes the diversification

167

:

of career paths and enabling people

to define their own career shape.

168

:

And then that shape becomes the shape

against which they are to be evaluated.

169

:

So they recognize the

diverse career paths.

170

:

They also focus on quality rather

than quantitative measures.

171

:

, Stimulating open science is another plank.

172

:

, Also recognizing the balance between

individuals and the collective.

173

:

And stimulating academic leadership.

174

:

We can start to see lots of

common themes across these.

175

:

So in terms of recognizing

that science requires both

176

:

individual and team performance.

177

:

Yeah.

178

:

This reflects some of the conversation

that we had with Tanita, Casci

179

:

and Elizabeth Adams about what

Glasgow university are doing.

180

:

For example, around.

181

:

Rewarding collegiality and their

assessment and promotion criteria.

182

:

Various such university councils

and funding councils have also

183

:

collaborated to produce a document

called strategy evaluation protocol.

184

:

And this sets out aims

and specific methods.

185

:

, as advice to be used to assess

research at Dutch universities.

186

:

And, , they often go through

this process of assessment every

187

:

six years at Institute level.

188

:

So this might be where faculties

bring in an external expert committee.

189

:

, the faculty produces a report on their

activities and performances over the last

190

:

six years on the external panel will, come

in and have meetings with various relevant

191

:

groups, , and discuss , the report with

them and write some sort of assessment.

192

:

And again, what's really interesting

about this strategy evaluation protocol.

193

:

Is that it's really trying to take into

account the issues around diversification

194

:

of career profiles and open science

and the importance of culture in

195

:

producing high quality research.

196

:

And so the assessment committee

is explicitly asked to look at

197

:

things like research, quality,

societal relevance, and viability.

198

:

And the unit that is reporting and

writing the report are encouraged to.

199

:

write their, report it gain in terms of a

more narrative qualitative based argument.

200

:

And wherever possible, still using

factual evidence, but not relying

201

:

on indicators is the report.

202

:

and isn't driven by the indicators.

203

:

And the guideline document.

204

:

Also has in the appendix, some

very specific examples about

205

:

how organizations might go about

reporting against the criteria.

206

:

It sets out.

207

:

So while this document is definitely

targeted and written for the Dutch.

208

:

Audience, I think any institution about

to go undergo a faculty review could pick

209

:

up this document and think about how to

reinterpret it for their own context.

210

:

I think it's really encouraging that

these sorts of initiatives are happening.

211

:

And that people are trying

to put it into place.

212

:

I'll put links to all of these

documents and initiatives that I've

213

:

referred to in the podcast notes.

214

:

So, , you can see, you can

go and follow up yourselves.

215

:

So I think there's lots of really

exciting stuff happening top down.

216

:

But we know that these initiatives.

217

:

Slow to make, to be put in place

that change on the ground takes

218

:

time and can be really difficult.

219

:

And it really does take

significant culture change.

220

:

And while we might have some

really encouraging pointers for

221

:

larger scale systemic change.

222

:

We still need the bottom up change of the

individuals every day, sitting on a panel

223

:

or discussing a promotion case and so on.

224

:

And that's challenging because many

of us have grown up in systems.

225

:

Where there's a certain sense of, I

don't know what we would call it.

226

:

Sort of sense of security or

safety or familiarity with

227

:

our quantitative measures.

228

:

And, you know, the seduction of

numbers that can also seem like these

229

:

quantitative measures are more efficient

and also more reliable and comparable.

230

:

So I know that even if we have

these international agreements, that

231

:

organizations have signed up to, or

if we have national initiatives and

232

:

national guidelines on take more

qualitative approaches in the end,

233

:

it does come down to you and me.

234

:

, as reviewers, as people sitting

on evaluation committees,

235

:

promotion, committees, and so on.

236

:

And that's where I think a lot of

work has to be done in terms of

237

:

how to practically operationalize

the aspirations and commitments

238

:

articulated in these documents.

239

:

And that's one of the

commitments of CoARA.

240

:

Around doing the communication

and guidance and training.

241

:

And ,we need to also feel like, , find

new ways to feel confident that

242

:

we're making good decisions, because

when we're talking about qualitative

243

:

assessments, and we're talking about.

244

:

individuals and institutions being

able to define their own shapes

245

:

of, you know, research profiles

against which they're evaluated.

246

:

And that's going to take a lot

more work because previously in

247

:

the past, we've relied on naive

notions of being able to do direct

248

:

comparisons, comparing and contrasting.

249

:

So I wanted to just share now some

examples that we might call bottom

250

:

up examples from my own attempts at.

251

:

Trying to walk the talk a little bit

around some of these initiatives.

252

:

So I can think about this

in particular, in terms of.

253

:

Being a referee, for example, in writing.

254

:

doing evaluations of someone's case

for promotion to professor, or if

255

:

I'm on an appointment committee and

being asked to evaluate candidates.

256

:

And one of the things that I really first

need to do is my own self reflective.

257

:

Self-awareness about trying

to check in with my own biases

258

:

and just being careful that.

259

:

I'm not being sucked in by H indexes

and looking to compare people,

260

:

just on H indexes is an indicator.

261

:

and we have the discussion recently

with Sarah Davies, where she certainly

262

:

critiqued our notion of excellence that

are based just on these indicators.

263

:

So if I'm a referee , and looking at

these, , guidelines and documents,

264

:

They've really challenged me to

think about how I might write a more

265

:

narrative qualitative referee report.

266

:

I'd like to give some specific

examples, all anonymized of course,

267

:

, of things that I've done when I've

been asked to assess a candidate.

268

:

Often still in terms of impact

factors and personal metrics.

269

:

You know, where people are still using.

270

:

Some of these and institutions are

still using some of these old criteria.

271

:

So one example of what I've done

when I'm writing a references.

272

:

Where I have been asked explicitly

to assess a candidate in terms

273

:

of impact factors and personal

metrics, such as H index.

274

:

I'll often make an explicit statement

that says something like this.

275

:

I'm very surprised to be asked to

explain the channels of dissemination

276

:

in terms of impact factors and

personal metrics such as H index.

277

:

This is out of step with many of the

current research assessment reform

278

:

initiatives, such as DORA CoARA.

279

:

Leiden manifesto, et cetera.

280

:

These initiatives make strong cases

for moving away from such quantitative

281

:

metrics or using them responsibly.

282

:

If they are to be used and focusing

more on quality indicators.

283

:

End of quote.

284

:

So I'll often make that explicit comment.

285

:

Speaking back to the assessment committee.

286

:

Before I then go on and provide some more

qualitative contextualized assessment

287

:

of that person's outputs and impacts.

288

:

Because I know that where committees

are still using these old indicators,

289

:

they will be influential still.

290

:

So one of the things that I do, , in

terms of trying to contextualize

291

:

work is where the case that

I'm reviewing is particularly

292

:

cross-disciplinary or multi-disciplinary.

293

:

And we know from research that

people who work in publish across

294

:

multiple disciplines can often

have challenges getting published.

295

:

So in the abstract, if they were

working with solely within a single

296

:

discipline and on a very well-defined

topic in an area that does tend

297

:

to have high citation rates.

298

:

We could see that this person's H

index could be deemed to be higher.

299

:

And so what I try to do in a referee

report is to contextualize their H index

300

:

relative to their multidisciplinary work.

301

:

So again, just to read an example

of something that I've written.

302

:

It needs to be acknowledged

that the candidates research is

303

:

extremely challenging because

it crosses multiple disciplines.

304

:

This is in terms of both

the knowledge and skills and

305

:

collaborations they need to draw on.

306

:

And the challenges of speaking

back to diverse disciplines.

307

:

Who all have different ways of

doing research and evaluating

308

:

and reporting on research.

309

:

End of quote, from that reference.

310

:

Similarly, , another issue that I

think we can help contextualize as a

311

:

reference referees is to contextualize

different types of research that people

312

:

do that may entail different timeframes

and complexities, not one being better

313

:

than the other, but just being different

and having different implications for.

314

:

Publication outputs and

you know, and the numbers.

315

:

So some of us might do research

where we can work intensively

316

:

for a month on a project.

317

:

I get some really deep,

interesting results and write them.

318

:

Um, up and get them published.

319

:

And other others office might be

working on projects that take a number

320

:

of years that might involve, I don't

know, doing some contextual research

321

:

and developing technologies and

deploying them and running for example.

322

:

Some full randomized control trials.

323

:

Afterwards.

324

:

And these sort of projects are at very

different levels of scale and complexity.

325

:

And so do have different

implications for publication cycles.

326

:

So that means that we really can't

compare one person, seven journal

327

:

papers in a year with another person's

one or two journal papers in a year.

328

:

When they're doing these very

different styles of research.

329

:

Again, as an example, I can read from

a reference for someone who I'm trying

330

:

to, again, trying to contextualize some

work for this latter style of research

331

:

that happens over longer periods of time.

332

:

So to, to read from that reference.

333

:

It's worth noting.

334

:

That the candidates publications are

based on years worth of research.

335

:

When many of their peers

publishe papers based on work

336

:

from a scale of months, if that.

337

:

Evaluation is a case in point.

338

:

The scale and difficulty of conducting

full randomized controlled trials.

339

:

Are exponentially greater

than the scale of evaluations.

340

:

Of many other.

341

:

The high quality publications

in this disciplinary.

342

:

Every.

343

:

Area.

344

:

Yet RCTs are what is needed.

345

:

If this research is to speak to the

clinical community and to impact

346

:

clinical practice and patient outcomes.

347

:

And this candidate is evidently committed

to having this impact and doing these more

348

:

difficult longer-term evaluations for the

sake of impact, not just publications.

349

:

End of quote there.

350

:

And this reflects what.

351

:

Karen talked about in the, the

decisions that we often have

352

:

to make as researchers about.

353

:

Are we going to do research because we

want to have impact or do we want to

354

:

build up or need to build up our CV?

355

:

And I do appreciate that.

356

:

Uh, you know, being at

different stages of our careers.

357

:

These sort of questions may create

pressures in different directions.

358

:

Of course.

359

:

I can give another example of the way

in which initiatives and documents like.

360

:

The CoARA agreement have really

helped me in writing fair

361

:

assessments of someone's performance.

362

:

Again, it has taken extra work, I

think, on, on my part as the reference

363

:

writer and it has required me to be

more considered and reflective and

364

:

to take more of a qualitative lens.

365

:

So one example is, where I was

asked to write an assessment

366

:

for someone's promotion.

367

:

Uh, to professor.

368

:

And this was from a university

that in, in their criteria

369

:

document that they sent to me.

370

:

Set out all the usual very standard

metrics around H index and impact

371

:

factors and level of funding and so on.

372

:

So, what I did at the beginning

of the letter was just clearly

373

:

named the elephant in the room.

374

:

If you like, where I

clearly said that, um, Yeah.

375

:

By some of the criteria that

the university was asking me to

376

:

judge the candidate on, They

weren't going to be very strong.

377

:

But then I could go on and say in

the evaluation report and to stop

378

:

at, to read from that reference.

379

:

However to only view the candidates

achievements through very narrow notions

380

:

of quantitative scientific excellence,

misses the significant and unique

381

:

contribution that this candidate has made.

382

:

The international standing

in the field and the diverse

383

:

outputs practices and impacts.

384

:

And if I may, I will address

these in, in the rest of the.

385

:

Referee report.

386

:

As I qualitatively judged their

promotion application, according

387

:

to the principles from CoARA.

388

:

I consider these

principles more inclusive.

389

:

And they explicitly recognized diverse

career paths and contributions,

390

:

as in the case of this candidate.

391

:

And of quote.

392

:

I was able to go on then and what I did.

393

:

and this was an institution that had

not signed up to CoARA, but what I

394

:

did was it wasn't just me saying are

they're doing great work or they've

395

:

had these other impacts I could

quote directly from the agreement.

396

:

So again, like I might say something

like the CoARA agreement specifically

397

:

promotes recognition of diverse

outputs practices and activities based

398

:

primarily on qualitative judgment.

399

:

, and supported by a responsible

use of quantitative indicators.

400

:

So.

401

:

I would put that in

quotes, um, from the CoARA.

402

:

The agreement.

403

:

And then I would also

point to the fact that.

404

:

The agreement also talked about

recognition for different types of

405

:

impacts beyond scientific measures.

406

:

So I would say.

407

:

, and it was here in particular that

this candidate was outstanding in their

408

:

ability to bridge, research and practice.

409

:

And they've had really

significant impact on practice.

410

:

So what the CoARA agreement gave me

was some specific language and some

411

:

credibility to argue for why I was

choosing to review the candidate

412

:

against a different set of assessment

criteria than what I'd been asked for.

413

:

, And then I could finish off.

414

:

I still need to go on and do the

work to make the arguments for why

415

:

they were significant impacts, but

it did allow me to sort of, again,

416

:

just reinforce and at the end of

that description of how they made the

417

:

impact, I could say, you know, these.

418

:

Valuable contributions that

researchers make to science for the

419

:

benefit of society, including diverse

outputs beyond journal publication.

420

:

So I could explicitly acknowledge

that they might not have had

421

:

the journal publications, but

they had lots of other impacts.

422

:

Another point that I often

get asked to do in writing

423

:

reference letters that I really.

424

:

, don't like at all is when I'm.

425

:

Asked, , and this was, this is from

a specific request that I received.

426

:

Yeah.

427

:

How does the candidate compare with strong

researchers in their or closely related

428

:

fields at similar stages of their careers?

429

:

The mention of specific

individuals would be appreciated.

430

:

So that's what I was asked to do.

431

:

And you may remember that.

432

:

One of the commitments from CoARA

was about avoiding the use of

433

:

rankings of research organizations

in research assessments.

434

:

What I did in response to that

particular question was explicitly to

435

:

say, no, I wasn't going to do that.

436

:

And again, to quote, uh, from the

reference that I wrote in this case,

437

:

I've chosen, not to name specific

individuals as point of comparison.

438

:

As I consider the candidates

achievements to stand on their own.

439

:

It's also very difficult to compare

researchers when this candidate has

440

:

such a unique disciplinary profile.

441

:

A comparison with anyone within

a single disciplinary area

442

:

would not be relevant or fair.

443

:

So even though the CoARA commitment

specifically talks about not using

444

:

rankings of research organizations, I

think we can extend that to interpreting

445

:

it as not doing the comparison of

individuals either specifically, when

446

:

we talk about the fact that we can have.

447

:

Diversification of career paths

and recognition of diverse types

448

:

of outputs and, and in particular

contextualizing people's performances.

449

:

And that makes it really ridiculous

to think we can directly compare

450

:

and contrast individuals.

451

:

We can also see the way in

which this might discriminate

452

:

against particular groups.

453

:

So even if a couple of individuals

might be comparable on some

454

:

measure, like, I don't know.

455

:

Let's just say they're

being five year, five years.

456

:

Post-doc.

457

:

How do you compare a person who's single

unattached at, totally focused on their

458

:

work at a well-funded institution.

459

:

He gets lots of researchers support.

460

:

How do you compare that person?

461

:

With someone who has new parenting

responsibilities, who's on a

462

:

precarious contract and at a

less well-funded institution.

463

:

Or how do you compare the really

selfish researcher who doesn't do

464

:

any service and just expects everyone

else to provide peer service in

465

:

reviewing their papers, et cetera.

466

:

Versus the people who are doing all

the work to organize the conference

467

:

or edit the journal that, that

first person's going to publish in.

468

:

So I just point blank, refuse to do that.

469

:

And again, I appreciate being able to draw

on some of these documents and agreements.

470

:

To argue why I'm doing that.

471

:

It's not just me being

difficult to get along with.

472

:

And in a similar way, another issue

that often comes up in evaluations

473

:

that we're often having to deal with.

474

:

, is around service.

475

:

And we know again from the literature that

women in particular are overrepresented

476

:

in performing service roles in departments

and faculties and in peer communities.

477

:

And often at a cost to

their own promotion chances.

478

:

And this is work that's often

under-recognized and under rewarded

479

:

when it comes to promotion and

appointments, especially when we.

480

:

Implicitly fall back or

explicitly fall back on.

481

:

On the quantitative metrics.

482

:

, so again, I, draw explicitly on

the language of the CoARA agreement.

483

:

And I might say something like, you

know, the CoARA agreement argues

484

:

that while the agreement focuses on

the specific challenges of improving

485

:

research assessment, the principles.

486

:

Should also be extended and reinterpreted

into broader academic service, , that

487

:

includes teaching and service to

society so that we can, and again, like

488

:

the, , recognizing such as the Dutch.

489

:

, Room for everyone's talent you about

achieving, recognizing that the

490

:

importance of collaboration and those

interpersonal skills and the service.

491

:

And , the Glasgow work at

rewarding collegiality.

492

:

So in all of these.

493

:

They're just some examples.

494

:

I've really appreciated having ,the

international national, ,initiatives

495

:

and agreements to draw on.

496

:

And just to try to give some

credibility to taking different stands.

497

:

And recognizing that it does take more

work to write a qualitative argument

498

:

and to contextualize, peoples profiles.

499

:

I'm also not sure, particularly how.

500

:

How the committee's receiving my letters.

501

:

, I read them or accept them.

502

:

I, I do know that, , in a couple

of cases, the candidate did.

503

:

Get their promotion or did get the job.

504

:

, to what extent that may have been down

to my arguments and I'd, I, I don't know,

505

:

but, , it, it has been good to at least

have these initiatives to draw on to, to.

506

:

Contribute.

507

:

So, I guess in some ways the

examples I've just given are

508

:

examples of bottom up, me taking my.

509

:

Opportunity as a reference writer

or a committee member or whatever.

510

:

To influence specific cases.

511

:

And I'm also hoping that they're

also a little bit middle out and

512

:

here I would define middle out as

middle out, being able to try to

513

:

influence both the specific decisions

being made on this specific case.

514

:

As well as trying to influence,

and, raise awareness, change some

515

:

of the practices and influence

some of the broader structural

516

:

issues or institutional, issues.

517

:

So I'm using the opportunity as a

reference writer or committee member

518

:

to raise awareness., because many

people on the committee may not

519

:

have even heard of some of these

initiatives or may still be really.

520

:

,set on applying old criteria.

521

:

Or at the very least they might

stimulate some discussions.

522

:

And that can be done in directly through

the letter, or if I'm actually sitting

523

:

on a panel or a committee I can directly.

524

:

Stimulate some of those discussions.

525

:

What I think is useful if you're

actually on a faculty evaluation panel

526

:

or an advisory board or whatever.

527

:

, is also to go and just check if that

institution has signed up to an agreement.

528

:

Because that can also provide some

very specific hooks when you're

529

:

sitting in a committee meeting to.

530

:

, remind people around the table

that the institution has signed up

531

:

to the agreement , and to promote

discussions about how they might be

532

:

going about actually operationalizing

and implementing the principles.

533

:

I think they might also be some bottom

up, maybe middle out influencing

534

:

by mentoring roles that we play.

535

:

, I know for example, one institution

had signed up to the LERU agreement.

536

:

And, , there were some members

of their faculty were needing to

537

:

write some new policy documents

for the faculty about promotion.

538

:

And they weren't even aware that the

institution signed up to this agreement.

539

:

So I was able to point them.

540

:

As a mentor to the, to that.

541

:

agreement.

542

:

Around, the pathways towards

multi-dimensional academic

543

:

careers that I mentioned earlier.

544

:

I was able to point them to that

document and, uh, that may have

545

:

been able to help them in write.

546

:

Some arguments in their policy,

reflecting the agreement that

547

:

the institution signed up for.

548

:

And they were aware that this might

have been contentious in the faculty.

549

:

You know, where there was still a lot of

emphasis on more quantitative measures.

550

:

And we can also influence others as

well in encouraging people to think

551

:

about how they might frame their CV or

their annual report or their promotion

552

:

cases or their job application cases.

553

:

In more qualitative terms.

554

:

Again, you know, you may be able to

reference some of these initiatives,

555

:

even if they're not directly asked

for even if a qualitative, ,narrative

556

:

CVS and particularly asked for.

557

:

And I know that there's a whole lot.

558

:

We still have to learn about working

with narrative CVS, both in terms

559

:

of how to write them as, as well

as how to evaluate them, because

560

:

they can be more challenging.

561

:

But I had a colleague recently.

562

:

Tell me that they chose to write their

annual report as a narrative CV.

563

:

Even that wasn't asked

for by the institution.

564

:

And they just said what a great experience

it was that it really made them step

565

:

back, and appreciate what they'd actually

achieved and the impact that they had.

566

:

And it made them feel

really proud of their work.

567

:

In a way that they're bullet listed.

568

:

old version of the CV didn't do at all.

569

:

And I'm also aware too, of how I

might role model that and thinking

570

:

about, Walking the talk a bit myself.

571

:

And I realized that on my web

page, you often had that bio

572

:

that just reported on that.

573

:

The numbers and the output

driven metrics driven type CV.

574

:

, So I've, I've tried to reframe.

575

:

At least some of the introduction

to, to put it more in terms of

576

:

what drives me and what I focus on.

577

:

that points to the impact

that I'm trying to have.

578

:

So just wrapping up, what I've tried

to do is just point to both some of

579

:

the, what I think are some interesting

top-down initiatives that are happening

580

:

in creating momentum for change.

581

:

And I'm also talking about some of the

ways that we might influence change.

582

:

From the bottom up and middle out

as individuals and being part of

583

:

trying to make that change happen.

584

:

And of course we can try to

encourage our institutions to sign

585

:

up to these agreements as well.

586

:

And you can also think about getting

involved in working groups, if you want

587

:

to . And I just also want to just say that

in sharing my reference writing examples.

588

:

I'm not asking now for lots of requests

to write references, I'm actually

589

:

going to be doing fewer of these.

590

:

If any, as I move into different roles.

591

:

But I'd encourage you to all think

about how you're writing your reference

592

:

letters or how you're playing out

your role sitting in committees.

593

:

And ways that you might contribute to

enhancing and improving and developing

594

:

our institutional policies and practices.

595

:

So I'd really liked to

encourage us all to do our part.

596

:

And see what we can do.

597

:

I think it will take some years for

it all to play out in real practice.

598

:

I think there's in part some sort of

generational change that's needed.

599

:

But there's momentum now towards this.

600

:

And we all want better forms of

research, evaluation, and assessment

601

:

that more sustainable and more

appropriate for the diversity of

602

:

contributions that we can make.

603

:

And we can all argue in different

ways, in different forms for

604

:

how we enable, um, diversity.

605

:

And recognize the importance of team

collaborations and diverse roles in

606

:

producing good science and recognizing

diverse forms of impacts and outputs.

607

:

And recognizing the value of good

leadership and good interpersonal

608

:

skills in the middle of all of this.

609

:

And in creating research

cultures that are sustainable

610

:

and collaborative and supportive.

611

:

And I'm excited that there are

moves happening in these directions.

612

:

Let's make it happen.

613

:

You can find the summary notes,

a transcript, and related

614

:

links for this podcast on www.

615

:

changingacademiclife.

616

:

com.

617

:

You can also subscribe to

Changing Academic Life on iTunes,

618

:

Spotify, and Google Podcasts.

619

:

And you can follow

ChangeAcadLife on Twitter.

620

:

And I'm really hoping that we can

widen the conversation about how

621

:

we can do academia differently.

622

:

And you can contribute to this by rating

the podcast and also giving feedback.

623

:

And if something connected with you,

please consider sharing this podcast

624

:

with your colleagues together.

625

:

We can make change happen.

Links

Chapters

Video

More from YouTube