We spend a lot of design effort on how data gets into a system — the forms, the validation, the transactional database quietly recording everything in neat rows. We spend far less on how it comes back out. And when the person who needs a weekly report is a loss prevention officer rather than a software engineer, "just use the API" turns out to be no answer at all.

In this episode, Andrea Magnorsky and Kenny Schwegler are joined by Kat Hempstalk, a data and AI professional who has been doing machine learning "since before it was cool." She brings what she calls "a bit of a tragic tale" from her time as head of data and AI at a company handling retail crime reports. Before she arrived, the company had solved the export problem by hiring a graduate to write Python and SQL scripts and email spreadsheets to customers by hand. It worked — until that person went on holiday and literally handed their laptop to a colleague so the reports could keep going out.

An engineer eventually received that laptop, thought "this is a bit scary," and added a little automation: a Windows VM, scripts synced through GitHub, scheduled tasks, files dropped into blob storage. The manual button-pressing stopped. The underlying problem just got faster to reproduce. Within a few years, three or four people spent their days copy-pasting scripts into 200 bespoke customer reports — reports that turned out to be five or six shapes wearing different hats. Collapsing them was the easy part. Telling customers their format was about to change, when the customer success team said "No, I'm out — I don't wanna give the customer bad news," was not.

This conversation is about where data engineering and software engineering stop talking to each other, and what leaks through the gap. We dig into invisible contracts (a column heading is an API), why the order you migrate customers in is an architectural decision that cost this team a year, what embedding data people into software teams actually requires, and Kat's discovery that most of the customers fighting to keep their scheduled email had never once opened it.

Key Discussion Points

• [00:01] A Bit of a Tragic Tale: Kat introduces the retail crime platform — reports go in easily, and getting them back out was never designed
• [00:03] Your Users Know Excel, Not APIs: Why loss prevention experts don't want a programmatic endpoint, they want a spreadsheet
• [00:05] The Laptop Handover: The graduate goes on holiday and passes the laptop along so the reports keep flowing
• [00:06] "A Bit of Automation" Becomes Load-Bearing: A Windows VM and scheduled tasks make the problem faster to reproduce, not smaller
• [00:07] Three People Copy-Pasting Scripts: 200 custom reports, an unhappy analyst team, and customers waiting months to add a column
• [00:11] The Query That Came Back Too Fast: Moving to a data warehouse, and the insights analyst who assumed the speed meant they'd made a mistake
• [00:13] Two Hundred Reports That Were Really Five: Consolidating into templates — and why that's a conversation problem, not a technical one
• [00:19] The Emails Nobody Opened: Customers insisting on scheduled reports they'd never clicked, because they wanted reassurance, not data
• [00:24] "No, I'm Out": Customer success refuses to deliver bad news, so the data team has to own the conversation
• [00:26] The Order You Migrate People In: Starting with friendly customers built momentum, but the resistant ones set the pace for everyone
• [00:29] A Column Heading Is a Contract: Customer-defined attributes reordering spreadsheets, and moving to nested JSON so nothing downstream breaks
• [00:34] Clear Fences Make Good Neighbours: Andrea on the boundary between data and software teams, and Kat on embedding people across it
• [00:39] What Goes With the Person, What Stays: The handover conversation they skipped, and the systems that broke silently as a result
• [00:42] Collaboration in the Agentic Era: Why the people best at sharing context and contracts are also the people getting the most out of coding agents

Guest: Kat Hempstalk

Hosts: Andrea Magnorsky, Kenny Schwegler

Hello, and welcome Stories of Facilitating

2

Software Design and Architecture.

3

Today, I'm here with my usual conspirator,

Kenny, well as with a, a new conspirator,

4

that is gonna join us for this event, Cat.

5

Welcome, Cat

6

Kathryn Hempstalk: Thanks, Andrea.

7

Um, nice to be along to, to this podcast.

8

Um, I am a data and AI professional,

so, I've been doing machine

9

learning since before it was cool.

10

And, my, career has largely been

about embedding AI into software

11

products, so features that are

driven by AI behind the scenes.

12

but today I really wanna talk about

a story that's about the intersection

13

of, data and AI with software

engineering, and, a very common sort

14

of challenge about, getting data

out of an application that, I think

15

a lot of software engineers face.

16

it's sort of a bit of a tragic tale.

17

does have a happy ending, hopefully.

18

so yeah, happy to share

that with you today.

19

Andrea Magnorsky: Excellent.

20

Well, we're very much looking

forward to that golf recount

21

Kathryn Hempstalk: Cool.

22

All right, so, um, this is a story

that actually started before I joined,

23

my, company, AURA, that, I have been

working for as a head of data and AI.

24

And, before I joined, they had, a

growing number of customers for their,

25

product, which is on retail crime.

26

so AURA takes, shoplifting

reports from, like supermarkets

27

and big, big box retail stores.

28

when someone comes in, they, they,

go to a shelf, they steal something.

29

One of the staff in the store will

go and write a report and put it

30

into the AURA platform, where it

will get sort of consolidated with

31

other reports and, law enforcement

can come and, and look at this.

32

And so as you can imagine, this is quite

a common problem around the world, and

33

there's a lot of these reports happening.

34

And when a retailer gets these, reports

into the system, they also want some

35

mechanism to get the reports out of the

system, like, a way of downloading things.

36

And I think as software engineers,

we often think about, transactional

37

relational database stores as a way

of storing the information that's

38

being entered in these reports.

39

But when we think about getting the

information out of the platform, it's

40

either in the UI of the application

itself, so being presented back to

41

the users in some way, maybe like

on a one-on-one, "Here's the report

42

that you entered," back at you.

43

or it might be via an API, right?

44

Where, you're giving a programmatic

way to, to grab large volumes

45

of, of data out the platform.

46

So, you know, give me all of the reports

that have been entered over the past day,

47

week, month, whatever period of time.

48

The, the problem with those kind

of approaches is that a lot of

49

the users using these kinds of

platforms aren't software engineers,

50

and they're not, experts at

working with APIs and databases.

51

And, even if the APIs exist, they may

not know what to do with them, right?

52

They…

53

These people that we were, working with

were often loss prevention experts, so

54

people who worked on the security side,

who were good at, accounting for all the

55

sort of, shoplifting and other sorts of

crime events that might happen at stores.

56

So they don't know about APIs.

57

What they know is something like Excel.

58

And so what they were wanting is,

basically a spreadsheet of all of

59

the records that they'd entered.

60

So, you know, the store staff are entering

records one by one, and at the end of

61

each week, the team leads can go and

grab, a download of all the reports that

62

have been entered over the past week.

63

So before I joined, Aura they had

started, solving this problem of

64

getting the data out in spreadsheets

by hiring someone who was, fresh out

65

of university and just basically just

employing them to write some scripts,

66

some, Python and SQL scripts that would,

export the data and, then they would

67

manually email it to the, the customers.

68

So not, not the greatest customer

experience if the customer, has

69

to make a request, I want this

data in this, this format and, you

70

know, getting it back via email.

71

but the customers did get what they

were asking for, so they could actually

72

customize the individual columns

and data and how it was filtered and

73

how it was presented back to them.

74

But as a startup grows and scales up and

acquires more customers, that becomes a,

75

a problem that's limited by that person

running scripts, pressing the buttons.

76

and, you know, I think they did

the best that they possibly could

77

with the task in front of them.

78

But, they, they knew the tools.

79

They could write Python

and SQL pretty well.

80

They didn't really understand how much

to automate beyond that, and because it

81

was a small business and everyone was

super busy with, the company scaling

82

up, there really wasn't the support

from the software side to kind of put

83

some automation in behind the scenes.

84

And what ended up happening was, when

that person would go away on holiday,

85

they would hand their laptop over to,

to somebody else on the team, who would

86

then press the buttons to s- send out the

reports at, at the right kind of time.

87

And, very fortunately, an engineer, was

one of those people who was given that

88

laptop and sort of went, "Ooh, this,

this is a bit scary to have a very

89

manual process as part of the system."

90

So, they decided to put a

little bit of automation in.

91

So they spun up a, virtual

machine, a Windows remote,

92

um, remotely accessed desktop.

93

They, used GitHub to sync all of

the scripts into, that desktop

94

machine, and then they used, Windows

Schedules Tasks to basically fire

95

off the scripts at a given time.

96

And then they wrote a

small piece of software.

97

we were using, Microsoft Azure.

98

And so, what they would do is, put the

output CSV spreadsheet files onto Azure

99

Blob Storage, and assuming that the path

had the right sort of information in it,

100

it would actually just get automatically

emailed out as a link, to, the customer.

101

So, um, effectively, there was no

more manually pressing buttons.

102

There would be a schedule that

would go out at a given time of day.

103

And as the business continued to

grow, it still became an unmanageable

104

problem, and they hired more people

to basically cut and paste each of the

105

scripts, make a new version that was

that customer's customized version of,

106

a report and manual, um, manually send,

107

set the schedules so that it was

sent out at the right time of day.

108

You know, some customers would

want it at 5:00 PM at night.

109

Some would want it first

thing in the morning.

110

Sometimes it was daily,

weekly, fortnightly.

111

You know, different sort of cadences

and, and timings for all of it.

112

so by the time that I, I joined Aura,

they had had this team working on it for,

113

a few years, and it had grown to sort

of three or four people whose day-to-day

114

was basically cutting and pasting these

scripts, modifying them a little bit,

115

and Very scary if something was wrong

with one of the scripts and you found

116

out with one of the later copies that

you made, and you'd want to go back down,

117

the path to see where it had come from.

118

And I think just all round just a

bit of a tragedy for everybody that

119

the analysts weren't happy because

they weren't doing anything that

120

was really using their brains.

121

It was cutting and pasting and, you

know, not to their, to their skill set.

122

The, customers weren't happy because

they had to wait a long time if they had

123

a change because there was so much work

in the pipeline that if they said, "Hey,

124

can you just add this extra column?"

125

Sounds simple in practice, but when

you've got, you know, a backlog

126

of other customers asking for that

thing, you know, there's gonna be time

127

involved in getting that out to them.

128

And I think not great for the business

either if you're scaling and you're

129

gonna have to add a person every time

that you're adding, more and more,

130

customers and scripts to the system.

131

So when I joined, I was asked

to do a couple of things.

132

So I, I mentioned my background

is AI, and that was one of the

133

tasks that, I undertook initially.

134

But the other side of things was adding

a data warehouse, partly to support

135

that AI side of things, but also partly

to support what that, insights team

136

were doing for those manual reports.

137

Um, so how that, um, initially got built

was, um, we had a secondary server,

138

so a, a mirror of what our production

server was, and we, had the reports

139

initially running off that, server.

140

So they weren't impacting production.

141

But that caused its own problems, right?

142

That you've got a secondary server, you're

not gonna pay for production transactional

143

SQL Server the same size as what your main

customer thing is just for that one task.

144

And so it meant that actually

it took a long time for some

145

of these reports to run.

146

So not only do you have them, very

manually managed, but there's also

147

a big risk that they could fall over

because something might time out, right?

148

And some of our customers were

really big customers and entering

149

in thousands of reports in a day.

150

And so what you would find is that it

might take a couple of hours to run

151

a report, and if another report was

attempted to run in that time, there was

152

a good chance that it could fall over.

153

And so, you know, people were spending

nights and, and weekends trying to,

154

keep the thing up and running, as well.

155

And so, yeah, the data warehouse presented

an opportunity to, to start to change

156

that, I think, because, it was something

I think the software engineering teams

157

weren't so interested about, because

they were focused mainly on, on the

158

platform at the time and, and building

new features into the main, main part.

159

And where their heads were at

with the data warehouses, it,

160

it's something that's used for

internal analytics that, um, uh…

161

and th- this isn't, um, strictly

an internal analytics problem.

162

It was being solved by a team that would

typically do, um, that kind of analytics,

163

but, um, you know, uh, in a different way.

164

And one of the cool things about the, the

modern data warehouse technology that's

165

around, we were using Snowflake, but

BigQuery and, the things like, Databricks

166

where you can, get some of the power

of data warehousing out of a bunch of

167

tools, that are, are wrapped together.

168

They, they're really good

at the analytics workloads.

169

Like, what they're built for is

big SQL joins, big aggregations

170

across vast amounts of data.

171

And what's really neat about some

of the, managed ones like Snowflake

172

and BigQuery is that you don't have

to think too much about the indexing

173

and, how to organize that data to

really get that power out of it.

174

That the magic of the platform is that

they do all that behind the scenes.

175

So we could take a query that would,

take a couple of hours to run on this

176

SQL secondary server with a few joins

in the mix, and it was, you know, less

177

than a couple of minutes to run on the

smallest possible compute on, something

178

like Snowflake just because it's the right

tool for that kind of task and, and job.

179

when we first put in Snowflake, I

actually had one of the insights, folks

180

come up to me, and it was a real moment

when I thought, "Oh, they've, they've

181

finally understood how this can help

them," when they said, "Oh, I ran this

182

big query and, you know, it normally

takes a couple of hours, and I thought

183

I'd done something wrong because it

came back, you know, almost immediately.

184

And then I realized I

had done something wrong,

185

that I'd even forgotten to filter

it, so it had d- gone over the

186

whole data set, not even just the

little chunk that I was after."

187

And so they were just astounded

it could work that fast, to…

188

Yeah, s- just absolutely

mind-blowing to, to them at the time.

189

So, so that was a, a real sort of

win and the initial step that we

190

took was actually just moving the

SQL onto the, the data warehouse.

191

So same queries, same data structures,

but now it's running faster 'cause

192

the compute is, is designed for that.

193

and over time, we built, I guess,

views and materialized tables

194

within that data warehouse where

it's layering it up to have, more

195

of those joins just done for you.

196

So the data's pre-prepared.

197

When you go to it and run the query,

it's going to, not require you to

198

think about the dozen or so tables

that might need to come into the mix.

199

It's just a single table that's already

got the information formatted into it.

200

and we use DBT to do that,

which is an infrastructure as

201

code tool for data warehousing.

202

So you write SQL queries and it, puts

them all into, tables and such for you.

203

so that worked quite nicely, but we

still had 200 of these custom reports

204

that were being run for, um, you

know, lots of different customers.

205

So, what the team started to do was

realize because they'd been cutting

206

and pasting things, it was actually

maybe, like, five or six sort of

207

main reports that they were running.

208

And actually, they could consolidate those

200 reports down to the, the five, or

209

so, templated versions of those, reports.

210

So a standardized version, change it

in one place, it changes for everybody.

211

what that does mean is that customers

might get more data than what

212

they were expecting before, right?

213

Because you, some might have only had a

few columns, but the templated version

214

has, has extra ones in them, that you try

and give the, smallest version that works

215

for everybody, out of, out of doing that,

that kind of template And where, um, it

216

became a real, um, challenge for us to, to

navigate was actually the customer side.

217

You know, having the conversations with

the customers to go, "Okay, so you've

218

been getting these custom reports, and

we've been firing them off to your email.

219

and you know, it's been falling over all

the time, but we want to make it robust.

220

But in order to do that, we

also need to template it.

221

So actually, you're gonna get the

data in a slightly different format.

222

You're gonna have to change your own

systems that are, reading from that data

223

and using it in, in different ways."

224

and some of the customers just

go, "No, we don't, we don't

225

have time for that," you know?

226

And when you're a growing company, it's,

often the customers unfortunately that do

227

dictate, you know, where, you, you spend

the time and, when you can do a thing.

228

So you might have a great plan,

as we did, to, you know, move

229

the customers in tranches off the

reports and, you know, navigate them

230

onto the, the templated versions.

231

And, you know, we'd have to push the

dates out or move them to a different

232

tranche of, of people, just because

they didn't have the capacity to, to

233

deal with it from, from the other side.

234

at the same time, one of the

cool advantages that, the data

235

warehousing, tools provide now is

the capability of data shares, like

236

direct data shares often within the

same data warehousing technology.

237

So, one trick we decided to, to do with

a few of the bigger customers who had

238

really big, volumes of data that, you

know, they're not gonna want to spend

239

time downloading a CSV file, regularly.

240

And we weren't sending the CSV

files as, attachments to emails.

241

They were sent as a link that

they would then download off the

242

platform, but you know, it could

still be quite a large file.

243

using these data share, capabilities

meant that we could actually create

244

a copy of their data within our

data warehouse that was just their

245

data, but in a set of tables that,

that were really well-defined,

246

and they could get those live.

247

Like, if they were using Snowflake

like we were, it would be live.

248

If it, if they weren't using

Snowflake, then we could dump it out

249

to, something like BigQuery could

pick up and just load straight in.

250

So it meant that actually there was…

251

it was possible to get a whole

lot of automation in there.

252

I think as an engineer going back to

the, "Oh, well, they could have used

253

the APIs to, to do that," well, one,

our APIs at the time didn't quite

254

have the capability to, to pull it

out, in the way we would have liked.

255

so they couldn't have used our

APIs to do that at the time.

256

But assuming that they were, a lot

of data people really like dealing

257

with snapshots of data, so they don't

want to, necessarily have an endpoint

258

that they're gonna have to query and

think about how they're querying that.

259

But, um, often, it will still be

spreadsheets and, CSV files or, a

260

folder full of CSV files if it's

multiple tables of information.

261

that is something that they can

inspect directly and go, "Yes, it's

262

got the whole day's worth of data."

263

They can deal with all the duplication

that might happen if, there's

264

overlap between the time zones.

265

Um, but, you know, the, the idea of

programmatically querying an endpoint

266

doesn't always come naturally,

in the data engineering space.

267

And, you know, there are tools

out there that, people sell that

268

make their money on the fact that

people don't want to deal with APIs.

269

there's lots of tools that people will

charge money for that's basically,

270

you know, here is the platform, get it

from this source to this destination.

271

You make magic happen.

272

Here's my key to, to make it work.

273

Yeah.

274

And here's a bunch of money

to, to, to keep it going.

275

And you know, it just,

just happens magically.

276

And, and there's nothing wrong

with that approach, whatsoever.

277

but a lot of the customers that

we were dealing with, wouldn't

278

necessarily have the funds to, to do

that, sort of API, querying piece.

279

and even if they did, wouldn't

necessarily have the expertise

280

or the time to pick it up.

281

So we're very much wanting data

files, one way or the other.

282

So w- we created two, two options out

of that data warehouse by having those,

283

reports available as CSV files and then,

as the, the direct downloads as well.

284

So story's not over yet.

285

There's more to, to this, to this tale.

286

So we started to migrate the

customers off the customized versions

287

and onto the, the templated bits.

288

but we still had scheduled tasks

running on a virtual machine that wasn't

289

really being updated as we would've

liked, didn't have all the security on

290

it that we would've liked, and would

fall over, particularly on public

291

holidays, as we often found when no

one's around to, turn it back on again.

292

So, we undertook moving, it into a

self-service element of the platform.

293

So actually just giving people the ability

to press a button in the platform and say,

294

"Hey, I want this version of this report,

and give it to me for this date range."

295

And then went back to all the

customers and said, "So you

296

know how you've been getting a

scheduled version of that report?

297

Well, now you can go and

get it whenever you like.

298

but also we're gonna turn off the

schedules, by the way, 'cause, you

299

know, we don't, we don't really think

that, they're in anyone's interest.

300

and That largely went okay.

301

There was definitely a few customers

that pushed back and said, "No, I like

302

getting that email in my inbox, thank you

very much, and I'm an important customer,

303

so please keep that up and running."

304

And then you realize that most of them,

when you look at the email tracks that,

305

that you might get out of, the, the

tooling that, that's sending emails

306

that actually they've not been opened,

they, they haven't clicked on it.

307

They just want the confidence that

that data is there if they wanted it.

308

and so you end up having conversations to

prove to them that they're still getting

309

the data the way that they want it.

310

It will always be there.

311

It will always be accessible through

the self-service and through the data

312

shares, and they don't, they don't need

to have lots and lots of links in their

313

email, just for safety sort of, reasons.

314

the other thing that we ended up doing

was, as part of that, getting it into

315

the platform side of things, is we built

an Azure Function that, would pull the

316

data directly out of the data warehouse.

317

so there was no need to have individual

Python scripts running, that Azure

318

Function was kicked off through,

Airflow, which is a very common

319

tool in the, data engineering world.

320

Um, I think it came out of Netflix or

something like that, um, many years ago.

321

But, it's an orchestration layer,

so, you know, something happens at

322

a given time, sequence of events.

323

And in this case, it was, you

know, where do these, files need

324

to go when they press this button?

325

Or, with the schedules that were sometimes

still running on the platform, at this

326

time of day, this person gets this

email, and we gradually deprecated those.

327

where we, we, we kinda got to was

having a, platform that meant that,

328

yes, there, there was an API, and yes,

there was the ability to look up the

329

data from the UI within the platform.

330

but there's also these two

options with the data sharers

331

and those reports being able

to be downloaded by, you know,

332

non- non-technical people.

333

and the next phase of that, was to

start building, semantic layers.

334

So using the data that's there,

provide ways that, agentic systems

335

and humans can query that data and get

the meaning of the data alongside it.

336

So, you know, understanding it, not

just a spreadsheet with a bunch of

337

column headings and, rows that you can

actually, reason over the data as well.

338

Um, and so that- that's kind of

where I, I left it with Aura.

339

I've since moved on, from that role,

and the team are continuing the fight.

340

But I am pleased to say that before

I finished up, that they had,

341

turned off the reporting virtual

machine that had been the, thing of

342

nightmares for, anyone on support.

343

And, all of those custom reports were now

sort of self-service, multiple different

344

ways of getting data in the application.

345

And, I think it's very happy ending

to that journey even though it took us

346

probably about two years, to g- to get

there, where at least one year of that

347

was just that transitioning onto the

templated reports for the customers.

348

You know, working through in tranches

and spending time with them, helping them

349

change their scripts and understand, what,

what they were getting and looking at.

350

and it's a very common pattern, I

think, for software that when you

351

put it in, any sort of app, you're

expecting the data that you put in you

352

own, you should be able to get it out.

353

And there's even regulations in

the world, you know, GDPR has,

354

requirements to be able to, get the

data out for access requests and such.

355

And so, I think if we'd, as a company,

been more aware of that upfront, maybe

356

it hadn't started with a little bit of a

sad tale of someone doing a lot of manual

357

work, that maybe some of the upfront how

we're gonna export data out might have

358

come in earlier and sort of saved a whole

bunch of, frustration for a lot of people.

359

But we got there in

the end, which is good.

360

Hooray.

361

Kenny Schwegler: Thanks.

362

Yeah, so, um, lots of

bits and pieces, in there.

363

my first thing is the, what we didn't

touch upon before in the recording

364

is with the customers, right?

365

That was a whole, whole thing because

early on you just try to implement every

366

request from a customer because you're a

startup and a scale-up, so you're like,

367

"Oh, we need to keep this customer happy."

368

And then later on, I hear is like,

"It's not, it's not doable anymore.

369

We need to go more to a

product mindset," right?

370

It's the product versus customer I guess.

371

how did you start off these, these

conversations with these clients, right?

372

Did you first start it off or did you

first this is how we want it and then

373

s- kick off with the c- with the client,

374

Kathryn Hempstalk: Yeah, it was

375

Kenny Schwegler: a whole

playing field there

376

Kathryn Hempstalk: Yeah, it was a

really big challenge at the time.

377

we had a customer success team, so

customer support, they would be the

378

people who usually talking backwards

and forwards to the customers.

379

And they would be the ones that would

get the requests for, changes to

380

the reports even though they weren't

the team that would action it.

381

And what was fascinating to me when

we said we wanted to start templating

382

these reports and migrating the

customers over, that we thought

383

that that would be, you know, a, a

better way to approach the problem

384

long term, is actually the customer

success people said, "No, I'm out."

385

Like, "I'm s- I don't wanna

give the customer bad news.

386

This is not gonna be … You

know, I'm, I'm there to support

387

them and make them happy.

388

I want them to, to be feel that,

everything that we do is for them

389

and successful, and this feels

like we're gonna tell them that

390

they can't have something that

they've been having for a while."

391

so, it was kind of actually put back

on us and, we, we deliberately started

392

with customers that we thought were

friendly to, to kinda help with,

393

with that, pathway, and framed it,

not so much of a this is, this is

394

our problem, but, you know, this is

actually improving the world for you.

395

Right?

396

That you're gonna get standardized data.

397

It's gonna be, available more frequently,

more reliably, and, you don't have to

398

wait two months whenever there's a change.

399

We'll notify you because there's a

standard format that, that's going in.

400

And if we do this, we can actually

make it self-service in the platform

401

so you can get it whenever you want.

402

You don't have to rely

on the email to do it.

403

And actually, that sold most of

the customers o- on it, right?

404

That, actually they were very happy with

the idea of self-service, and over, you

405

know, maybe a handful of people who, it

was definitely the minority of customers

406

that wanted to keep the emails going

and they wanted their customized formats

407

that, you know, they had a way about doing

things and little bit adverse to change.

408

I think if I was recommending for

other people who might be going down

409

this path, I think it was good to

start with the easy customers to give

410

us some confidence and some momentum.

411

I think it was the more challenging

customers that were more resistant

412

that actually meant that it took a

year, and if we'd started with them

413

earlier on in the process, we could

have shortened up some of those t-

414

timelines, because they were the ones

that were pushing back and saying,

415

"No, I can't do it until this date."

416

And so it meant everybody was delayed

waiting for the, for the things, uh, those

417

customers to be ready, rather than if we'd

dealt with them upfront even though it

418

would've been a hard conversation to have.

419

Especially without feeling that

momentum on some of the easier

420

customers, it would've actually moved

faster for everybody, along the way

421

Kenny Schwegler: That's

what we usually don't see.

422

I had that epiphany at some point,

is that I guess these larger

423

customers use it in their own

integration maybe as well, right?

424

I've seen this happening as a software

engineer when I was thr- three years in.

425

I just changed the library, which

changed, like, the, the, the layout,

426

and all of a sudden I got these requests

like, "My report is, is, is, screwed up.

427

There, there's an o- other title and

now the rest isn't working anymore."

428

I'm like, "You're using that

for your further integration?"

429

"Yes."

430

"Oh."

431

So I guess that's the

struggle to find out, right?

432

That we usually don't that I think

you mentioned it at the start, right?

433

These people know Excel,

so they, they would

434

Kathryn Hempstalk: Yeah.

435

And, and the kind of tools that

they're using is, will ingest

436

typically Excel, CSV-type files,

so the structured data files.

437

but depending on the tool that they

could use, they could struggle with

438

columns appearing in a different order.

439

so we actually had, some custom

fields that, I guess were

440

attributes about some of the sites

where shoplifting was happening.

441

And this, these were

customer-defined things.

442

And so if they added a new

attribute, it would come through

443

as a new column in, in the report.

444

And then we'd get an email to

say, "You've changed the format."

445

And it's like, "No, you changed the

format 'cause you added this extra thing."

446

And so one thing that we were really

careful to do when we moved to the

447

templated version is anything that had

the ability to, to change the order

448

of the columns or the add a column by

itself actually became a nested field.

449

So you would actually have structured

JSON or, or the likes within that

450

field, which can still be read by a

lot of the, the main sort of tools.

451

think like Tableau, Power BI, if

they were, taking it out of Excel

452

and, and putting it into a platform.

453

For the customers that, I guess were more

power users and they had sort of experts

454

behind the scenes, they were taking the

data shares, and then they would mix

455

those data shares with their own data.

456

So, um, for example, we would have

information about what had been stolen.

457

We wouldn't have information

about what had been sold.

458

And so they might want to do a

report that, has those two numbers

459

in it, and so they're getting their

sales data from a different system.

460

They're wanting to take those, tables

that we've given them and join them

461

up against that sort of sales data.

462

So, um, yeah, there was lots of,

possibilities about, if we changed a

463

report when it's custom, because they

were defining the fields, they were

464

getting it in the same order, the

same fields, and just those fields.

465

As soon as it's a template, you have to

think about even the order of the columns.

466

Um, you know, changing a column

heading name will break everything.

467

And so, we ended up getting really

good also at, sort of the, details

468

about the changes to reports.

469

So, giving people warning about

470

Andrea Magnorsky: yeah, I was

471

Kathryn Hempstalk: how we've got-

472

Andrea Magnorsky: if you were

gonna version the reports out, like

473

if that was a way to kinda say,

"Well, you're in version 1.0.235.

474

version two's available.

475

You can basically find the same report,

but you hit this different endpoint, and

476

Kathryn Hempstalk: Yeah, it's

actually very unusual to, to version,

477

I guess, CSV files in that way.

478

Like, I know from a software

engineering perspective, you know,

479

there's different versions of APIs

and, and, bits that you would talk to.

480

what we tended to do was say,

"Hey, we've got this change

481

coming out in a month's time.

482

You know, aim for the start of

the month for, for new change.

483

Um, it's available to you in our, test

systems from now, so if you want to, you

484

know, grab a copy and, see what it might

look like, you can go, go and do that.

485

but also, here's a list of all the changes

that we think are breaking changes or

486

could be breaking changes from your side."

487

You know, a column that might

no longer be, provided into

488

the system, was a typical one.

489

So, a- and we, we ended up doing

that a lot initially, and then as the

490

reports, got, got more comprehensive and

probably a little bit tidier as well.

491

So one of the things we did on that path

is removed all the personally identifiable

492

information out of the reports.

493

Andrea Magnorsky: The PIR

494

Kathryn Hempstalk: you know, when

you're dealing with shoplifting,

495

you've got people's names, you know,

witnesses, victims, and you don't

496

want that to end up in a big file

that, anyone can possibly look at if

497

they've been sent an email with it.

498

You want it to be protected by

the platform and auditable as

499

to who's seen that information.

500

So, the early on r- custom reports,

because people could say what they

501

wanted, we would give them what they

wanted, and when it's a templated

502

version, you would have to have make

some sort of concessions on that.

503

And it turns out, you know, most

of them didn't really want or need

504

that extra identifiable information.

505

It was just convenient to, to have it, and

when you talk to them about it, they would

506

have that realization of, "Oh yeah, if

something bad did happen, you know, gosh,

507

yeah, that's gonna be bad for not just me,

that's gonna have a bad day out of that.

508

It's gonna be a whole lot of other people,

509

Andrea Magnorsky: Yeah, especially

if your market includes, those

510

countries that really like suing

companies as a way of business.

511

Kathryn Hempstalk: Yep

512

Andrea Magnorsky: It's funny, you

mentioned a couple of things that I

513

really would like to hear more about, and,

one of them is, you kinda talked about

514

contracts in different ways as, like,

what I think about as a, as a contract,

515

and between, like, you and the customer

in the CSP file, a- about the APIs.

516

And I was wondering, in this story, you

basically talk about the boundaries of the

517

data system and the software system, and

I kinda am really curious about how you

518

conceptualize and think about in general.

519

'Cause I find that, data people and

software and I don't wanna even give

520

much more brand names than that, I

think we kind of tend to think about

521

them in slightly different ways.

522

And, and, uh, and this is to the detriment

of both of us, because, if we could have

523

something a little bit more standard

that talks about the needs of data

524

and the needs of software engineering,

as in, like, those boundaries are

525

there, you know, like, Clear fences

make, good neighbors, that thing.

526

Kathryn Hempstalk: Yes

527

Andrea Magnorsky: that's not

happening, then it creates confusion

528

and, a lot of extra work, and, well,

some of it, surfaces in your story.

529

So I'd love to hear your take on not

just what happened, because I think

530

you kinda explained some of it, but how

you're thinking about it these days.

531

Kathryn Hempstalk: Yeah.

532

It, it's an interesting one because

I think, most software engineers and

533

data engineers, you're right, are

typically thought of in separate circles.

534

That, the, the data people are busy

looking after databases and indexes and

535

SQL and optimizing it, and yeah, and

then data engineers will be pipelining

536

from that to, you know, getting it

into the, the data warehouse and

537

transforming it and aggregating it.

538

And, I think traditionally a lot

of those aggregations had been done

539

for internal purposes, like, you

know, how many customers do we have?

540

What are those customers

doing in, in the application?

541

And that, that side of things.

542

I think those boundaries are

becoming more and more blurred as

543

the, data warehousing and other

tools become more and more capable.

544

And how we kind of, broached that, at,

at, at Aura was, whilst my team was

545

largely data people and would work with

software people, we actually ended up

546

embedding some of the data people into

the software engineering teams to try

547

and build those relationships, and

get some of the information early of,

548

"Oh, hey, we're changing this database

field, and that's gonna affect the data

549

warehouse downstream," and then all the

reports and anything in the application.

550

And then on the, the flip side, "Hey,

we've made this new aggregation in the,

551

the data warehouse, and now those elements

in the interface can be, you know, so

552

much more, quicker or more interactive."

553

And we actually ended up embedding

the data warehouse, um, results, if

554

you like, into the platform directly.

555

So, um, there's tooling.

556

We used, CubeJS as an interface

layer between, the C# application

557

and the, data warehouse.

558

that meant the software engineers

didn't need to write SQL per se, that

559

they could just say what sort of fields

that they wanted and filters on the

560

data, and Cube would automatically

work out the SQL that was needed.

561

and it was basically querying it

from the data warehouse instead of

562

the transactional database simply

because it was faster, more efficient

563

to do it, that, you know, you've

got a big load of data that's being

564

queried via the data warehouse.

565

The transactional store just wasn't

geared up for those kinds of aggregations.

566

So I think we bridged that, by

putting the teams together and

567

making them feel like they're,

hopefully more cohesive as a group.

568

Andrea Magnorsky: That's

569

Kathryn Hempstalk: it, it's still-

570

Andrea Magnorsky: collaboration pattern.

571

Kathryn Hempstalk: Yeah

572

Andrea Magnorsky: it has the, the, the…

573

It's like how do you taper it off?

574

'Cause you wanna kinda do that

possibly temporarily if you…

575

Yeah, I'm kinda thinking with team

topologist in my head, and I'm

576

thinking, oh, well, if I, if I was

thinking of doing that as a kind of

577

way to build a bridge, at some…

578

It would be like, hey, let's

do this for insert time box.

579

You know, some reasonable time

depending on the problem, N

580

months, you know, N, N weeks.

581

But you kinda say, okay, we'll

start there with this and see.

582

You're gonna evaluate halfway through

and see whether we need more or

583

less time, and then kinda taper

it off, as- so that the risks, and

584

eventually, well, what is the contract?

585

Or what the collaboration model, what

should the collaboration model be?

586

but, do you think that that

would be how, or do- did you

587

wanted to keep this forever?

588

Kathryn Hempstalk: So it actually

depended a bit on the teams.

589

So some of the teams were working

more on, the data, data heavy sort

590

of, tooling, and so they, they would

want a particular expertise in their

591

team, so a data engineer or someone,

with the kind of skill set to pipeline

592

and, really write good decent sort of

SQL queries and how things are joining

593

and the data is modeled appropriately.

594

So they might say, "Hey, I want this

person actually permanently assigned

595

to my team for the, for the long term."

596

That's what's gonna help this team go

fast, is having someone who's got a foot

597

in the camp of software engineering, but

also is part of that, that data team.

598

We also found that there were teams

that were more, sporadic with their,

599

their needs from a data perspective, and

so they might need a machine learning

600

engineer for a couple of months for a

project, and when the project is done,

601

maybe there's some responsibility on

that machine learning person to, you

602

know, keep an eye on what that system is.

603

But it's not the day-to-day build

integration, you know, building

604

the monitoring, the pipelines,

that, that side of things.

605

And so we'd put them together for the

time that was relevant and then, move

606

them on and make sure that the data team

ha- continued those responsibilities.

607

And probably the thing that we didn't do

especially well, at, at least at first,

608

was perhaps having the conversation of

what, what goes with that person when

609

they leave and what stays with the team.

610

Because there was definitely a few

times when we had people who'd been

611

in a team, they've moved out of the

team, and something would break in that

612

system and- No one would know, right?

613

It might be the

614

Andrea Magnorsky: will look

615

Kathryn Hempstalk: customer who

said, "Did you know this, this

616

thing was, was broken," right?

617

And it was because no one had had the

conversation of, okay, what happens?

618

Who's now responsible?

619

Who does it go to?

620

And, what we eventually started to do was

push everything to the product managers,

621

for those software engineering teams

as the first point, and then have the

622

product manager engage with the right

sort of person that, you know, they

623

could be the one to figure out, "Oh, I

think this is a problem with the, the

624

software side," before it gets into the

data pipelines or machine learning piece.

625

Or they would go, "Oh, something's

broken in the algorithm.

626

We'll go straight to the

machine learning person."

627

And I think one of the cool things I,

I particularly found at Oura, and I've

628

had it in other businesses as well, is

it had a really collaborative nature.

629

So, you know, people who weren't an

expert would jump in to help out.

630

usually once you realized that there

was a challenge there, everyone was

631

willing to jump in and help out and

see that thing through to being solved.

632

There was none of this, "Oh, well,

it's not my problem, it's somebody

633

else's problem," um, side of

things, and that was really nice.

634

And I appreciate you don't always get

that in, some companies that you work for,

635

but I've had the pleasure of working in a

company that was really good on that side.

636

Kenny Schwegler: I want to make

one comment because this is a

637

challenge I see, and I've been

in a lot of discussions, right?

638

We have the PO that's usually busy

with the user and the needs, and I

639

have this in my company now as well.

640

I hear it from engineers.

641

But then there's downstream teams, like

the machine learning team or other teams,

642

that are also sort of like stakeholders,

and team topology says you don't want

643

to really have an excess of service

with another team or whatever, right?

644

But, that's like competing goals, and I

think this is what you just mentioned.

645

I just wanted to make

this out as a comment.

646

This is like one of the

challenges that teams face, right?

647

And one thing you notice is pushing

it to the PO, and I think a PO should

648

be much more responsible for their

data as a product a way, right?

649

Knowing like we…

650

This is my data.

651

This is my responsibility as a PO,

and I know there's downstream teams

652

who depend on it, and I also need

to, know and understand that field.

653

So I just wanted to make

that as a general comment.

654

I see that POs are much more busy

with the user needs, like the users

655

itself, but these are also users.

656

You're the ones using data, like the

machine learning team or like downstream

657

teams are also users in a way and that

whole field is, can only be solved by

658

what you mentioned, like the facilitative

aspect, the collaborative aspect.

659

So I like that comment.

660

I, I wanted to make that, uh,

661

Kathryn Hempstalk: Yeah,

no, that's awesome.

662

I, I think one of the things that I been

thinking about in the, I guess, agentic

663

era of, software engineering, where,

you know, you're working with, large

664

language models who might be doing some

of the coding for you, is those skills

665

and collaboration and being able to share

the, the contracts and the boundaries

666

and your expectations actually helps

you immensely with dealing with agents.

667

Because, you know, just telling them

to do a thing doesn't give them the

668

context that they need, and the people

that I've seen do really, really well

669

early out of that sort of new approach to

programming have been the people who've

670

been great at building relationships,

good at, you know, thinking about how

671

they're gonna collaborate with their

peers and what information they need

672

to share to, you know, give that person

the best chance of, succeeding as well

673

Andrea Magnorsky: On that amazing

note, I'm gonna wrap it up for today.

674

Thank you so much, Kat,

for an amazing story.

675

I think I have about 100 questions that I

still want to ask, but, I think we took,

676

way longer than we thought, of your time.

677

So thank you, and see

you next time, friends.

678

Goodbye

679

Kathryn Hempstalk: Most welcome.

680

Thank you