In this episode, Frank La Vigne and Andy Leonard dive deep into the world of developer advocacy and data observability with special guest Chris Cooney from CoreLogix. From discussing the evolution of Microsoft Evangelism to the intricacies of data retention and real-time decision-making, this conversation covers a wide range of topics, including the impact of AI technology, the importance of community in software engineering, and the challenges of connecting with engineers at trade shows.

Join us as we explore the intersection of technology, advocacy, and data engineering in this insightful episode featuring Chris Cooney on "Data Driven".

New Season Means a New Theme Song

Let us know what you think! Don't worry, we will use a shortened version for future shows. We were just so excited to get to Season 8!

Show Notes

03:54 Experienced engineer transitioned into leadership in technology.

07:40 Dan delivers insightful speech on tracking activity.

12:33 Developers must adapt to new technology continually.

13:45 Conference talk success measured by engagement metrics.

17:15 Uncertainty about outcome of video creation.

20:00 The trend in the field is evolving.

25:45 Retain all data, use case-driven storage. Avoid rehydration.

27:11 Core principles: smart data science, streaming architecture.

32:34 Efficient streaming processes allow for easy scaling.

36:16 Instantly triggered alarm blocks malicious IP addresses.

37:22 Enormous architecture demonstrates remarkable data management practice.

42:13 Struggle with learning Arabic dialect using AI.

44:41 Language differences reflect cultural and historical influences.

47:55 Regularly listens to audiobooks, recommends "Team Topologies" and "Team of Teams."

51:32 Data-driven podcast season 8 debut summary prompt.

Mentioned in this episode:

WITI BOGO Deal!

Special Offer: WITI is offering an incredible Buy One, Get One sale on memberships until Labor Day in the US. This is a perfect chance to access valuable networking opportunities, cutting-edge resources, and career advancement perks. Exclusive Discount Code: Use the coupon code DATADRIVEN at checkout to take advantage of this special offer. Whether you’re aiming to elevate your career or support a fellow tech enthusiast, now is the perfect time to join WITI. Visit WITI.com to grab your BOGO membership before it’s too late!

Hello, data aficionados and tech enthusiast.

Welcome to the first episode of season 8 of the data driven

podcast. I'm your host, Bailey, your delightful

AI guide through the fascinating world of technology and data.

Now, before we dive into the data laden depth of today's

episode, we've got something rather special for you. Brace

yourselves for our new theme song, hot off the silicon press,

entirely AI generated. Yes, even our theme

music has joined the generative AI revolution.

So plug in your headphones, turn up the volume and let's give

it a listen.

From trends to bikes, we light up your nights

with inside

it up loud. Let's get this party started.

Well, what do you think? A symphony of zeros and

ones, or should we stick to human composers? Feel

free to drop your thoughts on our socials. Today's episode

is a treat. We're joined by the brilliant Chris Cooney,

a maestro in the realms of data, observability in production

systems, and developer advocacy. We'll be delving into the

intricacies of keeping an eye on your systems, the art of data

observability, and why developer advocacy is crucial in

today's tech landscape. So grab your favorite

cuppa, get comfortable, and let's get data driven.

Hello, and welcome to Data Driven, the podcast where we explore the

emergent fields of AI, data science, and

machine learning, and, of course, data engineering, because without data engineers,

the world would stop revolving. And with me is Andy

Leonard, my favoritest data engineer in the world. How is that for

a new intro, Andy, for season 8? I like it, Frank, and

welcome to season 8. Cool. Cool. Yeah. So

I'm gonna tie in our guests, at least geography,

with the theme of this season 8. And my promise to our listeners

and viewers, is that we will not disappoint people like

Game of Thrones season 8 did, and,

our guest is nodding. And, as folks know,

a lot of, Game of Thrones was filmed in and around Northern Ireland,

where oddly enough, as as the coincidence would have it, I have a a

family history there going back, well, 2 generations from

me. But, our guest today is,

Chris Cooney, who is a software engineer,

SRE principal engineer, and he is

now the head of developer advocacy at Coralogix.

Hopefully I pronounced that right. And he's worked on everything from embedded

systems, for controlling industrial battery

units on the UK power grid,

and to payment processing systems, that process,

up to a 1000000000 a year, and just helping

shape technology strategy, with 100 of millions

worth of cloud and on premise infrastructure. Welcome to the show,

Chris. Thank Thank you very much for having me. I'm super excited to be here.

Awesome. Awesome. We had a great chat in the

virtual green room, but, so tell tell us a little bit about yourself.

You're currently located in, Northern Ireland now, but you

you your entire career, from what I can tell, spans kind of the UK.

And you were at I see on your LinkedIn that you did one

time work at Sainsbury's, which if memory serves, because I used to live in Germany

and I would go to UK quite a bit, that's a grocery chain?

Yes. It's a it's a retailer. Yeah. It's the 2nd largest retailer in the

UK. So my background, I've been engineering

now for, oh, gosh, like, 11 years now, I

think. I think I'm starting to get some gray hairs now. The,

the the the gist of it is started out the application level Java,

back then, then moved into React engineering because I realized

I couldn't design a front end to save my life. And then, I

real after a while, I thought, well, how do I run this thing? So then

I moved started moving into the DevOps and SRE side of things and started

reading a lot about SRE developer experience,

the what was emerging then, which was platform engineering, which was slowly

slowly coming to the forefront, and then ended up

replatforming, got really into the whole Kubernetes

space, and then started thinking really heavily about, well, how do I keep track

of all this stuff? And after a few roles here and there, I went into

engineering leadership. And, while I was in leadership for a

few years as the principal engineer, I was responsible for there was, like, 22

different teams. They all had very different portfolios. And I was just

trying to, understand what each of them were trying to achieve and then

maximize that outcome. I realized very

quickly that the thing we were lacking desperately was,

observability. And so when I started to look around,

I spoke with Ariel Asarath, the CEO of CoreLogicix.

And, if you have ever or will ever speak to him, you'll know he's

a very compelling guy. And I pretty much signed up signed

on the dot. I was like, let's do this for a 100%. I'm I'm ready

to go. And then I've been there for 2 years now, started out as the

advocate. And we the past few years have really just been about

understanding what the what the what advocacy looks like both in the observability

space, but also at CoreLogic specifically. We have a pretty good picture of that

in terms of content, tone, speaking, that kind of thing.

We're hiring more, and we're growing that advocacy function

to be much more proactive and outreaching and have a lot more

fun with it. And so, yeah. And and and,

obviously, the reason for being so excited doing this podcast is

the thing with observability. You know, most people will say, what's the problem with observability?

Most often, the the thing that comes back is cost. People say, like, it's just

so expensive. But, actually, like, if you deconstruct that

slightly, really, the problem is data, volume, and how to manage it

because that's what's driving the cost. And so then you go back and you analyze

the behaviors and understand what decisions we made as an industry that drove

that. And all of that has got me really into the data space now to

try and understand better what we can do both as engineers and

consumers to make sure that we feel we're getting the

best return on investment for every dollar we spend on our observability.

So it's fundamentally a data problem, and that's why this podcast was so

exciting for me. Oh, awesome. Awesome. So, I'm gonna go

was excited to talk to someone who's in developer advocacy. I worked

in, event what we used to be called evangelism, for

Microsoft for a number of years, but it's the same it's the same thing. Yes.

What's fascinating is I think

one of the key challenges of developer advocacy or evangelism

was always the traceability. Now there's a number of problems.

Right? But but one of them is the traceability of

a a particular advocates activity

to actual revenue. I think that was always that was always kind

of a something that I know Microsoft struggled with.

Right? Yeah. And I remember when they when, you know, you know, this

was well, 2011, I interviewed and they were like, do you have any questions for

me? I was like, yeah. Well, how was an individual evangelist tracked?

Right? Yeah. And the answer to that was a really good

stump speech by the hiring manager. His name is Dan. Shout out

to Dan if you're listening. Was he goes, you know, you can't

how do you track someone's individual activity? Right? At this time,

Vista was still fresh in people's minds because, well, if you're a Windows evangelist, you

can't blame you on Vista. Can't play

Vista on you. Right? Okay. So how do you how do you kind of work

that? And how do you how do you message that? How do you how do

you track that from, you know, inception to purchase?

And it was a very interesting kind of thing. I never look at it the

same way again because on the outside, it looks like the

funniest job in the world, and it is a fun job, but but there is

a business side of it too. And, Andy and I actually

got involved in developer community, and that's actually how we met through the

efforts of another, evangelist. His name is

Andrew. Okay. So, like, we got involved in developer

community. At the time, the focus of Microsoft Evangelism was was

building up a kind of a community of folks, user groups, and

such. And it's fascinating to see how evangelism

has evolved over the years. Yeah. Do you wanna talk a little bit about

that? Because it's not. User groups are still important. They're still a thing, but it's

actually, I think, a little different now. So you you mentioned writing and things like

that. What what what makes up a the the the

offerings of evangelism these days? Sure. So I think the early days

of evangelism were where this reputation for it being remarkably

fun job came from. Because as is often the case, people are quite

idealistic when something's new. So when evangelism came about, it was

like, we need to be able to talk to engineers. What do

engineers like to talk about tech? We're gonna hire people that talk about tech, and

that's their job. And that was that was the line of thinking, and then that

spawned, you know, YouTube series and and and various different

media that that that came from that. And then what happened over time is, like,

as as it often does, economics came into play and was like, well, how what's

the return on investment? Like, great. This guy just gave a talk on some very

deep corner of Prometheus querying or something, or

or, you know, Python. And what do we

get for that? And so now what's what's what is still actually

happening, I think, is, organizations are

transitioning away from a model where evangelists just talk about open source

tech, and they just, quote, unquote, connect with engineers.

And there's something of a pipeline in place there

where they they they are if you like, the toppest of the top

funnel. They're right right at the top of the funnel. They are talking about

open source technology or something that people will find interesting in general,

And their goal is to talk to people often. Now when I speak

to our advocates, the the the goal is to talk to people who might be

interested in the product. They're not there to sell the product. They're not even there

to market the product. They're just there to talk to those people and be like,

okay. You're all interested in the observability space. For example, for

me, here's a talk that's useful. Regardless of whether you're gonna buy

CoreLogix or not, here's what's useful for you from an engineering perspective.

And, normally, what I'd like to do is 90%

independently valuable information, 10%,

sort of, the by the way, I work at CoreLogix. This

might solve some of the problems you've seen. So I give a talk, all over

the place, which is kind of all about

the summary of it is basically, like, how you can

get the most value out of your observability spend. That's the kind of the the

essence of it. I do not sell CoreLogic at any point

throughout this talk. Yeah. But at the end of the talk, I say, by the

way, if you're looking to cost optimize, CoreLogic has some great tools. It might solve

a problem for you. And so I'm I'm I'm the pre product

marketing person if you like. And that's kind of of ad where

where I see advocacy at the moment. Yeah. That's one of the parts of it.

And then there's the community building and the content creation, that kind of thing. And

in terms of content creation, you know, the word content

really has only been a a descriptor for the kind of media that

we're creating, for a few years now. It's a relatively new word,

And I think that the ability to create compelling content

has is is becoming further and further to the foreground because the

impact of one good video is, like,

you know, it could be huge. It could be nothing. Could 2 views or you

could get a million. You don't really know. And a lot of companies want to

throw those dice. And I think that's where it's interesting. Well,

you're fascinated by your by your that's okay. I'm fascinated by your

division of, topics. They're 90% just trying to

help Yeah. And, you know, share with people from your

experience, which is impressive. It's vast. And, you

know, doing someone who's who's come through the

field as the field has matured. And I I think that's very easy to

overlook. Yeah. You know, how you have to as a developer,

especially, how you have to continually shift gears to grow with

the technology as the new technology shows up and understand,

okay. This is the problem it's trying to solve now. And that gives you

a little bit of a, you know, a projection maybe or

or a feel for maybe where it's going, a little bit of predictive

analytics type thought. Sure. And then

knowing that the company that you're working with, that the the

platforms y'all are building and and making available

also address some of the current problems and also some of the predictive

ones. I like that 9, 8, 10 mix. I I find that makes

for a compelling talk. And when when I'm sitting in one of those

talks, I don't feel like I'm being sold something. Yeah. And

No. And I I understand that. Yeah. What's funny is that,

I've given the talks, and sometimes salespeople say that was very clever. And I said,

what was very clever? And they say, you know, you you, I I

you know, the sales pitch was there, but I didn't feel like I was being

sold to. I was like, because you weren't being sold to. I wasn't I'm not

trying to sell my job. I don't I'm not measured on sales or revenue.

I don't get, commission. Nothing. Yeah. And so,

what I measured on is actually, I have a personal KPI, which I really like,

which is if I give a talk at a conference, I measure the number of

people that come to the the the booth that we have at the conference and

just ask questions about the talk, whether they're interested buyers or not. In other words

Nice. Like, people are interested and engaged enough that they wanna come and

talk to us afterwards. Like, whether they're buying or not is not my problem.

But what is my problem is did I give a decent talk? And that that's

usually a pretty good metric because the ones that I felt have gone really, really

well, and then no one's come to the booth. And I've spoke you know, there's

been drinks afterwards, whatever, and they were, oh, you were on stage. What were you

talking about again? Like, no remembrance. And the ones that I felt were kind of

okay, 30, 40, 50 people come to the booth. I I I just

I'm I'm struggling to field all the questions, and then I have the drinks afterwards.

And everyone's, oh, I saw your talk. It was brilliant. There were you know? So

my own sense of how good a talk was is pretty off, basically,

last time in the audio. So so so that that that's a nice little KPI

for me personally. Yeah. But, yeah, the com yeah. Sorry. Go

ahead. That's okay. I was just saying that's a great KPI. And I just like

one quick follow-up, and I'll shut up and let Frank talk. Does anybody

ever you mentioned the person that said, you know, that was clever. Does anybody

in the audience ever provide feedback and say, oh, it was a big sales pitch.

You ever get that? I so sometimes when the talks are very,

very short, you know, I only have, like, sort of 10 minute lightning

talks are different because I think it's a different head space. But when it's, like,

a 10 minute, 15 minute slot. In

the past, not so much these days, I'm better at it now, but a mistake

I've made in the past is allowing that ratio to slip from 90:10 to

kind of 50:50. And I think that I've come off and they've gone,

you know, it would have been nice if you focused on the open source stuff

a bit more. And so but I think that 90:10 ratio,

even the people that say at the end, they noticed that that's 10% as a

sales pitch. They go, well, yeah, but 90%, no one was trying to sell me

anything. So whatever. That's fine. You know? Yeah. And I

think as well, you just so much of it, like, engineers

try 2 things that engineers are often, like, numb to now is

1, recruiters and 2, salespeople because they are just

everywhere. You know? Right. It's very difficult. So to not

if you if you seem like a recruiter or a salesperson, you just become part

of the background noise of an engineer's life. And so so

my goal is always to make it really clear. I actually when people talk to

me at the booth, I'm like, look, we could really solve your problems. And I

can see them, like, go, oh god. Here we go. And I hold it quick.

Just been saying, I don't work in sales. I'm not sales. Don't worry. This I

have no best interest in you. Yeah. And so that's that I

think from a from a from a sort of logical business perspective,

that's a really powerful trust connection that you have with with engineers that,

that businesses can use to get their message out. That's the economic side of it.

And from a, from a just sort of human

perspective, that's the really fun part of the job. It's just like finding

something you're passionate about, being in a room with people who are also passionate about

it, and then having a giant conversation. You know, and if you're a bit

extroverted like I am, then being the center of attention is is good as well.

It's always nice. So, yeah, it's it's it's it's wonderful.

No. I think you said it best when you said it's the top of the

funnel. Right? The tippy top of the funnel. Right? Because Yeah. One of the ways

I've I I I've been I've had a

number of roles that kind of dance between I've always been on

the more evangelism side of sales when I've been in Sales Works. Right? Where I

do create content. Because I gotta create content anywhere. You kinda just once you get

the content creators bug, you have it. You know what I mean? Yes. And

and to your point, you know, I don't when I make a video, I don't

know. Is it gonna get 5 views or is it gonna get 5,000? I don't

know. I really don't know. I haven't been able to figure out that, you know,

try as I might, try as I try try to figure out some kind of

algorithm for it. I I haven't really cracked that, and

it's it's also true for, for speeches too. Like, I totally get

it. But I think the best way that I've used to explain

evangelism to non believers or advocacy to non believers is

that, you know, salespeople go to the person in the corner office and ask for

the deal. Mhmm. Evangelists

warm up the crowd. Right? So they basically

cubicles knock on the door of the corner office saying, hey. We need this. So

that way when the salesperson does land, it's a warmer it's a

warmer call. And and that's something that,

I think that when you're talking to salespeople, they do get that notion of,

like, you're the warm up act. Right? Yeah. Certainly, all the salespeople

in in CoreLogix were a bit confused by advocacy initially.

And then when you know, one of the things you'll know you'll

you'll all know this from working at trade shows and and conferences,

there's the you have people at the booths, and they're trying to snag people and

be like, hey. Yeah. What are you interested in? And then suddenly that changes to

30 or 40 people just turning up to the booth to ask questions. It's like

it's so much easier for them. And then they kinda they okay. Right. No. I

I get why this is good for me. Like, I the leads come to me.

I don't have to, you know, I don't have to go out and find them.

So that so so that that yeah. I think the salespeople are certainly the ones

that I've worked with now get the value. But I I I

still think we're only doing sort of 30,

40% of of advocacy and evangelism and let us hire more people, and

hopefully, we can do the other side of community building and all that sort of

stuff, which is all gonna be new for the company as well. Yeah. And,

you know, it's it's very avant garde that your company is even doing it. Right?

Like, you know, it's it's it's it's something that even large companies don't

really do. Obviously, FANG, to a certain extent,

does. And I posit that because of Guy Kawasaki's work in the

eighties. Okay. Right? He was the he was the first person that I'm

aware of that had the title of evangelist. Right. Apple I think a

big part of why the Macintosh has this cult following, and

and I know all the Mac lovers go, don't call us a cult. Right? I

know. I have a I have a MacBook. Right? But,

it's because of his work, like, you know, 30, 40 years ago. Right? I I

don't think that's a coincidence. And I think that,

when Microsoft wanted to have the uptake of dot net,

they reinvested heavily in in evangelism. And I

think we saw the the fruits of that. So you no. I mean, it's it's

one of those things where I've noticed that it kinda ebbs and flows. Right? It's

like, you know, there's a the tide is up, everybody's all into

it, and then the tide kinda goes away and people kinda Yeah. Down on it

and they reorganize and they get rid of it. But I think that it is

it's a field that I think has still maturing, I think, to your point.

Right? Like, you know, because how do you define it? So there was

something going around on LinkedIn where a lot of folks

who were advocates tagged in an avocado emoji on

this. Yes. Yeah. Yeah. Could you explain that? Because I only know part of the

story. Do you It's an Amazon thing. Oh, okay.

Because Advocate and Avocado kind of sound similar.

As far as I'm aware, that's the that's the depth of it. I mean, I

think it was changed with Amazon, and then, people just started to do

it. I did it for a while. I think I don't know if I still

got it now, actually. But, but, yeah, it was just a nice thing to

signal you as an advocate. And the advocacy community, I will say,

is is singularly wonderful. They are such lovely

people and, you know, all of them, I've been with competitor

companies. You know, sat on on the front row ready ready to go up. And

if someone's nervous, like, because no one's we're not we're not competing

with each other. We're just giving talks. Everyone's super friendly, relaxed, talk blah blah blah.

You know, everyone's like, the and the the way the community is,

that that is one of you know, and you make friends pretty much instantly,

with with with all these different people at the booth. So you suddenly you go

to one conference. You just make 5 new friends. And, you know, 2 of them

are have been doing this for 20 years, so they're just fountains of absolute

wisdom. You know? And so it's the community actually makes

a massive difference for it. And that avocado thing I might it's just a sing

a signal, like, you know, we're we're in we're in the this weird little club

with you. And and, you know, what is ostensibly a very strange niche, a

soft niche of of software engineering. My boss, when he, when

Ariel interviewed me, he said, congratulations, Chris. You're an extroverted software

engineer. You're one of the 3. And I was like, yeah. Because it's

it's, you know, it's it's a it's a weird space.

And not that you have to be extroverted to be fair, but it it does

kind of help, you know, going out and meeting people and shaking

200 hands and and answering all the questions and being on stage, it does help

if you get energy from that, within which I definitely

definitely do, thankfully. Well, let's

talk about the data part because I think that's the part that,

so and I think it I think it also dovetails too. Right? Because there's certain

you mentioned k p a KPI that you have. Right? And there there's clearly data

that you have to collect as part of just being an evangelist to show your

your and I'm sorry. I keep using the word evangelist. That's just the old

habit. It's fine. Fire department. But,

there's clearly data you track. But what's the core core core what's the

core of CoreLogix? That sounds really weird. But, like, what's the core of the business?

You mentioned observability. Right? And observability implies data. So talk to me

about this. You mentioned Prometheus. So

explain a little bit. I'm I'm giving you, like, a ton of questions together. Sorry.

No. It's okay. Yeah. Yeah. No. I get it. So You're drinking Diet Coke. I

have the monster energy drinks. I see. Okay. Right. Yeah. Okay.

Something of a higher caffeine content, I imagine. So Right. The

the the the gist of CoreLogix is this, full stack observability,

processing logs, metrics, and traces, and we have a security offering.

The the the essence of the platform can be distilled into,

taking decent data science principles around how to manage your data

and then baking them into observability. One of the

problems that we so we actually did some investigation a few years ago,

about how people are actually using their observability data. And we

found some really, really surprising statistics. Like, for example,

99% of index data in in a in a in an elastic

search, or an open search cluster, for example, is never

queried. So 99% of it is is ingested into the cluster and then just never

touched. And and it's one of those things where it's like, wait a minute. Like,

why? And then we realized, well, it's not queried, but they're maybe

visualizing in dashboards. So I was, okay. So that's

interesting. Then another one was, a large volume of data. I can't remember

the exact number, but a large volume of data is only interesting for a single

number in the log. This is primarily focused on logs. It's

just one log has a latency field, and that's what people really care about. The

rest of it is just noise. For example, the

the average historical query length is a week, so people

tend to query back a week and not much further. The retention

time is between 2 4 weeks, on average. So at

the very least, we retain in high performance storage for twice as long as we

need to, on average in 4 weeks, the upper end

of that. When I do this at conferences, I say, hands up if you

retain for a week, 2 weeks, 3 weeks, 4 weeks. And inevitably, there's

always some poor guy with his hand up who's like, you know, we're on

3 months at this point, and I'm like, can you just tell me how long

you retained? Was that a year? And I'm like, okay. We would've been here for

a while. So so so what we found in the

industry was that there was this perception of we need to send less

data. We need to, we need to we need to do more with less, I

suppose. And what we found was, no, the problem isn't

data volume as such. We have this data because it has a

purpose. It has a function. It has a reason for being. We don't just generate

the data for the fun of it. We data we generate data because our infrastructure's

becoming more complex. Our, the the

solutions that we have to come up with micros microservices. You know, at some point,

somebody had, you know, decided that they have a 200 user a month CMS, and

they were like, we need 15 microservices to run this thing. I don't know when

that, like, gripped the the popular consciousness, but it has become a big thing

now. So all these practices drive up the

data. We need the data. So instead of deciding what data we're

gonna chop and change and and get rid of and that kind of thing, Let's

say, how would we go about retaining everything? How would we go about keeping all

of our data? And then the question was, how do you manage the cost? The

answer is to be use case driven with how the data is stored and how

the data is accessed. So as I mentioned, some data

is only useful for a single number converted into a metric. Metrics

are a fraction of the the cost to retain. So there you go.

You just you just shaved off the vast majority of that document's overhead,

convert it into a Prometheus metric or a Victoria metrics or whatever works.

But the thing is, we don't wanna we wanna retain data for

a really long time because archiving and rehydrating is both

expensive and painful. So instead of that, let's break the rehydration

paradigm and directly query the data in your archive with no

dependency on indexing or reindexing.

Some of our data is queried. Some of our data is never used. Some of

our data we ingest just because we might need it in the future. Okay.

So make different levels of pricing based on the use case for that

data. And so that was the that was the, the

pinnacle of the sort of the the the what what was distilled

down into what became the CoreLogic stream architecture. So I

find myself talking about that the most. There's a tonne of, you know, tracing

and, database APM and serverless APM, and

q Kubernetes and so on in the platform. But, ostensibly, at its

very, very core, it's just some smart data science and data engineering

principles, including the fact that we built everything on a streaming based architecture.

So, rather than doing everything in, like, a batch mode or

triggering everything from a database, we use Kafka and Kafka streams to process the

data and make decisions in flight rather than sort of

various intermediary storages that have IO bottlenecks and all the rest of

it. Makes it very scalable and also very efficient to run.

It's less which means less expensive for us to run the platform, less expensive for

the customer. So we drive enormous cost savings as well based on that.

So all those things, there's a lot of information there, but all those things kind

of, like, come together to form this platform that gives you

all the traditional observability things that you want and some really, really

advanced stuff. And it's also possible to query

your archive, which is actually an s three bucket or cloud storage in your own

account, directly from things like

you can query your metrics directly from your archive. You can query your

logs directly from your archive, your traces, but, also, you can

visualize stuff in dashboards alongside index stuff

from your archive. So it's it's all about and the the difference is

you wait maybe 2 seconds instead of a sub second rendering time. You know?

It's not much of a cost, but a massive, massive saving opportunity. And that's

that's kind of where we've gone. Now we're just building this wonderful platform

that's really, really fully featured and really mature. So interesting. Yeah.

Absolutely. When it terms the, like, the the the

record keeping of logs in terms of history, is there

a difference between different industries? Like, there's certain regulatory things.

Is it come up in digital forensics, I would assume?

Yes. So we have some really fun use cases. We have some companies that are

using us. So as I said, the core of CoreLogic is

just great data engineering principles. That's what our architecture is all about.

And, that's illustrated best by the fact that we've

got some companies that are sending, you know, maybe

40 or 50 terabytes of data through us a day. And they're using

us purely as a transformation and analytics engine. They don't index any of

their data. It actually goes back to the s three bucket. In this case, an

s three bucket in their account. And we

transform process and analyze that data. This particular company is in the

financial industry. So heavy, heavy regulation. Everything

has to be retained. Everything has to be accessible. They get regular audits where

they have to demonstrate. Like, someone will come in and be like, tell me what

happened with this user on the 1st June

2022, you know, to have to have the data that far back. If they wanted

to do that with any other platform, things would get very pricey very

quickly, and we offer that archive query at no additional cost. It doesn't

cost anything per query. It's purely based on gigabytes ingested.

So they get basically enormous retention, enormous

scalability without needing to pay the cost. And that that digital

forensics thing is a really, really interesting part because people think of

observability as a purely DevOps or resiliency kind

of discipline. Yeah. It's not. It it's it's all about understanding what you've got,

data observability, measuring the freshness, distribution volume, and so on

of your data, is within that observability realm. And it's

possible to do it in the same platform provided that platform has been built with

those decent data engineering principles in mind. And so, yes,

the data forensics piece, analytics sort

of, you know, aggregations across 1,000,000,000 and 1,000,000,000 of

documents to get some really some insights that would otherwise be hidden away. Yeah. Those

are all things that we run into regularly. I I think it's really

interesting that you took the approach of streaming first.

And I could see several advantages to doing that. Often, people architect

data engineering, even near real time analytics,

collection to be more focused on

historical and being able to move the window around. And

then they have an option that they kinda bolt on, And,

you know, they're just trying to pound on the server

Yep. Pull the server, throw it into a loop where every second is

grabbing the most current data. And if you do it

that way, I mean, you're you all are nodding your heads. You you get it.

It's you've gotta architect from the ground up, I

mean, you know, to have that perform at all. But if you take

that approach of reading real time data, to

start with, that's your focus. It's very easy, I would think, or

easier to expand the window and say, no. Let's look at

the last hour, the last day. Yeah. And and more more

than that as well. The it's not just so so the streaming architecture,

basically, as far as, as far as I

see it, then I I I this may be more of an opinion than an

engineering position as such, but the streaming gives you the ability

to make decisions in flight, and it makes it really easy to

perform transformations. And, as long as you're disciplined

about minimizing side effects, it also unlocks, the

scalability. But it's the things that sit around

the stream, which is so we have this concept of, source,

stream, and sync. And, Yoni Farrin, the CTO of the

company, and I believe his team kind of came up with this. But a source

is like, you know, a a a the data source, the actual database, that kind

of thing. The stream is obviously the Kafka stream that's processing it, and the sync

is where the data ends up eventually. It's a pretty normal thing in data science

anyway. But the the key thing here is that

none of our, sort of in stream transformations are reading

from external sources. All the data is loaded first, pushed into

the stream, and then written. And what that means is that the

individual streaming processes can be horizontally scaled very, very easily.

We can so what it means is that we could scale up and scale down

very, very effectively and very efficiently. That's part of that cost

optimization on our side, but it means that that's a thing that,

I think is really important to highlight because people often I worked in Sainsbury's for

a long time and I saw many Kafka projects appear and fail.

And it was because while they built the Kafka someone built some

Kafka solution, They constrained it left, right, and center

with IO and database sort of operations. And so when it came to

scaling it, it was like, well, this isn't the bottleneck isn't this. The bottleneck

is the 30 or 40 database and Redis clusters that are sitting around this

thing. So it's not just the foresight to build a streaming architecture. It's a foresight

to minimize side effects in your architecture, and that makes it much

more scalable and, ironically, much more observable as well because the the

everything's just an in out transformation. Makes it a lot easier to monitor and maintain.

Cool. That's a lot that's really

clever, actually. Like, the more you the more you explain it, I'm like, that's

brilliant. Like, what? Why didn't I think of that? Just like I

tell you, like, these the part of the reason I joined this company was because

I had a call with I had a call with Yoni, the CTO. I had

called Ariel, the CEO, and, I left both of those

calls feeling like I was just like a chimp banging 2 rocks together. And

I was like, right. Okay. This is where I need to be. I need to

this is this is how I level up now. So and it's I've learned so

much since joining, not just as a I love the architecture.

Sorry. I didn't mean to cut you off. I I love the architecture, and I

love the patterns in both of those kind of appeals. I mean, but, you know,

I I would imagine that the IO on the data

store is critical. That's gotta be mission critical at that

point because you you wanna catch all the data as fast as possible.

And at the same time, you wanna be able to serve that data as fast

as possible. It's, you know, that would be the

bottleneck, I would I would imagine, and I'm sure y'all have solved

that at the, you know, tuning the hardware up. You

know? Yes. Precisely. Yeah. It's it's it's it's it's a comp it's essentially,

partially an architectural decision and partially an engine partially an engineering decision. So

the architectural decision is, for example, because of the

way Kafka works, we can parallelize a great deal of the processing. So to give

you an idea, a typical observability platform, one of the large

vendors now, you can expect an alarm to fire.

Let's say a metric crosses a threshold. It's anywhere from

2 to 4 minutes on the on the upper end before the alarm actually

fires. Wow. That's a big delay. That's huge. It's enormous. You

know? And if if you imagine you're a financial trading company, like, 4

minutes is, like, is is huge. You know? If you're if it's a security

alarm, that's massive. And that's because what they do is they

ingest the data, store it, normalize it, and index it. And then they trigger a

series of processes. And this normalizing and indexing process only gets worse with

time. That's an infinitely scaling dataset. What we do is instead is we do all

the analytics upfront, and then if you want, we store it

in high performance index storage at the end. Otherwise, it just goes straight to the

s three bucket archive or to the, to the cloud storage in your

account. Okay. So I love the flexibility of that. I I do. I could

see how that serves the architecture because, you know, often, 4 minutes

being notified 4 minutes after, a hack starts, it's

over. It's useless. It's some so we have this type of alarm called an

immediately type alarm, and I run some use cases with it where,

a an IP address appears in the logs from AWS,

web application firewall. We have an ability to

enrich data as it comes in, so you can we actually look at the top

15 threat databases when an IP address comes in and we say, oh,

is this malicious? And then the alarm was triggering every time there was a malicious

IP, and the immediately type alarm was triggering it under a second. So it was,

like, firing pretty much instantly. And in within 18 seconds,

it had invoked a Lambda function, which,

updated the IP set in the in the WAF instance.

And so it's like sub WAF itself does that in

in over a minute. So the data was leaving AWS.

Oh, that CoreLogic is in Amazon, actually. So so but but it was it was

going to CoreLogic's being processed along this extremely complex

pipeline. The alarm was firing, and the Lambda function was being

invoked in a 5th of the time it was taking for WAF to even measure

the data in the first place. That's huge. Yeah. It's enormous. And

that's that's like that it to me, that's the best demonstration of our architecture.

That's like Yeah. That that that's one of the reasons why it's just so remarkable.

And and it's also it's a it's like I say, it's a lesson in data

engineering. And when I when I go to conferences and talk about cost

optimization, I find myself talking more and more and more about just

good practice of managing your data, as opposed to

any secret observability magic source. It's always the way, isn't it? Like, a

relatively new industry has a lot to learn from the adjacent industries and almost

never does. It takes a lot of time to that passion. So yeah.

No. That's very true. Like, there's a lot of good lessons to learn from,

like you said, adjacent industries because we a lot a lot of the

problems we're facing are not necessarily new. The context is definitely

new, but the Yeah. Laws of physics are persistently

stubborn. Yes. Precisely. Precisely.

Interesting. Well, are we at that point, Frank, where we're ready to

pull up the questions? Okay. Alright. So we'll ask the pre

canned questions. They're none of them are real brain teasers.

Right? They're we're not we're not trying to be Mike Wallace, and I don't even

know if anyone get that reference these days. Okay.

But you can always ask chat gbd who

Mike Wallace was. I will.

But in order to save the environment from hitting those GPUs, all these

people hitting the GPUs, Mike Wallace was a journalist for

TV show called 60 Minutes who was notorious for, you know,

if there was a corrupt executive or politician, he was notorious for, like, sneaking up

on him and asking them, like, while they're, like, getting groceries or whatever. Maybe even

at Sainsbury's. Who knows? And saying, like, you know, hey. You know,

why did you embezzle $5,000,000? Like, what's going

on? It's a bad day if you walked into your office and

Mike Wallace was waiting. Just sat there.

We could do with some of that now. I think we should resurrect that practice

in the UK. We need more of my gualas', I think. Yeah. I think you're

right about that. Yeah. So the first question

is, how did you find your way into data? Did data find you,

or did you find data? Data very much

found me. I as I said before, like, I was primarily interested in

SRE, and then I realized, like, then observability became a real

passion of mine. And then I realized, well, what is the biggest problem? Oh, god.

It's data. Okay. This is okay. So that's that's how I ended up here.

So what would you say is the favorite part of your current job?

Every time I finish a

talk and someone comes to the booth and says, you know, I've been really struggling

that for a long time, and that thing you said there has just given me

a really great idea. And I that is, like,

amazing. That's just just, like, direct dopamine to me because it's, like,

engineer to engineer having a conversation and just solving problems is, yeah,

brilliant. So, yeah, right now, that's that's the the best.

Nice. We have 3 complete sentences.

When I'm not working, I enjoy blank.

Oh, god. Is this just like what I do in my in my normal Yeah.

In your spare time. Yeah. Yeah. Yeah. Spare time. So I

Yeah. Well, whatever that whatever spare time. Yeah. Yeah. I I

try and, I try and spend as much time with my daughter as possible. She's

10 months old and she's, just crawling and everything else. So

just absolutely fantastic. And, yeah, it's a level of, like,

it's I'm tired of doing way wrong, and I'm a bit stressed, but, it's

like the the the peaks of joy are just, like, unbelievably, like,

incomparable. As one, I like philosophy, and I like, I play

some guitar as well. So I've been kind of getting into that more as well.

Cool. Excellent. Yeah. Dad, Frank, and I are both

dads. We set up. And I'll tell you as a dad

of, daughters, I have 3 daughters and 2

sons. And I was I'm the oldest I'm I'm

the oldest of, like, 5 boys, so I had no clue

about either daughters, sisters, anything like that. But, yeah,

it was it's an it's an amazing experience. Yeah.

And watching them grow up and my baby girl is in college right now

at Virginia Tech. And so you I saw that

look, and it's like, yeah. That's gonna feel like about a month from

now when you look back when she gets there.

Yeah. For sure. The days are long, but the years are short. That's the Yeah.

Yeah. Yeah. I have 3 boys. 14 is the oldest,

9 18 months. So yeah.

It's pretty chaotic. That's great to hear that. Our our second

complete sentence, I think the coolest thing in technology today

is fun. I hate giving this

answer because I feel like it's everyone's gonna say the same thing, but it's it's

it's gotta be Gen AI. Right? Like, right now, it's it's it's just

like some of the thing. I so I, I'm half, Arab, and I've been learning

Arabic for the past few years. And the other day, one one of the problems

with Arabic is that, online, you basically find all the

lessons in classical Arabic, but you'll never find lessons in what's called the dialect.

So, like, I'm Jordanian. So the Jordanian dialect is is is it's not

completely different. It's very similar, but it's there's lots of details that are different.

And I went on to the new chat gpt model the other day, you know,

4 o, and I, opened it up and said, I hit the

the headphone thing to have a conversation. And I said in Arabic. I said, I

want you to speak in Jordanian dialect to me, and I don't want you to

speak in in in classical Arabic. And it responded

perfectly. And I thought Wow. Like Wow. I don't I don't even

know what resource you went to to get this information. But,

like, it that you know? And it was it was the the accent, like,

wasn't right. Obviously, the the release in the new voice model, I imagine that might

change things a bit. But just the grammar, the inflection, the

phrasing, the slang was was all that. And I thought this is

like, I can't find this information. I've tried I've

tried for for a long time to find this information in written form or

in just a one place where I can go to get a, like, a full

sense of Jordanian dialect, and it just doesn't exist. So so

I just thought, wow. That's that's pretty pretty crazy.

And that's yeah. So that's probably the coolest thing I've seen in a while technology

wise. I'm from the villages, so I should say that. Yeah. But Well yeah. But

no. But I mean, language learning is one of those things where I I did

take a couple of courses on Arabic, and the teacher was from

Syria. Yeah. But everybody in the class were

as was this was Jersey. Yeah. The state, not the island.

And most of the most of the other, participants were from

Egypt, and they would always argue over how to say things. Yeah.

And as a non native speaker, I'm kinda like

completely lost. Oh, mate. You know? And they

you even like, one of the things that's we we say Arabic in English, and

we what we're essentially describing is a whole family of languages.

Right. There are lots of Egyptians who if you ask them, do you speak

Arabic? They say no. I speak Egyptian. And if are you an Arab? No. I'm

Egyptian. And they because there's a 40% of the Egyptian

language is is influenced by Coptic language.

So and and then if you go to, like, Morocco and you ask them, do

you speak Arabic? They say, no. I speak a Tarija. Like, the I speak a

and and and it's like everything, you know, even even, like, Lebanese, they say the

Canaanites rather than rather than Arabs. Now there's obviously

politics and things mixed up in that, but it's also embedded in the language as

well. The Lebanese will use a lot of French, for example,

because of the history there, but also because of the

it differentiates them slightly from from their neighbors. You know, in Jordan, they use

a lot more classical Arabic in how they speak, but also use a lot of

English in how they speak as well. Again, because of the history, but also, again,

to differentiate themselves. They say different work, different letters differently, and so

on. So so when we say Arabic,

you know, the the general perception is that it's it's a one monolithic language,

and, actually, it's a very, very wide, the glossier of languages.

And, yeah, I've I've very rarely seen resources acknowledge

that. I've never seen a resource ever automatically

talk back to me in Jordanian Arabic. I've never ever seen that ever. So,

yeah, remarkable. Mostly remarkable. That is cool. Yeah. So the last

complete sentence, I look forward to the day when I

can use technology to link.

I am really looking forward to where

the VR headsets get to. I worked in VR for, like, a

week, and it was the Quest 2, so there was

no HD pass through. But it was still pretty great, and I

I I had to stop every few hours because I was getting headaches and

stuff, and I had to kind of, like, take it easy. And

we I really just feel like we've got the tracking sorted, we've got the

processing sorted. Almost everything's there. It's just

the comfort factor that we have to work on. That's just gonna come with

lighter and lighter components until it yeah. It was like the visor being made by

a company called the MERST right now, and they haven't done a demo of it

yet. I'm very excited for that demo, but, I I'm I'm

really looking forward to the day when I can just have an empty desk and

and just put the visor on and I've got all my screens and everything that

I that that's that's really what I'm excited about.

Yeah. Absolutely. Like, the the sweat that you build up around

the is is is is really a limiting factor. That

Yeah. That a little bit of motion sickness, but it depends on the game you

play, I found. Yeah. Yeah. For sure. So

share something different about yourself. But do you remember it is a family

podcast? Something different about myself. I

I boxed. Oh, really? Yeah. Not professionally

or anything. Just, I go into it for the fitness to start with

and then just got more and more obsessed with it.

Greatly enjoyed it, and, and I only stopped recently because I moved

house, so there's no gym near me. But I

absolutely loved it, met some really wonderful people, and learned a great deal about myself.

That's a quote fight club. You don't really know yourself until you get punched in

the face. And and you really do find out a lot

about yourself. So yeah. Excellent.

So Audible is a sponsor of Data Driven. Do you do

audiobooks? And if so, can you recommend a good one? I do audiobooks

all the time. Nice. Whenever I run, whenever I walk the dog, it's either

a podcast or an audiobook almost guaranteed.

Recommend a good book. I'll do 2, one tech and one non tech just because

life's too short for only tech books. For

tech, I will say Team Topologies, I think is one of

the books that impacted me the most when I read it and made me think

about how people collaborate with one another. That or Team of Teams by,

Stanley McChrystal is just, oh, amazing as well.

Both of those are kind of in the tech realm. Non tech, I would say

The Master and, Margarita by Mikhail Bulgakov was,

it's just I read it and I wanted to I read it. I've read about

5 times and I just loved it every single time. I think it's just wonderful.

And, also kind of morbid book Crime and Punishment by

Dostoevsky, I just think is, again, I read it

and I was greatly impacted by it whenever I read it. I thought it was

wonderful. So, yeah, kind of too dusty Russian writers there. But,

yeah, those are the, those are the recommendations. Well, that's the beauty of the

time we live in. Like, you could get an audio book on just about

anything. Anything. And listen to it just about anywhere. Yeah.

That's the thing. That's the thing. You know, when time is limited, I can take

the dog out for a walk, audiobook, and I'm I get to

hear about the, you know, some crazy you

know, some detail of the Napoleonic war while my dog's chasing a stick on the

beach. You know, it's it's like that level of convenience. You can't beat it. You

really can't beat it. And on the as well. Like, no. It's wonderful. For

sure. If you go to the data driven book, which I think

Andy is testing right now, see if it's a DNS that works. It's working.

You know, for 2 tech guys, we we have a lot of infrastructure challenges. We

definitely need some SRE. What do they say about Shoemaker's

children?

Where can people find more about you and CoreLogix?

So the easy one for CoreLogix, corelogix.com.

And there's, there's a whole host of different ways you can learn about

CoreLogix, video courses, Mostly me, so I

apologize if you're sick of the sound of my voice because it's gonna be a

lot more of it, I'm afraid. And then, we're on

YouTube. We have a blog and all sorts. That's

CoreLogic. For me, personally, I'm mostly on LinkedIn these days. I

have other social media, but I don't really use it. So, if

you just search LinkedIn for Chris Cooney, I think I'm the top one. I think

that's my accolade now. But if not, if I'm not coming up, Chris Cooney, CoreLogic

is guy. So that For me, you were the 3rd. For you, me, you were

the 3rd one. Yeah. But, I mean, you're first in my mind and my heart,

of course. But, I when I saw it when I saw the picture

of the guy that first come up, I'm like, doesn't look like you. And

she, I don't think you live in Miami. So

so I wish. It would be nice, but I'll yeah.

Miami is awesome in the winter. In the summer, it

requires a certain type of person. That's all I'll say.

K. I I I struggle in heat above 25 degrees Celsius, so I imagine

Yeah. Miami is probably not for you. No. I agree. Not for you in the

summer. Yeah.

But I love Miami. Big shout out to, Miami. I know a lot of folks

who live there and love it. Noel and Bill are the first two, 305

people that comes to mind. Of course, Pitbull, the the musician, but that's

a that's a more of an insight, I think, to my musical taste that

people want.

And with that, will it barely finish the show? And just like

that, we've reached the end of the first episode of season 8 of the

data driven podcast. We've traversed the fascinating

terrain of data, observability in production systems, and

developer advocacy, all thanks to the insightful Chris Cooney.

A big thank you to Chris for sharing his expertise and making the

complex sound oh so simple. Now, a quick note on our

new theme song. We know it's a bit lengthy, but fear

not, we'll be trimming it down for future episodes. Your

listening experience is our top priority after all.

As always, we'd love to hear your thoughts on today's episode.

Feel free to reach out on our social media channels or leave a comment on

our website. Don't forget to subscribe, rate,

and review us on your favourite podcast platform. Until next

time, stay curious, stay data driven and remember

the future is data shaped. Cheers.