Not all engineers are equally invested in understanding what our software is actually doing - should they be and why? Is building a successful software product more than just a technical exercise in observability? How do we get more insights into the business direction of our product? Can we only truly test performance in production and what does that look like?
I talk to Stephen Townsend from Squared Up and the Slight Reliability podcast, all about SLOs and SREs and observability platforms and patterns and trying to predict what our customers want in the future.
This was a really in-depth and wide-ranging discussion and a lot of fun to put together. Hope you enjoy!
NOTES
Sooner, Safer, Happier - Jonathan Smart
Observability Engineering - Charity Majors et al
Accelerate - Nicole Forsgren et al
Slight Reliability with Joey Hendricks (performance testing in production)
QUOTES
[03:19] "I've just come from a massive enterprise before and there was a massive disconnection between the day-to-day work or what's, what's on the minds of the people in those engineering teams versus the actual customers and the business." [ST]
[13:55] "It's a measure of how well you can understand and explain any state that your system can get into no matter how novel or bizarre" [ST quoting "Observability Engineering"]
[14:30] "now we're serverless and containers and all these ephemeral objects and well, they're being scattered everywhere. It's just hard to know what's happening" [ST]
[18:13] "investment in the customer experience for some is greater than for others" [RB]
[23:08] "Hey we are making technology changes. How does it change these other things?" [ST]
[25:37] "it's very weird to come from an engineering background and be talking about all these business objectives and things, but it's great" [ST]
[27:46] "if you're doing software observability, it needs to be thoughtful and continual. Like it can't be. We've set it up, we are done." [ST]
[34:33] "The most successful companies are doing that right now, and they're doing, they're following the best practices that we read about in all these great books, accelerate, et cetera, and DevOps handbook" [RB]
Hi, this is Richard Bown.
Richard Bown:Welcome to loving legacy.
Richard Bown:Now at the beginning of this episode, there's a slight bit of swearing, but I
Richard Bown:hope you'll agree to add some context.
Richard Bown:Let's get on with the show.
Stephen Townsend:The first panel's like this is how to draw an owl, and
Stephen Townsend:it's like a little bit of a squiggle.
Stephen Townsend:It's like the back of the wing and then a little bit more
Stephen Townsend:maybe the, the tip of the thing.
Stephen Townsend:And then it's the whole drawn picture of owl and it's like, step
Stephen Townsend:one, do this, step two, do this.
Stephen Townsend:Step three, draw the fucking owl, you know, That's what SLOs are like.
Stephen Townsend:And that's what everyone focuses on.
Stephen Townsend:But the other side of it was, it seems to me we need a way in the digital
Stephen Townsend:era to say we wanna achieve something.
Stephen Townsend:We wanna create something for, for customers and get some outcome,
Stephen Townsend:a business outcome, a customer outcome, whatever it is, and we go
Stephen Townsend:and build this stuff and then we never check whether we achieve that.
Richard Bown:hello and welcome to the Loving Legacy Podcast.
Richard Bown:I'm joined this week by Stephen Townsend and we're gonna talk about observability.
Richard Bown:Stephen, thanks for joining me.
Stephen Townsend:No problem.
Stephen Townsend:It's a pleasure to be here.
Richard Bown:So perhaps before we dive into the subject, which is
Richard Bown:fantastic, I think, um, maybe you could tell everybody a little bit
Richard Bown:about who you are and what you do.
Stephen Townsend:Sure.
Stephen Townsend:So I am currently a developer advocate.
Stephen Townsend:Just going back to the rest of my career, which has been about 15
Stephen Townsend:years now, I guess maybe a bit more.
Stephen Townsend:I studied computer science, uh, and then I.
Stephen Townsend:Decided during that time that I couldn't possibly work in technology.
Stephen Townsend:So I auditioned for, uh, , which is the New Zealand Drama School.
Stephen Townsend:And, and, and so I trained for three years to be a professional actor.
Stephen Townsend:I thought, this is it.
Stephen Townsend:This is the dream.
Stephen Townsend:I'm gonna, this is gonna be my life.
Stephen Townsend:And it was like a pretty awesome experience.
Stephen Townsend:But I'm from New Zealand, small country, there's not a lot of acting
Stephen Townsend:work . So , I started looking for technology work after three and a half.
Stephen Townsend:Having finished my degree, uh, no one would hire me, uh, and as a
Stephen Townsend:developer, but I managed to somehow land this job as a junior performance
Stephen Townsend:tester, uh, for a consulting company.
Stephen Townsend:And
Richard Bown:Yeah.
Stephen Townsend:then I did, uh, performance testing and
Stephen Townsend:performance engineering for 13 years
Stephen Townsend:. And then about a year and a half ago I
Stephen Townsend:site reliability, engineering, which or service reliability engineering,
Stephen Townsend:depending on, what your context is
Stephen Townsend:and then about three months ago, I joined a company called Square Up who do
Stephen Townsend:unified dashboarding and visualization, um, as a developer advocate.
Stephen Townsend:That's been pretty fantastic.
Richard Bown:Sounds good.
Richard Bown:You have to deal with developers a lot, presumably as well, cuz
Richard Bown:you're an advocate for them.
Stephen Townsend:It's true.
Stephen Townsend:Yeah.
Stephen Townsend:So that's, that would be, um, the community, um, our customers.
Stephen Townsend:But I'm also sort of working internally, almost like a consultant, uh, to sort of
Stephen Townsend:uplift their own observability and ways of working and sort of eat our own dog food.
Stephen Townsend:We say all these fancy concierges and let's try, let's
Stephen Townsend:try them internally and see.
Richard Bown:Fantastic.
Richard Bown:Cause I, I, I've found, so my background is development and I've kind of come
Richard Bown:through that path and done various bits and bobs and, but a large part
Richard Bown:of that, and the bits that I've always found the most enjoying are
Richard Bown:the customer touchpoints, basically.
Richard Bown:How do you find that juxtaposition of requirements?
Richard Bown:Because there's.
Richard Bown:Push for features.
Richard Bown:This is the whole point of it.
Richard Bown:I suppose there's a push for features, but observability and the monitoring
Richard Bown:and understanding of where we are actually are with our products.
Richard Bown:Sometimes it gets overlooked, doesn't it?
Stephen Townsend:So in terms of like developers or any, anyone
Stephen Townsend:working in technology even.
Stephen Townsend:Operations in the operations world.
Stephen Townsend:Uh, I've just came from a massive enterprise before and there was a massive
Stephen Townsend:disconnection between the day-to-day work or what's, what's on the minds of the
Stephen Townsend:people in those engineering teams versus the actual customers and the business.
Stephen Townsend:You know, that this is all for the massive disconnect because there's so
Stephen Townsend:many layers of bureaucracy, uh, that the, the idea of product management
Stephen Townsend:is supposed to be this conduit.
Stephen Townsend:Or product ownership that sort of brings it together.
Stephen Townsend:But it, it's in practice in very big organizations, it's,
Stephen Townsend:it seems very hard to do.
Stephen Townsend:It's, it's a bit different now working for a small product company where it's so
Stephen Townsend:small that we are all together and it's more about Trying to figure out, you know,
Stephen Townsend:how things are landing with customers.
Stephen Townsend:You can interview them of course, but then how do you get like bigger
Stephen Townsend:picture data about that that's actually meaningful and actually accurate that
Stephen Townsend:helps you direct your where to go next.
Stephen Townsend:So yeah, I think it's a, it's a pretty challenging thing, but, and what you're
Stephen Townsend:saying about, you know, tying it back to observability, um, observability
Stephen Townsend:is the buzzword obviously, and I'm sure we can go into lots of detail.
Stephen Townsend:What it is and what it isn't.
Stephen Townsend:Recently, I, I've always had this idea in my head, I didn't look up
Stephen Townsend:a definition and no one's sort of told me, this is observability,
Stephen Townsend:you know, this is what it is.
Stephen Townsend:So I just thought, okay, what does it mean?
Stephen Townsend:It must mean how observable something is.
Stephen Townsend:Okay, what does that mean?
Stephen Townsend:And it sort of, it seemed to me it was two things, but most people, when
Stephen Townsend:you think of one of them, and that is, it's the ability to sort of underst.
Stephen Townsend:The internal state of a complex system in order to better understand
Stephen Townsend:to understand behavior remediate issues, um, look for opportunities
Stephen Townsend:to optimize and tune and improve it.
Stephen Townsend:And that's what everyone focuses on.
Stephen Townsend:You know, massive, massive community and industry around there.
Stephen Townsend:But the other side of it was, it seems to me we need a.
Stephen Townsend:way in the digital era to say, we want to, we wanna achieve something.
Stephen Townsend:We wanna create something for, for customers and get some outcome,
Stephen Townsend:a business outcome, a customer outcome, whatever it is, and we go
Stephen Townsend:and build this stuff and then we never check whether we achieve that.
Stephen Townsend:There's this total missing element of we built this thing, did we actually
Stephen Townsend:achieve what we set out to do?
Stephen Townsend:And that seems like the most important thing in the entire world.
Stephen Townsend:to me when it comes to software and.
Stephen Townsend:I mean, there's this book I read recently, sooner, safer, happier by
Stephen Townsend:Jonathan Smart, and it talks about the digital era we work in and how
Stephen Townsend:we work with unknowable unknowns.
Stephen Townsend:So it's basically chaos.
Stephen Townsend:There is no way that anyone can go, here's an idea that's implemented
Stephen Townsend:and this is what will happen.
Stephen Townsend:It's literally impossible because of the complexity of the technology
Stephen Townsend:and business and indu industry.
Stephen Townsend:The only way we can actually carve a path forward is try something experi,
Stephen Townsend:you know, do a little bit of, so, Put it in front of real customers and business.
Stephen Townsend:See what happens and adjust that is literally the only
Stephen Townsend:way that you can succeed.
Richard Bown:Yeah, it's kind of like, sorry to interrupt, interrupt, but
Richard Bown:you sparked something with me there.
Richard Bown:It's almost like the Heisenberg uncertainty principle, isn't it?
Richard Bown:It's kind of like the act of observation changes what you're
Richard Bown:trying to do in the first place.
Richard Bown:You know, sometimes, because you kind of get to a point where, I mean even with
Richard Bown:a throw away feature implementation or something like that, you can still lose
Richard Bown:sight of what you're trying to achieve by the time it's gone into production,
Richard Bown:let alone be observed, you know?
Stephen Townsend:Mm.
Stephen Townsend:I think good observability.
Stephen Townsend:What you said, uh, change, you know, the, the act of observing
Stephen Townsend:it should change something.
Stephen Townsend:Otherwise, what's the point if we just observe it passively and go, Hey, look
Stephen Townsend:at that, it's crashing and burning, or, Hey, no one's using this thing and
Stephen Townsend:they're not doing anything about it.
Richard Bown:Yeah, well, exactly.
Richard Bown:Even get as far as a metric on a dashboard somewhere.
Richard Bown:It just gets ignored.
Richard Bown:So how do you close that loop, I suppose, is the thing.
Richard Bown:So that is, is that the, the kind of, the, the bit that's interests you the
Richard Bown:most, the kind of closing that loop?
Richard Bown:And, and if so, how would you go about that
Richard Bown:. Stephen Townsend: I think
Richard Bown:Yeah.
Richard Bown:And of, of course, just having observability, just answering those
Richard Bown:questions isn't the, the answer in itself.
Richard Bown:It's, it's kind of about that relationship between different
Richard Bown:stakeholders that make up.
Richard Bown:An organization or you know, they're putting it all together.
Richard Bown:So having that clear line of communication between the leadership
Richard Bown:and the executives who have these business goals they want to achieve
Richard Bown:between, you know, the products owners or managers that, those teams who want
Richard Bown:particular outcome for customers maybe.
Richard Bown:And then the engineers who want things to run smoothly and uh, and
Richard Bown:also want to do, to be able to.
Richard Bown:Quickly and reliably as well as operated effectively, uh, and combining all
Richard Bown:these things in a meaningful way.
Richard Bown:And I kind of say as I, as I say that, it sounds like this very
Richard Bown:challenging thing because each group of stakeholders speaks a different
Richard Bown:language and, but there has to be some common ground, some sort of sweet spot
Richard Bown:. Richard Bown: I mean, and I see,
Richard Bown:across industry completely.
Richard Bown:So how do you find, yeah, do you find that it does change according
Richard Bown:to the scale of the company?
Richard Bown:Because I've seen this happen even in quite small companies as well,
Richard Bown:the kind of, it's kind of like there's an industry standard that
Richard Bown:we need to do things a certain way.
Richard Bown:So for example, looking at the Dora metrics, which are, I think
Richard Bown:everyone kind of agrees are a great baseline for understanding.
Richard Bown:How well, yeah.
Richard Bown:How can we define this in a nice way, , because it's not
Richard Bown:about delivering customer value.
Richard Bown:It's kind of, it's, it's, it's really about understanding.
Richard Bown:We're not breaking stuff.
Richard Bown:That's the way I see it in some ways, that we're not breaking things as,
Richard Bown:as an experience and we're doing things at a reasonably fast pace.
Richard Bown:It's not really about value, I would say.
Richard Bown:What do you think?
Stephen Townsend:Kind of to me says that there's like a machine which produces.
Stephen Townsend:Outcomes.
Stephen Townsend:It's supposedly creating outcomes for business or customer, and it tells
Stephen Townsend:you how well all that machine is.
Stephen Townsend:It doesn't actually necessarily tell you whether the outcomes are
Stephen Townsend:happening, you know what I mean?
Stephen Townsend:It's just that the machine's humming along nicely, which, you know, to
Stephen Townsend:be fair, is something that, that knowledge is missing from most
Stephen Townsend:organizations that I've ever worked with.
Stephen Townsend:So it, it's a good starting point
Richard Bown:definitely.
Richard Bown:Yeah.
Richard Bown:And have you been involved in the implementation of DORA
Richard Bown:metrics at all or parts of them?
Stephen Townsend:So we are looking at them at square up internally.
Stephen Townsend:It's really interesting.
Stephen Townsend:There's actually someone else leading it and I'm just sort of
Stephen Townsend:assisting, but it's, I love it.
Stephen Townsend:It's really exciting to me to act, actually do it and, and
Stephen Townsend:have these things like, here's a metric, what does that mean?
Stephen Townsend:What does lead time mean to us?
Stephen Townsend:Like exactly from what point to what point.
Stephen Townsend:And then even if we measure that, what does it actually tell us?
Stephen Townsend:You know, how does it help us as an organization?
Stephen Townsend:You know what I think is interesting about trying to implement DORA metrics
Stephen Townsend:is you'll do a thing and you'll get a number and then you'll start
Stephen Townsend:questioning it and it just opens up really interesting questions.
Stephen Townsend:For example, you know,
Richard Bown:you know,
Stephen Townsend:we were talking about, um, meantime to recovery, I think it
Stephen Townsend:was today, and we had this, I dunno, we had this some figure with like five
Stephen Townsend:days and then we started saying, what does that, what does that include?
Stephen Townsend:Like what.
Stephen Townsend:Uh, bugs or incidences and does that include, and, and
Stephen Townsend:do we need to classify that?
Stephen Townsend:And, and then, oh, it actually turns out that there's two different kinds
Stephen Townsend:of things and you know, it, it just starts really interesting conversations
Stephen Townsend:about what does it mean and do we need to do something about it.
Stephen Townsend:Hmm.
Richard Bown:Yeah, for sure.
Richard Bown:And.
Richard Bown:I think it's great that, well, I find relief in it because I've been involved in
Richard Bown:meetings previously in previous jobs where no one knows what to define to begin with.
Richard Bown:So like what is useful lead time is kind of, is generally
Richard Bown:agreed, is a, is a good one.
Richard Bown:Uh, so that's the time from when a, um, either an incident or a feature
Richard Bown:is created in, like, for example, JIRA to the time that it actually arrives
Richard Bown:in production in front of a customer.
Richard Bown:As, as my understanding of it.
Richard Bown:There's, there's, there's debate over.
Richard Bown:Whether or not it's behind a feature flag.
Richard Bown:So is it in production?
Richard Bown:Is it actually released to a customer?
Richard Bown:That kind of stuff.
Richard Bown:And I think that's, that's fair enough.
Richard Bown:However, I found there's been so much debate over what the metrics are
Richard Bown:in the first place, that sometimes the implementation gets completely
Richard Bown:way further down the line, so you don't even get to that part.
Richard Bown:But yeah, as you say it's great to have those conversations, but when do those
Richard Bown:conversations start to be actually useful to the customer, I suppose?
Stephen Townsend:Mm.
Stephen Townsend:That is a very good question.
Stephen Townsend:Maybe that, maybe like, I, like I was saying, they, they probably, maybe they
Stephen Townsend:don't, they, what they do is they, they get rid of, All of this other mess,
Stephen Townsend:which is going to get in the way of technology teams being able to deliver
Stephen Townsend:outcomes to let them focus on that.
Stephen Townsend:But then there's some other, there's another world around, you know, stuff I
Stephen Townsend:don't know, maybe not that much about, around how do we either understand what.
Stephen Townsend:A business or customer group wants or needs, and then so we
Stephen Townsend:can articulate and deliver that.
Stephen Townsend:Or if it's a new product, maybe without a clear set of customers already, you
Stephen Townsend:know, to understand what the market needs or what, what the need is or
Stephen Townsend:opportunity, what's gonna excite people, delight people, or whatever.
Stephen Townsend:Yeah, that's kind of beyond my knowledge at the moment, how to do that.
Stephen Townsend:I don't know.
Stephen Townsend:I don't have the
Richard Bown:Yeah, I think, well, I think if you can answer that, then you,
Richard Bown:just to be honest, we've solved a lot of.
Richard Bown:The question of why we create software, something, which kind of
Richard Bown:bothers me, , because the name of this podcast is Loving Legacy and I'd
Richard Bown:like to give Legacy a second chance.
Richard Bown:Really like, kind of understand why we are so keen to rebuild stuff.
Richard Bown:And is that just a, uh, an itch that either a founder or an owner or a senior
Richard Bown:person coming into a certain role wants, they're kind of throw their weight around
Richard Bown:a little bit and create something new?
Richard Bown:Or is it just the case of the, the thing isn't fit for purpose anymore?
Richard Bown:The, the legacy system or the way that we're working stuff.
Richard Bown:Various other reasons, for example, that is laden with tech desk
Richard Bown:or the technology stack's too old or, or something like that.
Richard Bown:However, yeah, I, I'm really interested to, to dive down a little bit more into
Richard Bown:what observability actually is and how it's implemented like these, these days
Richard Bown:in inverted coms, because I'm from the old school where it's like diving into logs.
Richard Bown:So you would have a server log somewhere and you'd be SSHing to a machine and then
Richard Bown:grepping it and all that kind of stuff.
Richard Bown:What does observability mean for you these days and how that,
Richard Bown:how is that kind of approached?
Stephen Townsend:Okay.
Stephen Townsend:I think, well, I, it's definitely worth starting with the sort of
Stephen Townsend:classical, not the classical definition.
Stephen Townsend:There's no, it's not classical.
Stephen Townsend:The, the software observability definitions.
Stephen Townsend:I'm actually reading, I'm only a few chapters into observability
Stephen Townsend:engineering at the moment.
Stephen Townsend:Um, by Charity Majors, Liz Form Jones and George Miranda.
Stephen Townsend:Yeah, I think I, I think I wrote down how to describe it.
Stephen Townsend:It's a measure of how well you can understand and explain any state
Stephen Townsend:that your system can get into no matter how novel or bizarre.
Stephen Townsend:So that, that's the, that's the summarized, um, sort of
Stephen Townsend:definition from the book.
Stephen Townsend:And I tend to agree with it.
Stephen Townsend:And I think it's about, you know, I think there's been a lot of use of
Stephen Townsend:logs and, and metrics in, in the past.
Stephen Townsend:My entire career has been filled with it.
Stephen Townsend:I guess what's kind of new is distributed tracing because for, for better or
Stephen Townsend:worse, you know, maybe legacy was good in a way cause it was simpler.
Stephen Townsend:now we're serverless and containers and all these ephemeral objects and well,
Stephen Townsend:they're being scattered everywhere.
Stephen Townsend:It's just hard to know what's happening.
Stephen Townsend:You know, a user click to button.
Stephen Townsend:Where did it go and, and, and, and what was the path like?
Stephen Townsend:Did it just go to one thing or did it call something a hundred
Stephen Townsend:thousand times before it moved on?
Stephen Townsend:Like, how, what's going on?
Stephen Townsend:So, distributed tracing, I think is a good example of a technique which is
Stephen Townsend:trying to solve, you know, answer that, that question like what's happening.
Stephen Townsend:So that when something goes wrong and the customers, not getting
Stephen Townsend:what getting outcomes they want, then you can answer those questions.
Stephen Townsend:And I think that is by far, the biggest focus of what people are talking
Stephen Townsend:about in the world of observability.
Stephen Townsend:And don't get me wrong, I think that's hugely valuable.
Stephen Townsend:It's very valuable.
Stephen Townsend:I remember, you know, not even, not that long ago, like three or four years ago,
Stephen Townsend:trying to do exactly the same thing that distributed tracing does, but without
Stephen Townsend:distributed traces, just by getting logs from all these different servers all over.
Stephen Townsend:using, you know, uh, trace studies and then writing my own, like Python
Stephen Townsend:or power show codes, like pull it all together into data and then plotted in
Stephen Townsend:a visualization tool to see the, the, the flow chart of how, what's happening.
Stephen Townsend:You know, I, it's like, it's not new, but it's just becoming, it,
Stephen Townsend:it's becoming not a manual task That took me, uh, used to take me
Stephen Townsend:a week maybe to get one flow chart.
Stephen Townsend:Now you should theoretically be able to get it whenever you want it, but it's
Stephen Townsend:that it's the sitting out, which is hard.
Stephen Townsend:So that's,
Richard Bown:Yeah.
Richard Bown:So sorry to jump in there as well.
Richard Bown:The.
Richard Bown:And one thing that I noticed a few years ago was there was a great pop, uh, things
Richard Bown:like Splunk, for example, and Datadog like, um, proprietary solutions where
Richard Bown:you can just point stuff at a, a lake of data as they call it, and, uh, just
Richard Bown:ingest it all and then no one looks at it or you, or you write the wrong report.
Richard Bown:So it's kind of a, there's a.
Richard Bown:And then of course the bills start coming in and people go, oh my God, I
Richard Bown:can't afford to pay for this anymore.
Richard Bown:It's ridiculous.
Richard Bown:So, so would have you seen Yeah.
Richard Bown:Any kind of migration from the kind of one-stop shops, uh, solutions like that
Richard Bown:towards more kind of homegrown, we need to get back to the actual data and then Yeah.
Richard Bown:Seeing it as a kind of its own, um, how would you say it may
Richard Bown:be almost department within an organization to focus on observability
Richard Bown:alongside SRE or as part of sre?
Stephen Townsend:I do definitely see two things.
Stephen Townsend:In my last role, we, we were using Splunk.
Stephen Townsend:Um, there's a lot of Splunk and also New Relic, uh, sort of from more on
Stephen Townsend:application performance monitoring perspective, but even the little.
Stephen Townsend:You know, we're doing, you know, a fair amount of work with those things.
Stephen Townsend:They, but the bills were so high already that we were talking about, well,
Stephen Townsend:maybe we should pull some of the stuff OnPrem and use an open source tool.
Stephen Townsend:And our SR SRE team was building some of that stuff, you
Stephen Townsend:know, standing up on-premise.
Stephen Townsend:This is free for you to use Prometheus.
Stephen Townsend:We'll help you set it up and configure it.
Stephen Townsend:And you can use this Grafana instance to, you know, it's, it's different than what
Stephen Townsend:you had in Splunk, but it's free and, and we can make it scalable, you know.
Richard Bown:Observability is product, which is the the buzzword.
Richard Bown:Of course, everything has to be a product these days.
Stephen Townsend:Yeah, yeah.
Stephen Townsend:The platform.
Stephen Townsend:Oh, goodness.
Stephen Townsend:I guess it's a bit of an anti-pattern, but I, I mean, I think it is happening.
Stephen Townsend:Yeah.
Stephen Townsend:Well, we've always had sort of IT operations teams who seem to be
Stephen Townsend:the, the only people who cared.
Stephen Townsend:Did anything in the word of monitoring,
Richard Bown:That's it.
Richard Bown:That's it.
Richard Bown:You know, this whole DevOps nonsense, I would say nonsense, but it is,
Richard Bown:um, comes about because people want to know what's going on.
Richard Bown:You know, it's like a developer or an operations, but, or whoever
Richard Bown:in between whatever color of their skin or sex or whatever, you know.
Richard Bown:Some people are interested in this stuff and some people just want to turn up to
Richard Bown:a day job and do a day job kind of thing, and that's nothing wrong with that.
Richard Bown:It's just the investment in the product itself.
Richard Bown:I've used the P word again, investment in the, in the, um,
Richard Bown:in the customer experience.
Richard Bown:I suppose you'd say.
Richard Bown:The, the end customer experience for some is greater than for others, and
Richard Bown:that drives us or them to create these or look into creation of these tools.
Richard Bown:and that's why these amazing platforms like, um, Splunk
Richard Bown:do turn up because Cuz yeah.
Richard Bown:When it, when something like that does arrive and you can, oh, hang
Richard Bown:on a sec, this makes things a lot easier for me to, to investigate
Richard Bown:what was going on previously.
Richard Bown:Brilliant.
Richard Bown:Um, but I, yeah, I would say that the, the, the, the questioning thing has
Richard Bown:always been there in, in certain people.
Richard Bown:How do we make that, how do we bring that out?
Richard Bown:I dunno.
Richard Bown:Give 'em a difficult problem.
Stephen Townsend:I was talking to someone else recently, uh,
Stephen Townsend:who's an observability person.
Stephen Townsend:I think there's, there are observability people, I guess now I'm one of them.
Stephen Townsend:Um, and I asked the question how, you know, how do you get.
Stephen Townsend:You know, engineers just who aren't their job isn't to observe things.
Stephen Townsend:Uh, understanding the value of this and interested in it.
Stephen Townsend:And one of the answers is, well, we need to enforce you build it, you run it.
Stephen Townsend:What do you think of that, ? I'm curious.
Stephen Townsend:What do you think of You build it, you run it as a concept.
Richard Bown:Uh, yeah, I think as a concept it's okay.
Richard Bown:Although I just have so many battle scars from early on in my career as well.
Richard Bown:A long time ago when I was.
Richard Bown:Doing that, building it and running it or, and actually being on
Richard Bown:real page duty with a real pager.
Richard Bown:And I think that's how old it was.
Richard Bown:Um, and just being burnt out from that.
Richard Bown:So it's fine as long as you manage that, I think in the right way.
Richard Bown:So yeah, you can, you can do this.
Richard Bown:It depends on a lot of, yeah, like so many answers.
Richard Bown:It depends, you know, depends on a lot of things, but I like
Richard Bown:the concept of it for sure.
Richard Bown:And also as a big fan of, of.
Richard Bown:Books like, um, the Phoenix Project Unicorn Project and, and that kind of,
Richard Bown:that storytelling vibe is strong in, in the way that I feel that you mo you, yeah.
Richard Bown:You most connect with a bit of software that you're working on.
Richard Bown:You, you do have to put a little piece of yourself in there, you know, put a bit
Richard Bown:of skin in the game, as they say, um, to, to, to make it a reality, a good reality.
Richard Bown:And I don't think there's anything wrong with that, but you have
Richard Bown:to approach it and manage it.
Stephen Townsend:you asked me earlier what my def, what my definition
Stephen Townsend:of observability was, and I talked about the software side of it.
Stephen Townsend:I guess, you know, I'm thinking right now about a lot about this because I think
Stephen Townsend:the other side of it for me is I guess what you traditionally would call BI.
Stephen Townsend:Um, but when I think of bi , I think of like quite large teams
Stephen Townsend:of people doing a bunch of stuff.
Stephen Townsend:Huge amounts of data.
Stephen Townsend:Data, which.
Stephen Townsend:And like monthly and quarterly reports, you know what I mean?
Stephen Townsend:As opposed to, I think that there's something in that, in that world
Stephen Townsend:where we want to understand, you know, we've got a business objective.
Stephen Townsend:We want to gain 500,000 customers this year.
Stephen Townsend:Are we on track?
Stephen Townsend:How are we tracking?
Stephen Townsend:And I wanna see that in real time constantly.
Stephen Townsend:Like, just like we would monitor anything else.
Stephen Townsend:So that's the other side of it for me, is, is, is monitoring.
Stephen Townsend:Monitoring as if it's like a technology system business and.
Richard Bown:Yeah.
Stephen Townsend:Like I said, so, so, so once you're doing that,
Stephen Townsend:then you can say, are we on track?
Stephen Townsend:Do we need to pivot or adjust, or, you know, what we're
Stephen Townsend:doing to get their feedback?
Stephen Townsend:And I, I, I don't know.
Stephen Townsend:That's what I'm, I'm experimenting with right now at
Stephen Townsend:Squared Up as well internally.
Richard Bown:And I think, um, from, yeah, what I've seen that does, that is
Richard Bown:done typically by sales teams as well.
Richard Bown:So if, if sales teams have a, have a target, of course they
Richard Bown:will say, well, I'll, we on track.
Richard Bown:They want to know, you know, and how they, they're less concerned
Richard Bown:about what they're trying to sell.
Richard Bown:I would say a lot typically, I, I wanna be unfair to, to sales people
Richard Bown:generally, they're less concerned about what they're trying to sell.
Richard Bown:They're trying to achieve their targets.
Richard Bown:Through some method, and they will track this stuff because they have
Richard Bown:vested interest in making sure that those, those targets are met for
Richard Bown:the, obviously for their own benefit and for the company's benefit.
Richard Bown:So even in quite small companies, I've seen even scale-ups, for example,
Richard Bown:maybe 200 people, uh, I've seen.
Richard Bown:And get to a point where there's such a disconnect between the engineering
Richard Bown:output and what's going on in the, in the, in the product and the, the sales or
Richard Bown:the business side, as you might see it.
Richard Bown:That, that, yeah, it'll all be held in Excel.
Richard Bown:So typically the, and that's pretty much the answer for a lot of
Richard Bown:business pro problems, accounting and finance, all in Excel.
Richard Bown:Um, and sales or in Excel or something else, you know, or there, maybe there'll
Richard Bown:be Salesforce or some something that will help a CRM perhaps, but typically
Richard Bown:it'll be close, as close to the person as possible under their control.
Richard Bown:Anytime you put a system in between or, or a build process between you
Richard Bown:yourself and having that knowledge, it, it just, it kind of, uh,
Richard Bown:disappears down into a excel hole over.
Stephen Townsend:I, I guess the third group there is the sort of
Stephen Townsend:product ownership management team who might be look actually looking
Stephen Townsend:at information around the customer.
Stephen Townsend:Like say there's a customer journey that they want people to go, like what's
Stephen Townsend:the drop off rate on each step of it?
Stephen Townsend:And you know, how long are they spending that, that kind of stuff, which is,
Stephen Townsend:it can be really useful knowledge, but, you know, but I think the real
Stephen Townsend:lost, the lost opportunity is, hey, the, we are making technology changes.
Stephen Townsend:How does it change these other things?
Stephen Townsend:Like these, these metrics we're following which matter to us.
Stephen Townsend:, we made this big change.
Stephen Townsend:We expected it to change this thing.
Stephen Townsend:Did it like connecting up those three domains into one
Stephen Townsend:? Richard Bown: So maybe we could
Stephen Townsend:what does the company squared up?
Stephen Townsend:What's your, what does your company do?
Stephen Townsend:Yeah, it, it's basically the idea that, um, I'm sure from my
Stephen Townsend:experience anyway, every organization that I've worked with has these monitoring
Stephen Townsend:tools and usually quite a lot of them.
Stephen Townsend:So one team over here has this tool, that team has another tool and it's
Stephen Townsend:never unified and there's sort of these silos of knowledge and they don't
Stephen Townsend:necessarily share or make it open.
Stephen Townsend:So rather, I think the way, what Squared up does is that you have
Stephen Townsend:this one place that you can.
Stephen Townsend:Almost tells stories by dashboards and visuals, uh, and connect via in real time
Stephen Townsend:using APIs to all these different things and pulling all the data in real time.
Stephen Townsend:It doesn't store the data, it just displays it all and
Stephen Townsend:shows it all in real time.
Stephen Townsend:There's a bit more to it than that.
Stephen Townsend:There does store some metadata so that you can sort of see connections between
Stephen Townsend:things coming from different places.
Stephen Townsend:But yeah, that, that's sort of the general.
Richard Bown:Okay, so I'm, I've got, my first thought is like, so
Richard Bown:do you do historical data as well?
Richard Bown:So if you don't store it, do you handle history in some way.
Stephen Townsend:No.
Stephen Townsend:So you, no.
Stephen Townsend:So, so you'd need to store wherever it's coming from.
Stephen Townsend:It needs to be stored in order to have historic data there.
Stephen Townsend:Yeah.
Richard Bown:So bringing that knowledge into a place where it can be Yeah.
Richard Bown:Acted upon.
Richard Bown:So do you have any examples of when, of, of, of that in use, I
Richard Bown:suppose, or a typical use case?
Stephen Townsend:Yeah, well, it's a, it is a new product,
Stephen Townsend:but we are using it right now.
Stephen Townsend:So we have just taken a new direction, um, with the product,
Stephen Townsend:really focused for the year.
Stephen Townsend:Our CEO is stepping in as, uh, chief Product Officer.
Stephen Townsend:And he's really driving the product.
Stephen Townsend:He wants to know how is the product going in the wider context.
Stephen Townsend:And so I've been running these workshops with a few, few people, including him,
Stephen Townsend:sort of figure out, okay, in terms of business and customer and technology, both
Stephen Townsend:the technology itself and how operating and delivering it, what, you know, what is
Stephen Townsend:actually important for us to know, like.
Stephen Townsend:I keep using the word indicators, so you could say KPIs, maybe.
Stephen Townsend:What are the things which, which, if we look at this, actually tell us if
Stephen Townsend:we're heading in the right direction.
Stephen Townsend:So we've been having these really interesting workshops and uh, and,
Stephen Townsend:and it's very weird to come from an engineering background and be talking
Stephen Townsend:about all these business objectives and things, but it's, it's great.
Stephen Townsend:It's fantastic.
Stephen Townsend:So we have these workshops and then my job is to, once we figure out what
Stephen Townsend:these indicators or KPIs are to go and pull them together, build a view
Stephen Townsend:that shows all these things together and how they interact with each other.
Stephen Townsend:The first one I'm looking at, we had this, we figured out our North
Stephen Townsend:Star metric is monthly active users and we had to define what that meant
Stephen Townsend:cuz it could mean a lot of things.
Stephen Townsend:And then I've been using, um, going, you know, where do I get the starter from?
Stephen Townsend:Uh, there's lots of different places.
Stephen Townsend:Um, but I ended up going to Google BigQuery cause we seem to dump a
Stephen Townsend:bunch of analytics data on there.
Stephen Townsend:You, which is maybe something a traditional observability
Stephen Townsend:engineer wouldn't do.
Stephen Townsend:But it's fantastic, you know, it's the same process.
Stephen Townsend:but you're pulling in information about customers and
Stephen Townsend:business and it's, it's great.
Stephen Townsend:So, um, that's what I've been doing, um, and it's early days, but it's
Stephen Townsend:the first thing I've been working on
Richard Bown:Awesome.
Richard Bown:Yeah, cuz that's, yeah.
Richard Bown:Going back to what we, we talked about earlier, that connection
Richard Bown:between engineering and.
Richard Bown:Is, if you can close that gap a little bit and get excited about it, then surely
Richard Bown:that's kind of done deal really, isn't it?
Richard Bown:Because you understand where things are going.
Richard Bown:And I, I wonder, I suppose, and this is maybe a question for someone who is
Richard Bown:kind of running the company or running a company who, who would like, how
Richard Bown:it can also become a little bit of a following indicator maybe sometimes.
Richard Bown:And you kind of like you are attempted to do, keep on doing more of the same thing.
Richard Bown:So how you interpret that data is gonna be key, I suppose,
Richard Bown:depending on what you're trying to.
Stephen Townsend:Ah, this is terrible.
Stephen Townsend:I always get, whenever you say, like leading and lagging indicators,
Stephen Townsend:like I know what they are and then immediately when I think
Stephen Townsend:about it, I get confused again.
Stephen Townsend:, I dunno what it's about them.
Richard Bown:anyway, so yeah, , so I called it the wrong thing.
Richard Bown:But, yeah.
Richard Bown:Well, in that, in that, in those terms, um, you are, you're measuring what you've
Richard Bown:decided to measure, and then you are, and then you are making your, your decisions
Richard Bown:based on what you decided to measure.
Richard Bown:Maybe not looking at stuff, which.
Richard Bown:You should be measuring.
Richard Bown:You see what I mean?
Stephen Townsend:Yeah, but that's, that's why I think all of this, I
Stephen Townsend:think that anything in observability and, and, and the widest things that
Stephen Townsend:I'm talking about, or even if you're doing software observability, it
Stephen Townsend:needs to be thoughtful and continual.
Stephen Townsend:Like it can't be.
Stephen Townsend:We've set it up, we are done.
Stephen Townsend:You know, we don't have to think about it ever again.
Richard Bown:Yeah, and something sparked me there around a couple of things.
Richard Bown:Firstly, when do you know that a metric isn't working anymore
Richard Bown:and what do you do about it?
Richard Bown:So kind of like is it a case of you have to kind of prune occasionally as
Richard Bown:well and not show stuff, which is maybe not helpful to, to where you're going?
Richard Bown:And secondly, , what's the visibility on the metrics typically?
Richard Bown:I mean, yeah, it's obviously up to you or one as a business to decide, but would you
Richard Bown:advocate for sometimes it being completely open and global and everyone looking at
Richard Bown:being able to see it on a dashboard or, or on a Bayer or something like that?
Richard Bown:Or would you say no, sometimes we need to kind of bring, just
Richard Bown:keep this to ourselves, you know?
Richard Bown:And when I say public, maybe public internally rather than
Richard Bown:public to to, to customers.
Stephen Townsend:Yeah.
Stephen Townsend:Uh, I generally of the opinion, I like the, I've, I've liked the term
Stephen Townsend:democratizing data, so making it a free and available for anyone to
Stephen Townsend:consume should be no secrets to hide.
Stephen Townsend:Um, So that's generally my perspective.
Stephen Townsend:But I've been in situations going back earlier in my career where I know too
Stephen Townsend:much about the finances and the health of a company and the pressures that
Stephen Townsend:executives are going through, and that just trickled down to me, and then I
Stephen Townsend:felt an incredible anxiety about it.
Stephen Townsend:So I think there are, you know, everyone doesn't need to know everything.
Stephen Townsend:I think sometimes it's, it's good to not push the, all the stress
Stephen Townsend:and burden and anxiety down the company until everyone's feeling it.
Stephen Townsend:Yeah.
Stephen Townsend:There's a lot of that sort of short term thinking when , when the direction of
Stephen Townsend:a company is just constantly changing based on who, who's in leadership at
Stephen Townsend:that particular time, or what someone thinks they need to do to get a bonus.
Richard Bown:So it's not even about the product, it's more about the process.
Richard Bown:It's easy to get distracted and easy to kind of like get all these dashboards.
Richard Bown:Sorry, gone down the black hole there.
Stephen Townsend:No, no, no.
Stephen Townsend:I, I know what you mean.
Stephen Townsend:We change things too often and it's a huge, it's, it's, it's, it's a real shame.
Stephen Townsend:The, one of the things I used to love about performance testing is that when
Stephen Townsend:I used to do workload modeling, uh, which is you, you want to understand.
Stephen Townsend:, you know, what sort of load is a system gonna be under in terms of,
Stephen Townsend:you know, different user groups and what they're doing and the key, you
Stephen Townsend:know, services or transactions they're completing and, and how that fluctuates
Stephen Townsend:and changes over the year and month and the week and the day and the hour.
Stephen Townsend:That's, I love that.
Stephen Townsend:That was a really interesting part of that job.
Stephen Townsend:And some clients I'd go to and they would've like seven years of data
Stephen Townsend:and being able to go back and say, see their whole business and all the
Stephen Townsend:customers and maybe the orders for a retail site that have gone through for,
Stephen Townsend:for seven years and it showed them.
Stephen Townsend:It's just amazing, you know?
Stephen Townsend:And being able to see very clearly, yeah.
Stephen Townsend:Oh, year in year, 20% growth.
Stephen Townsend:We've got fair amounts of confidence that's gonna happen again.
Stephen Townsend:We've gotta be ready for it.
Stephen Townsend:Are we ready for it?
Stephen Townsend:You know, I think our historic data's fantastic.
Stephen Townsend:If you can, if
Richard Bown:That is, yeah, that is brilliant.
Richard Bown:So when you can actually put that in front of someone and show them, then
Richard Bown:it's like light bulb moment for sure.
Richard Bown:So how typically would that then move on?
Richard Bown:Would they say, great, fantastic, thank you very much.
Richard Bown:We can we, we've got it from now?
Richard Bown:Or would they then implement something which would allow them to have
Richard Bown:this, this insight on a, on a daily?
Stephen Townsend:Well, this is going back to the mean, this is one of the
Stephen Townsend:reasons I wanted to, I, I wanted to move out of performance testing and
Stephen Townsend:engineering, is that you'd do this thing once and use it as an input into
Stephen Townsend:how you test the, you know, low test.
Stephen Townsend:Uh, and you'd put a.
Stephen Townsend:You know, you'd say, here it is, and you'd put it down and
Stephen Townsend:everybody's go, that's amazing.
Stephen Townsend:And then never look at it again.
Stephen Townsend:You know, there would be no setting or something up to,
Stephen Townsend:to continue to look at it.
Stephen Townsend:It would just be a one-off thing.
Stephen Townsend:And so that's why I, I thought if I get into this field of sre, it sounds
Stephen Townsend:like, uh, I'm gonna get a chance to, you know, to set things up like that, which
Stephen Townsend:I think actually far more valuable in.
Stephen Townsend:Then creating, you know, spending weeks or months creating this elaborate low
Stephen Townsend:test suite and running it in, in this environment, which is in production.
Stephen Townsend:You know, it seems like a huge waste opportunity at times.
Richard Bown:Yeah, and you'd hope with the.
Richard Bown:Is that we have now in terms of observability, that you can do it
Richard Bown:in production basically as well.
Richard Bown:It's kinda like in some ways come full circle.
Richard Bown:We've tried to kind of do it in dev or do it in test by making all our
Richard Bown:envi, aligning all our environments.
Richard Bown:But again, it comes down to costs.
Richard Bown:A lot of the time you don't wanna be able to stand up a dev or test environment,
Richard Bown:which is the same size or performance as production because it costs too much,
Richard Bown:especially cloud-based these days or two.
Richard Bown:So the only place you're gonna get meaningful data is in.
Richard Bown:maybe that's, that's the convergence point now, isn't it?
Richard Bown:Is that, and that's why this is such an, an interesting field because it's got
Richard Bown:to a point now where we've realized that there's only one place where we can do it.
Richard Bown:So we might as well get this, uh, super fast.
Richard Bown:You know, same, same with DevOps.
Richard Bown:I mean, for many years it was, oh yeah, we need to create poor requests
Richard Bown:and we need to feature branches to be able to separate all of.
Richard Bown:Different streams of work, but now it's like, no, we all need to do trunk based
Richard Bown:development and we wanna get stuff into production as soon as possible, but make
Richard Bown:sure that it's good before we get there.
Richard Bown:How can we kind of shave that process off as well?
Richard Bown:So, or make that process slimmer, I should say.
Richard Bown:Okay, that's really interesting.
Stephen Townsend:I have this, um, I, I host a podcast called Slight
Stephen Townsend:Reliability, and it's about sre, but I got a, cuz I know from the community, a
Stephen Townsend:performance engineer called Joey Hendrix.
Stephen Townsend:And there's an interview I did with him.
Stephen Townsend:He does performance testing in production and he used this,
Stephen Townsend:um, analogy of Plato's forms.
Stephen Townsend:So Plato has this idea that there's this perfect form out there.
Stephen Townsend:It's the perfect of every, of, of a particular thing.
Stephen Townsend:And it just, and in the world of performance testing, that's
Stephen Townsend:production and anything else we do is like a imperfect, you know,
Stephen Townsend:representation of, of production.
Stephen Townsend:So, and everything we get out of it.
Stephen Townsend:It's not necessarily reliable.
Stephen Townsend:Yeah.
Stephen Townsend:Then that's, of course, you have to be very careful and thoughtful if you're
Stephen Townsend:gonna perform a test in production about when you do it and what you do.
Stephen Townsend:You know, you might can't necessarily add new customers and put financial
Stephen Townsend:transactions through without a lot of, you know, preparation and
Stephen Townsend:forethought on how you're gonna do that.
Richard Bown:It's a lot of as covering is all it is.
Richard Bown:I mean, that's, and that's it's all what it, that's all it comes down to.
Richard Bown:Someone doesn't wanna lose, lose their job because of the, it was,
Richard Bown:it was on their watch, essentially.
Richard Bown:So they said, okay, you can do it without doing, I dunno, several acceptance tests.
Richard Bown:Acceptance tests runs before you go somewhere, you know?
Richard Bown:Um, and.
Richard Bown:Yeah, I wonder if that's changing slowly.
Richard Bown:I mean, for example, I've worked in the financial sector for quite a while,
Richard Bown:um, on various large projects which are around, yeah, accounting and finance
Richard Bown:and the goli are always very slow.
Richard Bown:The development process was typically quite slow as well.
Richard Bown:So we'd be lucky if we get a couple of races out a year typically
Richard Bown:what's the worst that could happen?
Richard Bown:Yeah.
Richard Bown:you know what?
Richard Bown:The most successful companies are doing that right now, and they're
Richard Bown:doing, they're following the best practices that we read about in
Richard Bown:all these great books, accelerate, et cetera, and DevOps handbook.
Richard Bown:They do that already and they understand it fundamentally.
Richard Bown:So that kind of brings me round, that's, sorry I'm ranting slightly here, but
Richard Bown:it kind of brings me round to think, should we only spend our time working
Richard Bown:for companies who really get it?
Stephen Townsend:Um, I, I think so.
Stephen Townsend:If you're the kind of person who really cares about.
Stephen Townsend:You know, having an impact and doing, you know, doing good work
Stephen Townsend:and, and seeing things improve.
Stephen Townsend:Yeah.
Stephen Townsend:Um, I know I've got, I, I go crazy.
Stephen Townsend:I, I, I get really mentally unwell in a situation where
Stephen Townsend:there's, there's of stagnation and not nothing's moving forward.
Stephen Townsend:So yeah, if, if you're that kind of person, I think absolutely.
Stephen Townsend:Go find a, even, even a team within a company who's doing, I think potentially
Stephen Townsend:can give you that sense of progression.
Stephen Townsend:Um,
Richard Bown:I think this is a subject for a whole separate
Richard Bown:episode, to be honest, isn't it?
Richard Bown:? Yeah.
Richard Bown:Because yeah, if you're committed to a piece of work, you put a lot of
Richard Bown:effort into it, you wanna make sure that it's, it's the best that you can
Richard Bown:be and also that the company that you work for is the best that it can be.
Richard Bown:So that's the, going back to the pride, going back to the
Richard Bown:commitment that you put into a role.
Richard Bown:So yeah, let's talk about that maybe in the future.
Richard Bown:That'd be really cool.
Stephen Townsend:Okay.
Richard Bown:And you mentioned also slight reliability, which is
Richard Bown:how I first came to, first came to talk to you in the first place.
Richard Bown:I love your podcast.
Richard Bown:I think it's fantastic.
Richard Bown:Um, best of luck with that.
Richard Bown:Steven, I just wanna say thanks for joining me
Richard Bown:. Stephen Townsend: No, it's
Richard Bown:And I'd love to come back and talk about, you know, working for
Richard Bown:a, a company which makes, which aligns with how you wanna feel.
Richard Bown:Yeah.
Richard Bown:Yeah, that's a really cool way of putting it.
Richard Bown:Okay, great.
Richard Bown:Good talking to you.
Richard Bown:Many, thanks to Steven for agreeing to come on and thanks
Richard Bown:to you for joining me to.
Richard Bown:Join me next time on loving legacy.
Richard Bown:And if you enjoyed this episode, please feel free to tell all your friends
Richard Bown:and colleagues about it and give us a rating on your favorite podcast app.
Richard Bown:Until next time.