Making software is fun. Running software can be less fun. How do you do both and not go insane? How do you make sure you're not working all the time to keep your (virtual) baby up in the air?

The same rules apply whether you're a startup, a scale-up, a corporate, an individual, an open source project contributor. You're part of your own support problem.

This time I discuss what it's like to be a coder who wants to have it all - the good times and the bad times - but wants to limit the bad times to working hours only. I hope you enjoy it as much as I made recording it.

NOTES

XKCD 242 - The Difference:

https://xkcd.com/242/

Brian Scanlan about Intercom's approach to on call.

https://www.intercom.com/blog/rapid-response-how-we-developed-an-on-call-team-at-intercom/

And he talked about this recently on Adventures in DevOps:

https://topenddevs.com/podcasts/adventures-in-devops/episodes/reducing-on-call-engineer-burnout-with-a-volunteer-management-infrastructure-devops-121

Inspiring music that helped me write this:

https://en.wikipedia.org/wiki/Milk_%26_Kisses

https://en.wikipedia.org/wiki/Reading,_Writing_and_Arithmetic

QUOTES

01:25 - How did it feel the first time you wrote something in a computer language that maybe you just learned and it worked perhaps first time

02:50 - We're in that space where creation is flowing. But at some point reality bites. Then we need to do something with this thing that we've created.

5:05 - What is happening tonight? Mummy, why can't we go out to the zoo this weekend? I know guilt right. The point I'm really laboring here is that one should be proud of your creations.

05:18 - All of our creations need us. All of these things are calling for our time.

06:48 - Some of the most successful systems on the planet have been built on a knife edge of functional validity.

07:43 - Your job is therefore to minimize that risk and ensure that you can roll back or fix forward with the least possible effort.

09:21 - Be aware of your legacy as a coder. Think carefully about how your persisted data will be managed.

Welcome to the software delivery club.

Every week, I explore different aspects of the business of delivering

software, into production, talking about subjects that I find crucial to

good software delivery, as well as to industry experts about how they get to

keep their production software running.

I want to know whether they build or buy, how they cope with new

features and product changes.

How they keep the customer satisfied while also making that engineers happy.

I'll also talk about what tools we like to use, which ones we don't as

well as tales from the front line, when it comes to delivering and

supporting production grade software.

Thanks for joining me.

I'm Richard Bown, your host.

And I'm struggling with a post COVID brain and getting anything together has been a

bit of a struggle last couple of weeks.

However, with the help of my friends, I think I'm getting there.

So this is episode 14 of the software delivery club.

All about getting your evenings and weekends back.

So what's the most important thing about building software?

Is it the process we follow?

Is it the tools we use?

Is it the IDE we have to agree on, is it the command line

versus the click to click?

Do we all have a favorite infrastructure provider?

Do we have a favorite tool?

What about CD?

Did we do it?

Do we like it.

Does any of this actually matter?

Are we all caught up on too many details.

Think about something for a second.

How did it feel the first time you wrote something in a computer language

that maybe you just learned and it worked perhaps first time, perhaps not.

And it worked.

And just for a moment, she took a look at it and thought, huh?

Yeah.

I remember that moment.

It didn't last long.

Did it?

Because you were already onto the next thing.

How can I make the next thing work?

Challenge, fixed.

Move on.

Next challenge.

As computer nerds are software developers or engineers or testers or dev ops or

whatever we call ourselves these days.

We are not satisfied with just fixing it once or doing one thing.

We want to do it again, but different.

We are that XKCD cartoon 2 42.

It's called the difference.

You know, the one.

It shows how a normal person thinks versus how a scientist thinks.

The normal person thinks that's bad.

When they get electrocuted and the scientist or the engineer

or the wanderer wants to know if that's always going to happen.

We the software types are that wanderer.

We are that person.

The one who wants to know if that will always happen when you

do that and what happens next?

When you push things, when you break things.

Because inventing is breaking.

There's a world of difference between that feeling when we're

discovering and exploring and trying, and the world of reality.

When we are searching.

We are finding out we are creating.

And often making it up as we go along.

We're in that space where creation is flowing.

But at some point reality bites.

Then we need to do something with this thing that we've created.

It doesn't matter if we're a soloist or if we're an open source contributor, or if

we get paid for our code and our effort.

We want to do something with it.

It needs to find a life beyond what is found so far with just ourselves.

So we share it.

And we show it off.

And we let others play.

At that moment, our creation becomes someone else's.

So how do we make sure that the creation lives in a way that we want it to?

How do we ensure that the delivery of our idea.

As fully formed as our imagining of it.

It's time to get serious.

We put tests in.

We make CI pipelines.

We struggle with more tools.

We put change process in place we deliver, we fix, we deliver, we add.

The thing that we've created almost changes completely, before our eyes.

New features.

New bug fixes.

New opportunities.

New users, new requirements, the software that we once thought about

as ours is now fully someone else's.

How do we then prevent disaster?

When we make a change, we put more tests in, we do more integration testing.

We add a UAT phase.

We add more environments, more people, more complexity, more planning.

We get burned by bad deployments, perhaps.

So we invest in more automation.

We learned the limitations of our existing infrastructure.

We regret not building in a more scalable manner.

We try to integrate a microservice or two and get bogged down

in Kubernetes management.

Above all.

We work with more people.

Our creativity is not our own anymore.

We have to learn to share.

So to sum up life becomes more complicated.

And bear in mind, we don't live in a bubble.

This is only our worklife.

Our time is spent in front of a screen giving virtual birth to a new creation.

But other things happen to us.

During the time we work on our software.

We meet people.

Children are born and grow older.

Where are you?

Daddy?

What is happening tonight?

Mummy, why can't we go out to the zoo this weekend?

I know guilt right.

The point I'm really laboring here is that one should be proud of your creations.

Your code your product, your children.

If you have them.

All of our creations need us.

All of these things are calling for our time.

It's up to you to prioritize what that means to you.

These dependents should all have priority.

And the right priority that they deserve.

So let's get back to the subject.

How can you make delivering into production as simple as you can.

How can you get your evenings and weekends back without deployment hell?

When it comes to delivering software into production.

It's better to do it during the daytime, right?.

It's better to do it not on a weekend or on an evening.

We can lavish some of our precious work time on those needy dependents, but

don't let them take your sleep too.

So can we do this for all software situations?

For everything you betcha.

It's just a simple case of prioritizing this way of work.

Acknowledging that it won't be something you might be able to solve perhaps this

week or next week, or even next month.

But like anything else in your backlog of work, you can prioritize

downtime, free production releases.

If you truly value them.

But this will take investment of time and energy.

So let's, take a look at a few examples.

A SaaS is simple to instruments and automate, right.

Wrong.

A billing system or a back office system, like a general ledger might be simpler

still at first glance, but then there's environment management, data management.

It gets complicated.

An OLTP system is already architected in such a way that new

deployments are trivial, right?

No.

All wrong.

One thing you learn is that you never stopped learning.

Some of the most successful systems on the planet have been built on a

knife edge of functional validity.

And by functional validity, I mean the classic minimal viable product.

This is the knife edge upon which so many fortunes are made.

We are teetering on the edge of usability , scaling as we go.

Reinventing as we grow.

Perhaps it's just limited in functionality.

And when we have only certain behaviors that we can perform,

perhaps performance really sucks.

Now, perhaps we made some crucial decisions that we can't undo.

Systems are all complex.

How do we deploy them without running out of time?

How do we validate that our deployment will work in production every time.

Well, the simple truth is that we can't ever know for sure.

You can never know for sure.

I repeat.

You can't ever know for sure that a deployment is going to work.

There is always a risk with a change.

Your job is therefore to minimize that risk and ensure that you

can roll back or fix forward with the least possible effort.

As the system gets more complicated and groves, this becomes a more difficult

task in our rush to add more features.

We cut corners.

We do things that we might not want to.

We ended up making delivery more complicated for the future.

If we look ahead to our deployment story, some of it comes down to original

design and architectural decisions.

Is our system transactional.

Can you replay them?

Can you back up restore all or part of your data.

Are we tightly coupled, are we paying attention to availability during upgrades?

Can we undo everything and stand up a new system if it all goes to shit.

Some of it comes down to how we have grown.

How we have handled the changing requirements from our users,

from ourselves, from our support department, our product management.

How we've been managed, how the team has evolved, how the

company is growing or not growing.

Financial pressures or people pressures , personal lives , stories.

All intermingled.

Writing software is complex enough as it is.

Doing it in a group of people is more complex still.

Doing it for a product, which is evolving is also an extra layer of complexity.

This stuff is hard.

If you ever want to get your weekends and evenings back.

You'll need to make some tough decisions about how your app, your

system, your deployments will go.

You will need to prioritize the architecture that will enable

you to navigate change easily.

And deploy simply when you need it.

My summary of this discussion or monologue or rant, is as follows.

Be aware of your legacy as a coder.

Think carefully about how your persisted data will be managed.

Not just how easy it is to read and write.

How will your systems scale from UI, comms, backend?

Think of the implications of your eminent design decisions and how they

would look at 3:00 AM in the morning.

Always be mindful of the technology.

And the architectural choices that you make when it comes to

supportability and extendability.

How are you going to scale?

When will that be based on current estimate?

How will you support that?

When will you need help testing, building, deploying this thing?

This is really next level, thinking.

This is senior engineer, engineering manager, director, VP, CTO.

You won't be able to do this all of the time.

But you will need to be able to see the future.

A little.

Only then will you be able to plan.

To get your weekends back.

Like a pro.

Talking of which.

I listened to a wonderful podcast the other week.

It's with Brian Scanlon of Intercom and how they've turned that organization

into an on-call paradise with people actually wanting to do it.

I will link the blog and the podcast in the show notes.

TLDR.

Pay people well, Make it clear what your expectations are.

Consistent architecture, logging and monitoring for the win.

Until next time.

I wish you well.

Today's podcast has been powered by the Cocteau twins, "Milk and Kisses".

Plus the Sunday's first album.

Plus BBC six music.

And the drilling in our little office and Amstelveen.