What does it mean to support, extend or even replace a monolith and should we even try? This time I explore the landscape as it is now - when we feel under pressure to "do microservices" yet we have something that works but is perhaps too much of a monolith for us to work with effectively.

This quote from the end of the episode perhaps sums it up:

"Emergent architecture is used for understanding and extending complex systems, which have arrived at their state through an unknown path."

NOTES

https://richardwbown.com/build-now-vs-build-later-upfront-design-vs-implementation-costs/

https://richardwbown.com/beating-the-monolith-understanding-modern-software-delivery/

https://www.bol.com/nl/nl/f/domain-driven-design/9200000002151217/

https://richardwbown.com/defining-the-bounded-context-is-the-key-to-flow/

https://thenewstack.io/what-we-can-learn-from-twitters-outages/

QUOTES

01:25 - "emergent architecture is a smart way to say we built something and once we had a plan, but now we have to change it" [RB]

01:52 - "Documentation is of use in order for software archeologists to discover the business motivations for why we're built that way in the first place" [RB]

02:22 - "This is the very definition of the sunk cost fallacy. Those that have invested so much financially, emotionally, and reputationally, that they must double down on that approach until it's either universally accepted or until it's dead in the water. This approach, I call the "Abyss of Obsession", and it's a social construct" [RB]

03:19 - "it's important to dream big. We're all capable of it, and software is the perfect scaffolding for these dreams" [RB]

04:46 - "If you've ever tried to reverse engineer a system, then you will know that a complex system is exactly that, often intractable" [RB]

05:39 - "what you see from typical software projects is that either zero sum or significant upfront planning or design is made" [RB]

06:46 - "I call this cost to first 'gotcha'. The moment where we think, oh no, we forgot this fundamental thing" [RB]

08:07 - "Do not let our ideas become code." [RB]

09:41 - "Emergent architecture means that things change. We can be resilient to change, not just with how we build our applications physically, but also by expecting things to need to change." [RB]

10:30 - "This is why a legacy system is really every system we ever build" [RB]

11:03 - "I believe that just like a good domain model, we already have enough or perhaps too many words to describe the patterns that we see in enterprise systems and architecture." [RB]

12:02 - "Emergent architecture is used for understanding and extending complex systems, which have arrived at their state through an unknown path." [RB]

Hello and welcome to Loving Legacy.

This is episode 28.

I'm Richard Bown your host, and this time I'm talking about emergent

architectures and beating the monolith.

It's been a while since I recorded myself just talking about

things and to tell the truth.

I've been thinking long and hard about this next step.

I've been working on a course which will teach technical leaders why all of their

effort to gain clarity and control of software development and delivery is

essentially wasted unless some effort is spent to acknowledge how software

gets built outside of all the frameworks and techniques that we are bombarded.

This means consuming a lot of those frameworks, ideas and techniques

myself, and then putting some words together to form a common understanding

of what we're trying to achieve.

Ironically, for some, you might recognize this as a fundamental

principle of domain-driven design, defining a domain language, and then

modeling a solution using that language.

Okay, so let's dive in.

In Sam Newman's building Microservices second edit.

It's all about building distributed systems.

Pretty much any system is a distributed system right now goes the argument.

So should we all be setting out explicitly to build

microservices based architectures?

Many seem to think so and have invested heavily both in the technology

platforms to support this as well as the skills and learnings that

allow successful implementation.

However, is this a one size fits all?

emergent architecture is a smart way to say we built something and once we had

a plan, but now we have to change it.

This is therefore the definition of pretty much any software

ever a monolith can appear.

It's just a block of code, and we will find that difficult

to break down any further.

Usually due to design decisions made before or during its assembly,

a monolith could even appear nominally in a microservice, although

that would probably mean that it wasn't a microservice anymore.

Documentation is of use in order for software archeologists to discover

the business motivations for why we're built that way in the first place.

They're a great starting point to understand how your software

developed to the point it is now.

The history of software development is littered with the bones of great

technological frameworks and ideas that have briefly set the world a

light only to disappear Agonizingly slowly , as the zealots of the

ideology continue to clinging on to the investment they have made.

This is the very definition of the sunk cost fallacy.

Those that have invested so much financially, emotionally, and

reputationally, that they must double down on that approach until

it's either universally accepted or until it's dead in the water.

This approach, I call the "Abyss of Obsession", and it's a social construct.

having been there myself with ideas, designs, and architectures, I feel I

must talk them through with peers until we reach either an understanding that

could work or a feeling that it couldn't.

Any ideas that persist after this peer group round housing will eventually

start to take seed in our minds.

This will be our famous thing we think, and we get ahead of ourselves

in believing that we've solved the world's software problems.

If you're not familiar without feeling yet, then I think you probably need

to find a better class of software developers to hang around with and

just dream a little bit bigger.

Usually this kind of discussion happens in the pub after a few ales.

But it's important to dream big.

We're all capable of it, and software is the perfect scaffolding for these dreams.

Okay, so saying we implement something using this technique and it works.

Then we start to enter this abyss of obsession and we are most

likely not to return for a while.

We are a submarine disappearing to the Mariana's trench of ambition.

This applies as much to architecture frameworks as it does process frameworks

like Agile, as it does technologies.

We have a hammer and everything looks like a.

. Okay, so taking obsession around frameworks, technologies, and processes.

As read, we should be careful when accepting new ideas and new words

and concepts into our vocabulary.

The monolith doesn't actually exist as in as much a microservice

doesn't actually exist.

These are abstract ideas, or we can build our imperfect models

of the world and attempt to implement them and learn as we go.

What we end up with in software design, development and

delivery is a complex system

According to Dave Snowden's Canne Framework, a complex

system is the unknown.

Unknowns cause and effect can only be deduced.

In retrospect, there are no right answers.

Instructive patterns can emerge if the leader conducts

experiments that are safe to fail.

Can Evan cause this process probe sense?

If you've ever tried to reverse engineer a system, then you will know that a complex

system is exactly that, often intractable.

They are the result of forgotten rules.

One can design a system as well as you like, but if you've forgotten

a crucial rule until later in the process, then often the amount of

work to completely redesign the system to accommodate the fundamental

thing you've forgotten is too cost.

I've tried to graph this effect and I will link an article which shares these graphs.

We map cost of implementation versus cost of design, and we call design

anything in this case that isn't coding, testing, deploying, or supporting the

code directly via incidents, et cetera.

The cost of design therefore could encompass anything that is seen as

thinking about the system, refinement sessions about architecture.

Workshops, presenting facts or findings, learning new techniques.

Anything which falls into the category of talking about rather than doing

what you see from typical software projects is that either

zero sum or significant upfront planning or design is made.

If no upfront planning is made, then the project starts amidst a

flurry of activity, coding, design, experiments, trial and error, depending

on the ambition of the project.

This might be the right way to go.

No, upfront cost means jumping in, iterating quickly, and

learning as it moves forward.

Some design means that decisions perhaps around technology platforms

architecture have been made, and we perhaps know an outline of our

system design before we start coding.

Significant design means mapping out exactly how our system is going to look.

and proceeding on that basis.

This means that building is optimized, but we might end up building the

wrong thing for any approach.

The only time we can validate our assumptions is when we get

our system in front of the people it's supposed to be serving.

When we receive that initial feedback, we can iterate our design and depend on

how flexible our initial architecture is.

This will come with a variable cost.

I call this cost to first "gotcha".

The moment where we think, oh no, we forgot this fundamental thing,

and often software systems lurch from gotcha to gotcha until perhaps

they either fulfill their initial brief or requirements and become

useful to the customer or they fail.

But no system is ever completely free of gotcha's because users can always

think of things that we can't as system designers or builders and software being.

Being flexible and software engineers also being prone to

fall into the abyss of obsession.

No change is too great for our system.

No matter what approach we take initially, we will always be faced with a dilemma.

Should we extend this legacy system to include new functionality

or should we just write a new.

Therefore legacy is a social trap even more than it is a technological trap.

This is the very basis of emergent architecture and why we'll never

actually beat the monolith.

It's not something to be beaten.

It's something to be used when appropriate.

It's something to be used as any other tool in your toolkit.

often as programmers, we don't set out with anything in mind

other than a purpose for our work.

The rest of the discussion, the design architecture technologies and the

things that we use to build our ideas upon are often only abstract concepts

which we use to share our ideas.

Once they are made flesh in whatever tech we choose, they inevitably change.

Do not let our ideas become code.

This is why I have a somewhat ambivalent attitude to diagramming

tools and ways to turn ideas into code.

Things like the C4 framework are good at a high level, but these are

merely conceptual boxes of things.

Any detailed design driven from flittering ideas are usually

not proven to be helpful.

We will, at some point, have to compromise on these ideas, and usually at that

point we'll work too hard to modify our original design to accommodate change.

Then perhaps it's not even helpful.

The same happens when we consider any design paradigm, purely idiomatic.

If you read the Blue Book by Eric Evans on domain-Driven Design,

you'll find the premise of it is one.

For most software projects, the primary focus should be on the domain and

the domain logic, two complex domain designs should be based on a model.

For me, there is nothing to argue with here.

Understand what you're trying to achieve and then deliver a model of it.

Also talking the same language as the users, but it's surprising how often we

are drawn to focus on techniques or sub techniques rather than the bigger picture.

That's understandable because to form thoughts like this takes

a lot of hard work and time, and it's only conceptual work.

It's not a useful day to day for an engineer or engineering leader.

Even practical techniques are the things that we fall back on

and utilize, remembering to use.

And remembering why we're using them is good to always keep in mind.

Emergent architecture means that things change.

We can be resilient to change, not just with how we build our

applications physically, but also by expecting things to need to change.

To be aware that circumstances outside our control may intercede

to make us have to change.

One of the conclusions we must be able to draw now from the Elon

Musk evisceration of engineering at Twitter is that the system was

certainly well enough engineered to survive significant tampering.

It's still running.

Despite a few blips and outages, things are changing.

The architecture is adapting and growing.

The rest is supposition but all we can do is base an understanding on what

we feel and experience as customers.

Whatever we choose as a starting point.

Know some or lots of design, our decisions are always going to be

called into question, but new people challenging our ideas of technology,

fitness, for purpose and business direction are always going to appear.

Therefore, our starting point doesn't really matter, and by

extension, where we are now with our systems doesn't really matter.

This is why a legacy system is really every system we ever build.

If we truly value it and we want it to be better for our customers, then we must

treat every system in whatever state, age, or technology with the same reverence.

It's only dependent upon relevance in the market and relevance to the customers.

I propose a framework of the best practices that allow us to stay grounded

to our customers' needs as closely as possible, while reusing all the terms

that any good engineering organiz.

are already familiar with.

I believe that just like a good domain model, we already have

enough or perhaps too many words to describe the patterns that we see in

enterprise systems and architecture.

What is missing is a good Rosetta Stone that enables us to speak

the same language across ideas.

If we can reuse rather than reinvent terms and agree on a language, then perhaps we

edge closer to a common understanding of what it means to build great software.

This is where novels like the Phoenix Project or the

Unicom Project, are so strong.

They place these facts in a story as techniques to try and help us

understand where we want to go.

And this is something that we can all do.

Storytelling with practical examples from your business, working out how the

stories of your systems need to change in order for you to deliver the right

value to your customer at the right time.

Fundamentally, this is all we can hope to achieve from software engineering

. Engineering anything is about

Something that a customer will pay money for or utilize.

Therefore, we define software engineering as building the

right software for a customer.

Emergent architecture is used for understanding and extending complex

systems, which have arrived at their state through an unknown path.

Try to learn how you got where you are now through analysis,

reading, talking to engineers and customers, finding out the history.

Only then will you be able to plot a path to where you need to be next.

Your frameworks, technology choices, et cetera, either constrain

or enable its path and should also inform you of your future.

I hope I've given you something to think about in this episode.

I'll be covering more around this in the future, so, If you enjoy this, then

please like and subscribe to the podcast.

And, feel free to share it with your friends and colleagues, and also feel

free to reach out to me if there's anything you'd like to talk about

legacy systems engineering, or the future of emergent architecture.

Until next time, this is Richard Bown wishing you goodbye and good luck.