In enter­prises we tend to sell archi­tec­tural projects by show­ing you the cur­rent messy state and pitch you an ide­al­ized end state. But the prob­lem is that, the ideal end state we’re pitch­ing exists only as an idea, with­out hav­ing gone through all the com­pro­mises and organic growth that made the cur­rent state messy.

Once we have gone through the process of imple­ment­ing and grow­ing the sys­tem over the course of its life­time, how do we know we won’t end up in the same messy sit­u­a­tion again?

The 3-year Plans

Worse still, these pitches are always fol­lowed by a 3 year plan where you see no return on your invest­ment for 2 whole years!

And since 3 years is a long time, many things can hap­pen along the way - merg­ers, acqui­si­tions, new tech­nolo­gies, new CTO/CIO, etc. — and often we don’t ever real­ize the orig­i­nal 3 year plan before these forces of change takes us to a new direction.

And the vicious cycle con­tin­ues.…

 

Within a large orga­ni­za­tion there are always mul­ti­ple forces of change hap­pen­ing at the same time, and your per­spec­tive changes depend­ing on your posi­tion within the organization.

In a com­plex sociotech­ni­cal sys­tem like this, there’s no priv­i­leged van­tage point. One per­son can­not under­stand all ram­i­fi­ca­tions of their deci­sions, not even the CEO of a company.

The best you can do, is locally con­tex­tual actions.

 

8 Rules for Survival

1. Embrace plurality

Avoid ten­dency for the “one”, e.g.

• sin­gle source of truth;
• sin­gle sys­tem of record;
• one sys­tem to rule them all;
• etc.

The prob­lem with hav­ing a sin­gle sys­tem of records (SSOR) is that it’s ter­ri­bly frag­ile. Many real-world events (e.g. M&A, divesti­tures, enter­ing new mar­ket) can eas­ily vio­late your pur­suit for the “one” system.

In epis­te­mol­ogy (i.e. the study of “how we know what we know”), there’s the idea that we can only know what we are able to record in a sys­tem. If your SSOR is badly shaped for the infor­ma­tion that exist, you might be unaware of the rest of the instances of infor­ma­tion you’re look­ing for.

 

When mod­el­ling a domain, some­times it’s even tough to find “one” def­i­n­i­tion for some­thing that every­one can agree on.

For exam­ple, a “cus­tomer” can mean dif­fer­ent things to dif­fer­ent departments:

• for cus­tomer ser­vice, a cus­tomer is some­one who is pay­ing you money
• for sales, some­one is a cus­tomer as soon as you start talk­ing to them

 

Finally, there’s this philo­soph­i­cal idea that, the model you use to view the world shapes the thoughts you are able to think.

Your par­a­digm defines the ques­tions you can ask.

I think this also applies to pro­gram­ming lan­guages and par­a­digms, with the same detri­men­tal effects in lim­it­ing our poten­tial as cre­ative prob­lem solvers. And it’s a good rea­son why you should learn at least a few pro­gram­ming par­a­digms in order to expand your mind.

---

 

When deal­ing with mul­ti­ple sys­tems of records, we change our empha­sis and instead focus on:

• who’s the author­ity for what span?
• define a sys­tem to find the author­ity for a span given some unique iden­ti­fier, e.g. URIs, URNs (AWS’s ARN sys­tem springs to mind)
• agree­ing on a rep­re­sen­ta­tion so we can inter­change them across dif­fer­ent systems
• enable copies and have a mech­a­nism for synchronization

Some­thing inter­est­ing hap­pens when you do these — your infor­ma­tion con­sumers no longer assume where the infor­ma­tion comes from.

If you pro­vide full URLs with your prod­uct details, then the con­sumers can go to any of the URLs for your prod­uct infor­ma­tion. They must become more flex­i­ble and assume there is an open world of infor­ma­tion out there.

Along with the infor­ma­tion you get, you also need a mech­a­nism for deter­min­ing what actions you can per­form with them. These actions can be encoded as URLs and may point to dif­fer­ent services.

---

As I lis­tened, two words sprang to mind:

microser­vices - break­ing up a big, com­plex sys­tem into smaller parts each with author­ity over its own span; infor­ma­tion is accessed through a sys­tem of iden­ti­fiers and an agreed-upon for­mat for inter­change (rather than shared data­base, etc.)

HATEOAS — or REST APIs in the way Roy Field­ing orig­i­nally described; actions per­mit­ted on infor­ma­tion is pass back along­side the infor­ma­tion itself and encoded as URLs

 

2. Con­tex­tu­al­ize Downstream

We have a ten­dency to put too much in the data we send down­stream, which can often end up being a very large set.

But our busi­ness rules are usu­ally contextual:

• not all infor­ma­tion about an entity is required in every case
• oper­a­tions are not avail­able across entire classes of entities

---

Again, HATEOAS strikes me as a par­tic­u­larly good fit here for con­tex­tu­al­iz­ing the data we send to down­stream systems.

---

We build sys­tems as a net­work of data-flows, but we also need to look at the ecosys­tem of sys­tem of sys­tems and be aware of the down­stream impacts of our changes.

Some­thing like the fol­low­ing is obvi­ously not desirable!

So the rule is:

• Aug­ment the data upstream, add fields but not rela­tion­ships if you can avoid it.
• Con­tex­tu­al­ize down­stream, apply your busi­ness rules as close to the user as pos­si­ble. Because as it turns out, the rules that are clos­est to the users change the most often.
• Min­i­mize the enti­ties that all sys­tems need to know about.

 

3. Beware Grandiosity

Fight the temp­ta­tion to build the model to end all mod­els (again, embrace plu­ral­ity), aka Enter­prise Mod­el­ling Dic­tio­nary.

The prob­lem with an Enter­prise Mod­el­ling project is that, it requires:

• a global per­spec­tive, which we already said is not avail­able in a com­plex system;
• agree­ment across all busi­ness units, which is extra­or­di­nar­ily hard to achieve;
• tal­ent for abstrac­tion; but also
• con­crete expe­ri­ence in all contexts;
• a small enough team to make decisions.

Such teams only exist in Hol­ly­wood movies..

Instead, look to how stan­dard com­mit­tees work:

• they com­pro­mises a lot;
• they don’t seek agree­ments across the board, and instead agree on small sub­sets and leave room for ven­dor ventures;
• they take a long time to converge
• they delib­er­ately limit their scope

“All mod­els are wrong, but some are useful.”

- George Box

So the rule is:

• don’t try to model the whole world;
• find com­pro­mises and sub­sets that every­one can agree on;
• start small and incre­men­tally expand;
• assume you don’t know every­thing and leave room for extension;
• allow very lengthy com­ment period (with plenty of email reminders since every­one leaves it till the last minute..)

 

4. Decen­tral­ize

Decen­tral­iza­tion allows more explo­ration of mar­ket space because it allows mul­ti­ple groups to work on dif­fer­ent things at once.

You can also explore the solu­tion space bet­ter because dif­fer­ent groups can try dif­fer­ent solu­tions for sim­i­lar problems.

How­ever, you do need some pre­req­ui­sites for this to work successfully:

• Trans­parency — meth­ods, work and results must be visible
• Iso­la­tion — one group’s fail­ure can­not cause wide­spread damage
• Eco­nom­ics — cre­ate a frame­work for every­one to make deci­sions that con­tribute towards the larger goal

The Boe­ing 777 project is an excel­lent exam­ple of dis­trib­uted eco­nomic decision-making.

The no. 1 thing that deter­mines a plane’s life­time cost is the weight of the air­craft. So the weight of the air­craft is in direct trade-off with cost (e.g. lighter mate­ri­als = more expen­sive man­u­fac­tur­ing cost).

They worked out what the trade-off was and set out rules so that an engi­neer can buy a pound of weight for $300 — i.e. as an engi­neer, you can make a deci­sion that’ll shave a pound of weight off the air­craft at the expense of $300 of man­u­fac­tur­ing cost with­out ask­ing anybody.

Engi­neer­ing and pro­gramme man­agers and above have a big­ger man­u­fac­tur­ing cost budge per pound, and any­thing above these pre­de­fined lim­its requires you to go before the gov­er­nance board and jus­tice your decisions.

What they have done is set­ting the trade-off poli­cies cen­trally and by defin­ing the bal­anc­ing forces glob­ally they have cre­ated a frame­work that allows deci­sions to be made locally with­out requir­ing reviews.

In IT, Michael has observed that cen­tral­iza­tion tends to lead to bud­get fights between depart­ments. Whereas decen­tral­iza­tion allows for sliv­ers of bud­gets work­ing towards a com­mon goal.

 

5. Iso­late Fail­ure Domains

If you reverse the direc­tion data flows through your sys­tems and you can find the direc­tion of depen­den­cies between them.

And the thing about depen­dency is that some­times you can’t depend on them. So you def­i­nitely want to iso­late your­self from upstream depen­den­cies. In his book, Release It!, Michael out­lined a num­ber of use­ful pat­terns to help you do just that, e.g. cir­cuit breaker, bulk­heads, time­out,  fail fast. Many of these pat­terns have been incor­po­rated in libraries such as Netflix’s Hys­trix.

 

Another kind of fail­ure is when you have missed a solu­tion space entirely. This is where mod­u­lar­ity can give you value in that it gives you options that you might exer­cise in future.

Just as options in the finan­cial sense (a long call in this case), there’s a neg­a­tive pay­off at the start as you need to do some work to cre­ate the option. But if you choose to exer­cise the option in the future then there’s a long pos­i­tive payoff.

Whilst Michael didn’t spell it out, I think the value of mod­u­lar­ity in this case lies in the abil­ity to recover from hav­ing missed a solu­tion space the first time around. If the mod­u­lar­ity is there then at least you have to option to adopt a dif­fer­ent solu­tion at a later point (i.e. exer­cis­ing the option).

---

Michael then gave an exam­ple of a trad­ing com­pany who has build their sys­tem as many small appli­ca­tions inte­grated by mes­sag­ing. The appli­ca­tions are small enough and fine grained enough that it’s cheaper to rewrite than to main­tain, so they treat these appli­ca­tions as dis­pos­able components.

---

I’d like to give a shout out to Greg Young’s recent talk, The Art of Destroy­ing Soft­ware, where he talked at length about build­ing sys­tems whilst opti­miz­ing for deletabil­ity. That is, you should build sys­tems so that they’re never big­ger than what you can rea­son­ably rewrite in a week. And that is as good a guide­line on “how big should my microser­vice be?” as I have heard so far.

 

6 & 7. Data Out­lives Appli­ca­tions; Appli­ca­tions Out­lives Integrations

As things change, a typ­i­cal lay­ered archi­tec­ture breaks when there are pres­sure from either side.

A Hexag­o­nal archi­tec­ture, or Ports and Adapters style archi­tec­ture on the other hand, puts your domain in the cen­tre rather than the bot­tom of the stack. The domain is then con­nected to the out­side world via adapters that bridges between the con­cepts of that adapter and the con­cepts of the domain.

Hexag­o­nal archi­tec­tures make tech­nol­ogy tran­si­tion more of a bound­ary layer issue than a domain layer issue, and allows you to explic­itly rep­re­sent mul­ti­ple adapters from the same domain for dif­fer­ent purposes.

 

8. Increase Discoverability

In large orga­ni­za­tions you often find peo­ple work­ing on projects with sig­nif­i­cant amounts of over­lap due to lack of dis­cov­er­abil­ity about what other peo­ple are work­ing on.

Just as one would search Github or Google to see if a sim­i­lar project exists already before embark­ing on his/her own open source alter­na­tive, we should be doing more to enable dis­cov­er­abil­ity within companies.

Doing so allows us to move faster by build­ing on top of oth­ers work, but in order to do this we need to make our work vis­i­ble internally:

• inter­nal blogs
• open, eas­ily acces­si­ble code repositories
• mod­ern search engine

Inter­nal teams should work like open source projects whilst retain­ing a prod­uct mentality.

One thing to watch out for is the bud­get cul­ture where:

• con­tri­bu­tions to other people’s projects are not wel­comed due to bud­get fights;
• inquiries about each other’s work are alarm­ing and con­sid­ered pre­cur­sor to their attempt to suck your bud­get into theirs

Instead you should embrace an engi­neer­ing cul­ture which tend to be much more open. You could start to build up an engi­neer­ing cul­ture by:

• hav­ing team blogs
• allow users to report bugs
• make CI servers public

 

 

Finally, stop chas­ing ide­al­ized end state, and embrace the wave of change.

 

I was at QCon Lon­don ear­lier this year, and in Kevlin Henney’s Small is Beau­ti­ful talk he deliv­ered a sim­i­lar mes­sage. That we should think of soft­ware as prod­ucts, not projects.  If soft­ware are projects then they should have well-defined end state, but most often, soft­ware do not have well-defined end state, but rather evolved con­tin­u­ously for as long as it remains desir­able and purposeful.

 

 

Links

• Release It! by Michael Nygard
• Don’t learn a syn­tax, learn to change the way you think
• QCon Lon­don 15 — Small is Beautiful
• REST APIs must be hypertext-driven
• HAL spec