The Layered Look

When To Layer Your Software and When Not To

2008/03/26 Like the "flexibility and modularity" mantras, layering of software designs appears to be unquestioned these days. I think it needs to be questioned:

I heard on a radio show the other day, a successful business owner saying that "success is the ability to fail quickly and repeatedly." This rings true to me...unless of course, I'm up skydiving. :-)

For brand-new systems, in which the layers, especially the database layer, have not yet been defined, adding the traditional layers: presentation layer, business layer, persistence layer and database layer, usually causes more trouble than it is worth. The software textbooks are full of layering examples, but no software text book I've ever read has mentioned the essential problem with layering. It is this:

We humans are notoriously bad at "one-shot" design protocols, in which most everything has to be right the first time, but we are very good at design protocols that allow us to take many shots until we hit the target. In other words, we are very good at "failing our way quickly to success".

Allow me to paint the typical 3-tier design scenario, and I think the point will become clear:

Someone, or a group of someones, writes a functional spec, which, if well-written, is usually about 75% accurate and complete. On the functional spec cutoff date, everyone (end users, DBAs, programmers, managers, and so on) "signs off" on the spec, even though, like the "fine print" on the back of their credit card contracts, no one ever FULLY understands it. Design and coding of the layers begins, usually in parallel because the project is already behind schedule.

From that functional spec, the DBA writes the database schema, the middle-tier programmers design and writes the business layer and the persistence layer, and the front-end programmers write the presentation layer.

The completed system is now delivered to the users for acceptance testing. At this point, one of the users discovers that a critical piece of data is not on the screen. Somehow, it was left out of the functional spec. Oops.

Now, please don't get critical of the spec writers. NO ONE, not them, not you and not I, can write a perfect functional spec in "one-shot". As I said, we humans simply aren't designed that way....So, to continue...

Ooops. Now, the DBA has to add the column to the schema, and re-create the database along with all the data...oh, and in 4 coding environments now: development, test, QA and acceptance...lots of fun.

And since the auto-generated code had to be hand-modified for some of the persistence logic, the persistence layer coder has to manually add the new field to the proper object (class) in that layer...and again in all 4 coding environments...lots of fun for her, too. (For those of you who have never had to do this yourself, I'm being sarcastic here. Trust me, it's not high on the list of programmers' dream jobs.)

And of course, fields in the persistence layer have to go through the business layer to get to the presentation layer, so the business layer programmer has to add the field to one of her business objects...and again in the 4 coding environments.

And finally, the presentation layer programmer has to add the field to the screen coding, and again in the 4 coding environments.

And, after testing in all 12 layer-environment combinations (which only the truly anal-compulsive B/As and programmers ever get around to doing), the corrected system is delivered to the end user. She tests it out, and discovers that, while the data in the field is now coming through ok from the database, its name on the screen is not correct. By analyzing the new data on the screen, she realizes that it's actually a "Policy Account Number", not a "Policy Number", and it needs to say so on the screen. Oops.

Ah, but there is good news! The system architect breathes a sigh of relief, as she tells the group that the layered design has now made its first payoff, since the name only needs to be changed in the presentation layer!

So, the presentation layer coder changes the name, and all is well, until someone discovers that there really SHOULD be a Policy Number on the screen, in addition to the Policy Account Number. Now, since the original functional spec was signed off long ago, and the software consultant who wrote it was let go long ago, no one now has the political nerve or wants to take the time to change the functional spec, and email a heads-up to all concerned. So, the poor DBA, who wasn't in the presentation layer meeting when the name got changed, is now rightly confused, since there already IS a Policy Number in the database. Someone in the user group, who happened to be in that meeting, corrects the DBA, that this is really the Policy Account Number, to which he replies, "Then why did we name it 'Policy Number', and not 'Policy Account Number'". Hackles are raised, and feelings are hurt, coz obviously someone screwed up somewhere back in the functional spec stage that EVERYONE signed off...but several rounds of emails now have to go back and forth on this, and what to do about it.

They finally meet and decide that it's better to just bite the bullet and change all 4 layers, instead of creating a kludgey, vague POLICY_NUMBER2 in the database. (As bad as this sounds to a theoretical design purist, trust me, this is sometimes seen as less painful than changing it in all the layers.)

Now, iterate the above scenario over 20 or 30 such seemingly minor functional spec defects. Does this start to sound like your system "birthing" process? Have you felt the "money-meter" running while your teams test those 12 environment permutations? Have you felt it running in those meetings, with all those expensive consultants in the room, and thinking, "Gosh, there just has to be a better way.", but not knowing what that way would be? Can you now see the unstated, un-accounted-for cost of layering?

I hate to say this, but for all of our software industry "progress", in some respects, we have gone back to the 70's. For example, in the 80's, Oracle came out with OracleForms, in which the database layer was right up against the presentation layer. A simple field change that, in today's state-of-the-art Web 2.0 layered design takes several people several days to complete, used to take the DBA and the OracleForms programmer just a few minutes. Taking several days to complete such a simple change would have been unacceptable; it would have put us out of business.

How did we let that simple elegance slip away? Why is that a so-called "primitive" 2-layer system is so much worse than a 5-10-layer system that ultimately falls over of its own simple weight before it ever goes into production? Why is "several days of effort" acceptable now, when it was not acceptable back in the 80's? Are you sure you're going to need to swap out ALL those layers someday?

I can only surmise that new programmers coming out of college lately, do not have exposure to the supposedly inferior "legacy" systems and tools, They have been so busy just trying to learn the plethora of new software tools inundating them almost monthly, that they have had no time to learn from the past - to learn from software design history, and so they have had to assume that any software design models of the past are completely unworthy, and have thrown them out altogether - the good with the bad. Like Warren Buffett in the investing industry, it would seem that only the "old salts" have the benefit of that historical perspective.

Are there times when layering can be beneficial? Yes. If you are refactoring a legacy system, one with which the users are already pretty much happy, and one which you believe will be around for many more years, then layering can add some nice "swap-out points" without a lot of time-cost overhead, since the column and field names, and the business rules that use them, have already been firmly established. Relatively few of them will ever need to change.

For new systems, layers can also be quite beneficial but you need to "factor in the refactor time". By this I mean that you need to initially write a system without layers, or more correctly, with just two layers: the database and the presentation. Then, after that has been pretty much fully accepted by the users, consider it the same as any other legacy system (See above paragraph on that.), and refactor the new legacy system into the various layers. Again, just be pretty sure that EACH layer stands a reasonable chance of being swapped out at some future but not-too-distant point.

In summary, we humans are good at "failing our way quickly to success", so our design protocols need to cater to our strengths, instead of punishing us for our weaknesses, as layering often does.