Layering

When Flexibility and Modularity Matter...And When They Don't

2008/03/23 There is a simple, common-sense design principal in computer software engineering that says: "Separate the things that change from the things that don't."...

...and some brilliant, flexible, elegant software systems have been written with this simple premise as their driving philosophy.

The problem started when the "inmates" (the techno-geeks) took this principal too far, and decided that almost anything COULD possibly change, and therefore SHOULD be "soft-coded" (usually in XML). "Flexibility and modularity" became the mantra, and systems with "ultimate flexibility and modularity" became the Holy Grail of software design. This culminated in XML-configuration-based frameworks such J2EE initially, and then Struts and Spring.

At some point, "heretics" such as "Crazy Bob" Lee began asking why it is necessarily a good thing to be programming in XML, in which one loses the protection and power of a robust compiler. Predictably, this heresy initially caused much fury from the high priests, but common sense prevailed as usual, and lately, the pendulum has swung back (for the good, I believe) with the POJO (Plain Old Java Object) revolution and annotation-based programming (in which pre-compiler-like statements are added to the compilable code). However, to date, I have not seen where anyone has adequately explained the essential flexibility/modularity conundrum. Allow me to try:

Imagine that you ask a home building contractor to design and build you a 3-bedroom 2-bath house. Not knowing what product standards will emerge over the life of the house (50-100 years), the contractor tells you that, in order to save you from the future aggravation of swapping out various parts of the house as future product standards change, she's going to build the house with "ultimate flexibility and modularity" as a driving design criterion. Since you, the home buyer, have often heard that term bandied about on the TV ads, the radio ads, and in the home section of the newspaper, you naturally assume that this flexibility and modularity is a good thing to have. After all, "everyone else has it".

So, the builder adds window framings, in which square windows that open and close up and down, can quickly be swapped out for round windows that open outward. She adds "flexible, modular" roof joints, so that the entire hip roof can be swung out and swapped out for a flat rubber roof. The builder modularizes every room in the house, so that with a few turns of a screwdriver, the room can be completely unhooked from the rest of the house, lifted out through the now unhinged roof, and replaced with an entirely different room, complete with furniture already pre-installed.

The common-sense question that any rational buyer would immediately ask is: Well, just how often do people actually change out all this stuff?...and the answer comes back: Well, no one has actually had to change out anything yet, but a lot can happen in a hundred years, and surely you MIGHT want to change the roof out someday, don't you think?

The essential problem in the software industry, unlike the home building industry, is that the average software "buyer" does not know enough about the underlying technologies to ask such otherwise common-sense questions, or if she happens to know a bit, she gets put down as lacking the technical savvy to understand the underlying software design issues.

There is, of course, a price to be paid in both time (code maintenance time AND system performance time) and money for this added "flexibility and modularity", and just as in home design, that cost-benefit needs to be weighed by an assertive buyer, who is not easily techno-bullied, and who, unlike the builder, has no personal financial incentive to add it.

That being said, there truly are design points at which flexibility and modularity have a HUGE payoff. In fact, most of software design since the 70's has moved in this general direction, which is a good thing.

Flexibility:

What does this really mean? In wikipedia, we find the following definition: "In systems theory, which has applications in diverse fields including biology, ecology, psychology, economics, and management, the flexibility of a system is related to its adaptation to a new environment or its resilience in recovering from a shock or disturbance."

If we then define "super-hard-coded" as living in the machine code or hard-wired into the machine, and "hard-coded" as living in the compilable code (java, COBOL, C#, and so on) and "soft-coded" as living in the XML configuration, and "super-soft-coded" as living in the database, then as a general rule, the more frequently something changes, the more "flexible" it needs to be, i.e. the more "softly" it needs to be coded. The converse is also true, i.e. the less frequently it changes, the closer it should live to the machine level, ultimately to the point of hard-wiring it into the circuitry of the machine.

Here are some change dimensions to think about:

The thing changes for each location. How often does the thing change whenever the system is deployed to a new location? If the answer is "seldom", then it is often easier and quicker (in terms of both maintenance and performance) to make the change to the code and re-compile, although many software shop managers would cringe at this heresy! If the answer is "every time", then the thing needs to be kept in an XML configuration file.
The thing changes for each customer ("custom software"). How often does the thing change whenever the system is bought by a different customer? Are new customers being added once a year, or once a minute? If the former, then it would usually be more appropriate to make the change in the compilable code. If the latter, then it would usually be more appropriate to make the change in the database.

Modularity:

Now, let us consider modularity. Take the hot-selling Apple Ipod, for example. It has only two "ports" (connection holes) in it: One for the upload/download/charger connector, and one for the earphone (or other output) jack. Compare that with the typical 1980's component stereo system: An AM/FM receiver/amplifier, with dozens of ports in the back for various connections out to other "modular components". I can add a DVD "module" to it (still!), a CD module to it, a record player module, speakers, and so on. I can hook it up to most any kind of TV, using either the S-Video jack, the RCA jack or the cable-TV jack. It is highly modular (and flexible), with LOTS of wires running out the back to all sorts of modular components....but can I pick it up and put it in my pocket, and listen to its music while I walk, as I can with my tiny Ipod? No.

Now, I'm not saying that my old 1980's component stereo system was a bad design. In fact it was a great design and it still works great...for me...but for all it's planned-in flexibility and modularity, I cannot even sell it for $50 in a garage sale. It's going to end up in the dump, apparently.

So, my point is this: Don't COUNT ON modularity to pay off for you in the long run. If you reasonably think you will need to be swapping out modular software "components" in the short run, then by all means, design for those modular components, but most of the time, after a few years, the "cool" modules of yester-year become totally irrelevant this year, and the entire system, or at least large parts of it, will have to be "refactored" meaning overhauled - re-designed and re-written. Programmers, especially we software consultants, just love this part, because of the long contracts and large sums of money paid for the refactoring contracts, but I believe we as a group, spend far too much time over-modularizing (AKA "over-engineering") systems.

Over the life of the product, what stands a reasonable, common-sense chance of needing to be swapped out? For those things, modularity can have a huge payoff. For the rest, it, like the future is a crap-shoot, and one should err on the side of simplicity, and write POJOs in a TRUE object-oriented design.

Although the above, once understood seems like common sense, if you peek into some software "houses", you would be amazed at how many things which almost never change, are kept in databases whose performance is many orders of magnitude slower than compiled code. Open the door and walk inside and you will often find a house whose contents are so convoluted and confusing that even Dr. Seuss could not draw them. If you start asking why things are so convoluted, be ready for the ensuing fury.