Seeing the Spoon
It's time to take the red pill
By Julian Browne on April 1, 2007. Filed Under architecture, business, delivery
This article is something of a preface for the whole site and, as with other posts, will be updated from time to time.
The intention for the site is to create a kind of holistic view of what can be done about the current malaise in software development in medium to large companies. That there is a problem is in no doubt. If you work somewhere that experiences a multitude of difficulties in making your business happy by giving them what they want within reasonable parameters, then I hope it's not too depressing to know the grass is not greener elsewhere. The problems may different, but essentially the end result is the same.
I'm not so conceited as to believe I alone have all the answers, but I do believe that answers to each aspect of what can go wrong do exist. From requirements to support and all the management above, there are companies that do bits pretty well. The Spoon metaphor from The Matrix is just my way of saying that if technology people can be a bit less introverted, insular, and technical about what they do and see their business as it really is (a feat that many in the business are not able to achieve) then the malaise can be lifted.
Life will never be perfect of course. Who would want it to be? I for one thoroughly enjoy the intellectual battle with happenstance events that conspire to defeat my best intentions. But I find it profoundly disturbing that technology teams in many places have become the whipping-boy of the business, beating themselves up and spiralling down into a culture that almost expects to fail before they've begun. That's not healthy and, without getting too touchy feely about it, people deserve better.
Some of the answers necessarily lie in the technology itself. Too many IT organisations just follow the herd and deploy application servers and middleware because some slick salesman did a good pitch, or say things like "we're a Java shop" or "we only use Microsoft", or worse we do X "because it's what everybody else does". Well that can't be right can it? If everybody else isn't very good then you want to be a bit different, but you want to be a bit different in a way that relates to your business.
A great deal of the answers lie in thinking less about technology and more how you can pragmatically add capability rather than blindly try and meet requirements or pursue alignment with an imaginary business strategy.
And many answers lie simply with people. You will never get a team staffed 100% by effortlessly cooperating mind-reading geniuses.
Here are a few of the themes you'll find reappearing throughout the site:
Software Vitality
Software is an integral driving force in most businesses these days. Just like other revenue-enabling functions, such as marketing or sales. It's more fundamental than HR or finance because, unlike these noble pursuits, it can be a differentiator where it underpins the customer experience. An article on The Register recently suggested that "around three-quarters of jobs in the Western world trade on an optimistic rhetoric of technological innovation" - so many are saying it even if possibly not following through with the requisite grass roots support.
Anything that artificially separates the kind of software that can make you successful, from the managers of the business can negatively affect the bottom line. That wedge could be outsourcing, or it could simply be cultural.
It's easy to dismiss companies like Amazon or Google as just "IT" companies, but they're not. Amazon is about anything but IT, and Google is an information company. What they have in common though is they look at software for what it can do to make them better, not for how it can meet predetermined requirements. That difference is important.
IT people are expensive. They are hired because they understand things that the business doesn't. When you pay an expert to do something there's an implied mandate for them to take an active role in applying their expertise and not to sit back and wait to be led.
Methodologies Suck
All methodologies suck. They really do. A collection of good practices is fine, but like a richly endowed tool chest, you'll always have to pick the ones that are appropriate to each occasion, be prepared to modify others, and learn completely new ones.
Of course you need a repeatable process, so that everyone knows where they stand, but you also have to be prepared to adapt to changes and think on your feet.
Nothing is new
Except for some very specific techniques related to modern development, nearly every idea that makes a significant contribution to creating great software was defined before 1975. To our detriment, many of those ideas have simply been ignored.
The Blind Plan
Hardly anybody understands what a plan really is. When the business needs X and you have Y developers of Z capability, then the earliest date you can deliver X is predetermined. No amount of wishful-thinking will change that. A good plan will tell you this date. Then you can try and adjust what X, Y and Z are (and any number of other parameters). Most plans aren't good though. They assume inaccurate values for X, Y and Z. Then it starts to look like you are behind plan just because you aren't doing what it says on the date it says you should be doing it.
Not Science. Not Engineering. Not Art.
There's no good analogy for the process of creating software that satisfies the science vs. engineering vs. art mantras. For the most part it's bugger all like engineering, and bugger all like science, although a little bit of an engineering and a scientific brain helps. It's more like the traditional craftsman role (for years I used the cabinet maker analogy) except that what you make is ultimately an intangible set of instructions for a machine to follow that can then be mass produced.
Incidentally, the term 'software engineering' is actually a wilful misinterpretation. It's a lot like some creative arts, particularly poetry, except that you can't see or study the body of work that's come before you, because 99% of it is legally and commercially secret.
Languages should be Idiomatic
Paul Graham was right when he said all languages are progressing towards Lisp (if you haven't read Hackers and Painters, go out and get a copy this instant). I've always hated the idea of knocking any one computational expression medium in favour of another, but if you agree that a language is just that - a mechanism for expressing your thoughts on how an information processing solution should manifest itself - then the most efficient, elegant and precise medium must be the best. I'm shit at Lisp, having only really scraped the surface, but everything else is already looking annoyingly limiting. It's not that Lisp per se is some development nirvana just that it best embodies all the characteristics to build a domain-specific language of your own.
Designs should be Idiomatic
Architects (and software managers for that matter) that don't understand coding are almost completely to blame for the present gulf between IT and the business. Being and architect (or manager) means you also take responsibility for the link between the two.
Coding in a way that's friendly to the future is hard. A business that's unsure what it wants next week makes it harder. The answer isn't in more buzzwords and techno-babble, it's in relationships.
Architecture should be Idiomatic
All mainstream architecture approaches are too narrow and, like programming languages, are evolving to something that is behaviour-driven. A business isn't just about services or adaptors or events. They are just the things we use to represent behaviours. If you can see the real behaviours, you can create an architecture that looks like the real business.
All ideas should have a believable story
One of the solutions to the cultural and communication divide between business and technology is to understand narratology. Work out, succinctly, what it is you need to achieve, and tell the story. If the story doesn't work - just like a bad - movie, nobody will be interested. If you can't make a good story, the idea itself is probably wrong.
It's always difficult
Doing software well is hard (see Donald Knuth's many presentations on this). Even if you do everything right there is no guarantee of success. If anything goes awry in any of the lower layers of the abstraction model, you can come unstuck, and things can go awry at the most unexpected of times.
Last update: January 2008