If an organisation developed all the software it needed from scratch, it would, in theory, never have a business need go unfulfilled. With full and free access to its own source code, changes would only ever be a matter of further development and enhancement. I say in theory because the reality is that some needs would require skills so specialised, investment so large, or numbers of people so great, that the business would find themselves wanting despite the tantalising promise of their hard earned code-base. So the reality is that some software you have to buy. Even the smallest of companies buy their word processing software and in large enterprises this extends to all manner of ERP, ETL, CRM, IM, and the like. Then there’s the supporting software layer, between the OS and the application - you wouldn’t write an RDBMS for instance.
All bought software has to be made to talk to other bought software. For this you need integration software. You could write this, but in complex environments (and you don’t need that many business process or systems to be on the wrong side of the complex line) you would probably want to buy at least the message bus, ESB, or whatever.
Then, when you have your software products in and working, the passing of time deals its mischievous cards and products go out of date, out of support, and need to be replaced. Some with similar products, others by multiple products and some with yet more bespoke integration or application development. Whatever you do, it’s a complex heterogeneous world to manage.
For every communication from application A to the outside world, your chosen method of implementation must consider whether a response is required, the size of the data payload, its business criticality, the number of interested recipients, the reusability of the content, whether it forms part of an overarching transaction, the quality of service required for delivery, the impact the order of arrival will have on business processes, and so many other aspects they won’t all fit here.
The good news is that architectural thoughts have been maturing quickly over the last ten years. Although EAI is a much maligned approach these days, it did at least force businesses to look at patterns of application interaction. Sites like those run by Gregor Hohpe have created a shared body of knowledge to deal with the problems even if there wasn’t much they could do to reduce their inherent complexity.
The Service Oriented Architecture applies domain-specific thinking to the basis on which applications communicate, augmenting the world of EAI messages with services, and the Enterprise Service Bus valiantly attempts to wrap both these up into a consistent and cohesive whole. More and more is being written about the Event-Driven Architecture - communicating changes of state, and the complex events that can be derived from them - so whilst a complex and demanding business, like trading on the finance markets, will always have complex and demanding problems to face, there are lots of things that can be done to meet the challenges using just the right mixture of messages, services and events.
My point is that the reality of life, developing and designing applications in the twenty first century and trying to find the right integration approach, is like juggling a brace of highly annoyed ferrets (we haven’t even touched on enterprise data and process models). So it makes me laugh when I read edicts from global architecture teams mandating that the whole business (a body which they seem to understand less than they do technology) will adopt a ‘buy not build’ approach. Of course I get that you should buy when building would add unnecessary risk, or take you into an area in which you have few skills. I know of one cable network provider that decided to write their own billing system to replace an assortment they had acquired over time. What lunacy persuaded them that they had the wherewithal to pull it off I don’t know. Anyone familiar with the space knows that it’s hard enough to manage and configure the ones you buy, let alone start with a blank page and start coding one. It naturally ended in disaster, but with so much spent on it that its half-built carcass continues to limp on to this day.
But exactly what value does it add to issue statements like that? Anyone attempting to spend the businesses money on writing a commodity product should be sacked, and where the services provided by a new application add competitive advantage, or act as a market differentiator, you may well want to build (or at least know what you will do with a bought application that will make you stand out from the competition). There is no hard and fast buy over build, or build over buy, there’s only common sense, and if you have to explain that to highly paid technology teams then I’d suggest there’s a bigger problem that needs fixing.
I laugh when I see these statements not because they are irrelevant and ineffectual, but because they miss the opportunity to address a real and immediate problem in integration competency - what many businesses actually do is buy a product, forget all good architectural practice, and proceed to customise the hell out it, resulting in the worst of both worlds. If you bother to run an in-depth RFP process and assess a third-party product’s capabilities and direction against your own needs, then the winner presumably represents the best fit. What you may not realise you should also be doing is agreeing to tailor your business to make best use of that product, albeit with some latitude. I have my own set of Word templates for creating technical documents, but whilst I don’t hold its thesaurus in high regard, I certainly don’t intend to try and integrate my own into it. I have to accept Word’s limitations and alter my behaviour by going onto the web to use other services when it’s not up to snuff.
Salespeople won’t tell you this. Manufacturers have learned that exposing some internal bus or API is just enough to allow the sales teams to make the promise of ‘seamless integration’. But how could it be that easy? Their API is based on their one-size-fits-all domain model, not yours. Even if you only pass static data in and out you’ll have to transform, enrich, and standardise it. If you start having bits of internal functionality elsewhere things will only get worse. But just having heard the seamles integration promise is enough of an impetus for The Project of Integration Doom to kick off, and the business to unrealistically expect old hacked-up legacy apps, databases and spreadsheets to fit neatly in, whilst all aspirations to non-functional requirements like availability and integrity remain intact. Ah I see you’re laughing too now.
There’s no simple answer. The twin policies of application portfolio management and enterprise integration need continuous and assiduous efforts to keep them harmonised and cost-effective. But there are a few key rules that can help:
Never issue pat statements like we have a buy, not build, policy. It’s insulting to the project architects and developers who keep your enterprise spaghetti going. It’s also not true, because you do both, so own up and face it.
Instead work with the business to identify the areas where building can really give them a competitive lift. A lot of them will have worked in similar companies before and they can tell you what others do badly. Embrace the idea of these targeted developments and don’t be shy about it. Don’t allow people to say “We’re not a software business, we’re a [insert business type] business”. You probably aren’t a software business, but software is intrinsic to making your business successful.
Accept that enterprise integration requires a competency centre, and give it the support it needs. It’s not sexy and it’s hard to relate it to direct business value, but it is foundational to everything you do.
Always buy products where there’s a risk you would fail in building similar functionality, especially where what the product does is a basic commodity.
Use detailed RFP metrics to check you are buying the right product, not just for now but for the future. Whatever anyone says, it will be around for years and years. It’s tomorrow’s legacy nightmare. I’ll be publishing a full RFP spec from my archives soon.
When you buy a product, focus all efforts to keep it as it comes out of the box. You won’t succeed 100% but you will limit changes. It may sound harsh, but often a little effort with the manufacturer will enlighten you as to how to use the base product better.
Only customise where the manufacturer explicitly says they are happy to continue to support it and where a version upgrade won’t clash with all your hard work. Something like ninety percent of the cost of any application is incurred post-launch. You don’t need to add to that.