Why do software “integrations” suck?

If you ever worked in a corporate environment using multiple systems for various purposes chances are you probably heard about software “integration” a lot. But what does this seemingly magical word entail? Why should it impact me, and who is responsible if it doesn’t work?

Well I am hoping to be able to take you on a journey to find this out.

In order to understand what “Integration” is, we need to first take a look at a real world example, take 2 individuals each with their own needs and put them in an “escape room”. Now let’s analyse the options that they have:

1. Each one will solve each of the puzzles and eventually escape
2. They divide the work and then share the information that they revealed and thus not having to solve each puzzle

Which Option do you think will be slower and require more resources?

Which option do you think would make the task of escaping faster?

If speed and less resources used is what you are after than most probably you chose option 2.

The next thing we need to figure out is what is required to make option 2 work? The answer might seem obvious, but that one thing might be the single hardest thing to achieve:

“communication”

Communication facilitates the data transfer between the individuals. However in order to master it you need a lot more than good looks, and the following:

1. Air (obvious) but without it the voice would not reach from one person to another and they probably both be dead in matter of minutes. Paper / Internet / Phone for written forms.
2. A common language: any form of sound or sounds that have a well-defined set of rules which both individuals know. E.g. English language

And Finally

3. Relevant information. Crucial so that both concentrate on the same task at hand.

If all 3 are present we can consider that the 2 individuals have communicated successfully (and hopefully by now they have escaped and are enjoying a pint at the nearest pub). But wait a minute doesn’t this mean that we can consider communication to be the same thing as an integration? Yes, we most certainly can.  

To illustrate this let’s try to translate the 3 requirements for communication to requirements of integration.

1. Air, this is the equivalent of the internet
2. Common language, the methods and description of the Api
3. Relevant information, the information sent thru in the packet from one Api to another

If all this sounds easy and intuitive than why is it so hard sometimes to get this right? Why does it take so long? Most importantly why does it suck?   

I am sure everyone has their fair share of stories but based on my experience so far it is never about the software but rather the people involved. Yes, ladies and gents it all boils down to people. The ones using the software, writing the Apis, writing the documentation for the Apis, project managing the integration and countless people involved. Every integration is a story about communication.

Will improving communication stop integrations from sucking? In my opinion yes, it will. Wow that is quite a statement you would say but let’s just see how do you improve it?

Well there is one thing I have learned throughout the times and that is “how to eat an elephant?” and the answer: “one byte at a time”. Therefore let’s take a look at the individual processes that take up the “elephant” which in our case is an integration.

It all starts with a person asking for this to happen (usually called: stakeholder, sponsor, big boss man, big cheese etc. however for this document to continue to make sense let’s stick to “stakeholder”).

These requirements are often quite generic and have no or very little meaning in the IT world.

A few examples:

• “We need to get this data from that company.” – average case
• “We have signed a sales contract with this company and now we need to give them the service. Just make it happen”.  – worst case
• “Can you please give me a detailed report on how long and at what cost can we achieve integration with this company?” – best case

I can envisage this discussion taking place between the stakeholder and their most trusted IT person most probably the CTO (IT Director, Head of IT, Outsourcing company sales person etc. again let’s stick to the word “sponsor”). After this discussion the snowball starts rolling and more and more people get involved which requires more and more chains of communication.

Nonetheless to co-ordinate the effort and assure delivery usually the Project Manager (PM in short) is called in.

As you can see the PM (project manager) has a crucial if not the most important role in getting the project done successfully on time and on budget. Basically what the PM must do is to take the sales speech and translate it into IT speech. Oh and trust me the two are completely separate languages even though they are both in English. Once the PM has taken over than a new set of actions begin let’s take a closer look at the steps:

1. Creation of the requirements documentation (analysis)
2. Identifying the necessary people that need to be involved (scoping)
3. Creating a timeline (usually a Gant chart)
4. Setting up communication channels, meetings with middle management to make sure the resource is going to be available.

And most importantly

5. Set up communication channels with external resources and exchanging API documentation.

All the above points have only one thing word in common and that is “communication”. Beside a “good chap” the PM must be a good communicator. They will be the “glue” that holds the entire project together.

Finally all is well, the timelines and budget have now been approved and it is off to the races. The developers start their sprints, the sales people set up the contract in the system adding in the necessary configurations. Sometime in the future the developers finish writing the code, they give it to the testers to test and then sign it off and everybody is waiting the big “Go Live” moment.

In an ideal world the final step should be merely a formality, unfortunately however many of the things are often overlooked, poorly understood or just forgotten.

These are some of the reasons why:

1. poorly written requirements
2. poorly written or often obsolete API documentation
3. no staging environments
4. entry level testers (yes, I don’t judge but we all know that the junior developer is always the tester)
5. poorly understood reason for the integration
6. priority calls on other items that will take away resource from the project and pushing it onto a never ending delay

Gosh, I can bring up some negativity you would say, but here is the nice part: for every problem there is a solution. It is up to every company to identify and fix as they see it fit.

I am however convinced that communication would be a good cure for all of the above issues. 

Here are a few examples:

• Poorly written requirements: Using their listening skills the PM could identify all the requirements that have been enumerated and also put relevant questions to clarify the scenarios
• Poorly understood reason for integration: Every integration has its reason which is usually a commercial one. If the PM takes the time to understand those reasons and also presents it to the Business Analyst maybe they can get to the conclusion that same functionality already exists. Or better yet if the PM would ask the developers they could suggest ad already existing api that can offer the same data for free

If you thought that the above steps are many and complex think about multiplying these by 2. Exactly because the same process is happening on the other side. Remember there are 2 separate systems that need to connect using the same language. I will let you think about that for a few moments.

Ready to wrap this up? Thru the examples we have discovered that communication, although will not solve all your problems, it will most certainly help solve most of them. Having this in mind if you consider investing in a software integration please bear in mind to have your best communicator on the job!

All the Best

Zoltan Daniel