API design in large systems

Good API design can be an enabler for systems development and management. Decoupling using API gives the power to build service oriented applications, effectively boxing in complexity and the need for knowledge sharing of what's under the hood. But is the strength of this design also it's Achilles' heal for efficiency ?

In a small or even medium sized organization, the set of consumers of APIs is limited to a few groups. However, for large orgs especially with legacy systems, the API services are the foundational data sources for sometimes dozens of systems, and this can cause significant delays in delivery, increased technical debt and missed windows of opportunity.

Here are some of the challenges with API based designs:

1. Reusability creates more dependent teams introducing challenges in project prioritization, lack of interest in upgrades, version control and performance nightmares - especially when competing consumers are batch and online users.

2. Project teams get pulled into upgrades that in many instances, don't reward them with significant business value.

3. Infrastructure and technology upgrades become exponentially expensive in cost, risk and resources.

4. Generic apis built for all tend to have unnecessary information, and require tribal knowledge to even understand how to use.

5. It becomes harder to troubleshoot and maintain complex software requiring more layers of observability, analysis and documentation. Quality analysis becomes hard because of lack of relevant data in lower environments.

6. Over time, the teams develop a tendency to become segregated into frontend & backend teams creating artificial bottlenecks and disruption in agile functioning.

7. Expensive capacity management, infrastructure, caching and security monitoring apparatus is brought in to remedy the symptoms but the root cause remains and keeps challenging the team - usually in the most impactful sales or tax season.

8. With people turnover - especially in technical leadership, vision and views quickly change trajectory. Since it takes significant storytelling to slip in tech debt work with all other priorities, the cost of shifting right results in 'bandaded', patchy software that slowly nips into the organization's productivity.

Here are some ways to potentially manage these challenges:

1. Treat APIs like any other software. Introduce strong version control with minor and major revisions, release schedule and end of life rules for engagement. Publish a predictable and consistent schedule so that consuming teams can plan their work year effectively.

2. Separate online and batch api's - they will never co-exist. When customers wait for the browser to load their transaction screen for more than 5 seconds, they have already developed a negative opinion about your brand value. So in the long run, the cost of redundancy may be less than loss of business due to user perception and experience.

3. Start somewhere small, but eventually move to data pipelines and event driven architectures like Kafka. APIs were meant to call data from SAAS providers over the web. But if your servers are in your VPC, there's low value in using http to get this data due to performance overheads and challenges in reconciling missed communication packets. Use pub-sub to make solution straightforward for the admin systems - they provide the event data and let consumers decide what, when and how to use it.

Love to hear how you are tackling this challenge.

- Rakesh Gope - Solution Delivery Leader

Note: I am a big fan of AI but to respect your time and attention, AI was not used to generate this writeup. The illustration was generated using AI.