Your API is Showing: The Case for API Integration and Management Services
In many Universal Mind blog posts about digital transformation, we talk about front-end user experiences and development technology used to enable those experiences. This post is focused on another aspect of digital transformation, the importance of an API integration layer in architecting an enterprise solution.
The API, or Application Programming Interface, forms the backend surface with which front-end web and mobile applications communicate. Many organizations already have disparate legacy systems, built over many years, that provide some sort of API which can be consumed by a front-end application. The web or mobile application is often then expected to communicate directly with these different services to carry out a given task, such as placing an order, or making a reservation. In some cases, the client applications must also integrate with third party services to bring in data such as geocoding and mapping information.
Additionally, legacy services may be protected by different forms of authentication and the client app needs to work with each of those. The end result of this is a large amount of integration work between the client applications and the backend services. In addition, the integration effort is duplicated for each new application/platform that comes along.
The burden of direct service integration
Don’t Repeat Yourself
As illustrated above, the model of point-to-point integration can quickly get out of hand. Each client application must invoke several services in an effort to accomplish a single task. Tying the output of one service into another becomes the task of the client application developer, and moves the burden of dependencies and transactional logic to the client tier. That task has to be repeated for every new platform that has to be supported: iOS, Android, Web, and more recently, IoT (Internet of Things). The code for each client application platform has to be tested, and if changes to any core business logic or service orchestration becomes evident, those changes have to be implemented and tested repeatedly for each supported client application platform. This can significantly increase costs, facilitates unnecessary complexity, and negatively affects time to market for a specific solution. Where possible, client apps should be focused on business behavior and user experience, and not the integration or aggregation of individual business services.
An API integration layer should be responsible for this business and integration logic. This high-level API can then be exposed to the client apps in a form tailored to their needs. The client applications don’t need to know about the underlying granular business services and can potentially be insulated from internal service changes.
An integrated API layer
Lean and To The Point
Most granular business services are highly focused. A typical, granular inventory service, for example, is focused on delivering as much information as it can about a given inventory item. The focus is not on a specific business use-case or transaction. As a result, this service is likely to provide more information than the client application needs, and potentially, in a format that is not readily consumed by the client platform technology. In addition, the client applications may need to invoke multiple services in order to carry out a business transaction. The result is an increase in network traffic and unnecessary “chatter” between the client apps and the backend services. This issue becomes more critical in the context of mobile applications or end-points, where bandwidth is restricted and the high latency of cellular networks results in significant performance degradation when making several dependent network requests. Often, older services may not communicate in a form that is most efficient for mobile clients. Existing SOAP or XML based services add additional network traffic and complexity for mobile applications to consume and parse. Most development tooling for mobile applications is now focused on the JSON data format served over a REST API. Existing backend services may have been written years ago, when SOAP and XML were king.
The integrated API solves this problem by returning only the data required for the client’s use-case. Additionally, because the calls against the individual business services are done server side, the network traffic burden is not placed on the client application. It simply makes one call and receives a compact reply with only the data it needs. The API layer can return data in a format that is most acceptable to the client applications, such as the JSON format. The API layer can also be setup to return data in other formats, like XML, for clients that work best with that format. The performance benefits gained through optimally serving data to the client application results in a more responsive application, delivering a better user experience. These performance benefits are not just of value to web and mobile applications, as IoT devices often have even less computing horsepower than mobile devices and will greatly benefit from an optimized API layer.
The API integration layer allows the various applications using the API to authenticate via common mechanisms like OAuth. The integration layer can act as a “wrapper” to ensure the user is properly authenticated before the actual calls to the underlying business services are invoked. The integration services are the only services exposed to the “outside” world. This layer can also scrub incoming requests for any malicious payloads that might cause problems if passed along to the existing, internal core business services or applications.
In many applications special “API keys” must be used to access services. This is particularly true when accessing 3rd party services. In the diagram above, a cloud-based geocoding service is used to take address information from the client apps and turn that into latitude / longitude coordinates for locating nearby inventory. This geocoding service likely has an API key that we don’t wish to expose to the client side applications accessing the integrated service. By keeping this direct API access away from the client application, we “hide” the key and the underlying 3rd party service dependency from the client without concern that the API key might be hacked or discovered for use outside of our application. Shielding the API in this manner would also allow us to swap out one 3rd party service for another without changing the service interface for the client applications.
Measurement and Management of your API
The integration layer can be used to collect metrics on what types of applications, and platforms, are accessing your API. It can measure the number of requests within a given timeframe and provide statistics on response times associated with the clients consuming the API. Also of particular importance is the ability to measure the response time of the underlying services that are invoked by the integration layer. Collecting these statistics can indicate when the response times from an internal or external service may be reaching unacceptable levels and are therefore impacting the overall response time of the API integration layer.
The integrated API layer can provide a number of additional benefits including:
Response Caching: Requests to the API can be examined to see if they match prior requests within a recent timeframe. If so, results can be returned from a cache of previous responses rather than invoking the backend business services over and over again for the same information. This can greatly improve response time for requests that may be made by several users - like weather for a given zip code, or inventory data for a given SKU.
Rate limiting: Given that your API is likely to be internet accessible, it is susceptible to attacks by malicious users. Rate limiting can be used to govern the rate of requests the system will accept. This protects the underlying backend services from being overloaded with requests and hampering access for legitimate users of your API.
Monetization: The API integration layer is a great place to track the number of requests a given application makes. Some businesses generate income based on the number of requests made against their API. This is often the case for services which provide weather, financial, travel, and other similar information. For example, Expedia generates 90% of its revenue via its API. Sixty percent of EBay’s revenue flows through their API. (Harvard Business Review - January, 2015). Capturing requests at the high-level business layer allows you to limit access or charge per transaction to specific API users.
Documentation: Many API management tools also provide tooling to document the API integration layer for use by the developers. For APIs you wish to expose to external parties, the management tool may also include API “provisioning” features to allow new developers to access your API after completing a signup process.
API Integration / Management Solutions
While much of the above could be accomplished through the development of an in-house solution, enterprises are looking to focus on only the parts of the problem that are unique to their business. Other functionality like security, rate limiting, caching, analytics, etc. are often well served from a packaged offering.
Existing solutions such as Apigee Edge, MuleSoft Anypoint, Mashery, AnyPresence, and IBM API Management / StrongLoop can support these needs. Many of these platforms can be leveraged in a SaaS model to scale manually based on business need, automatically based on traffic demand, or as an on-premise solution for those organizations with data security needs demanding a local solution.
With any packaged solution it’s important to evaluate how your business logic and orchestration between business services is developed. Where possible, look for solutions which allow the business logic to be expressed in a commonly used language that your development team can readily support. This improves not only developer productivity, but increases the portability of your solution should it become desirable to switch API management providers. Increasingly, enterprise organizations are turning towards solutions like NodeJS for much of this work. Many of the packaged solutions identified above are able to support code written in NodeJS for business logic and service integration purposes. This capability should be a priority for anyone concerned in the long term maintainability of their integration solution.
API Integration and Management - Integral to Success
Hopefully this article has provided some insight into the value of including an API integration and management layer in your enterprise architecture. As demands increase for flexible and scalable solutions, the need for an optimized API layer becomes an essential part of any successful digital transformation.