Cost of Versioning and API or Service in the Data Economy
The Data Economy is booming, and much is being written about Software “eating the world”. many companies however, have not formalized a strategy on Developing and Versioning their API’s. In a recent Forbes article “Collaborate to Grow Says Deloitte Global CEO Barry Salzberg” MIT Sloan graduates Jaime Contreras and Tal Snir are quoted as saying “the peer-to-peer exchange of goods and services – is being called the next big trend in social commerce, and represents what some analysts say is a potential $110 billion market.” Last month, Information week did as entire special issue on the “Age Of The API“, the enablers of the Data Economy.
This exponential growth in API’s is however creating significant versioning concerns and many organizations are beginning to consider their strategy for API versioning as their current strategies become insupportable. For example, the business needs a REST version of an existing SOAP service for mobile access. Should the migrate the entire service and all the client Consumers of the service to a new REST API and end of life the existing SOAP service? Or perhaps develop new API and leave the SOAP service in place? Whatever the reason is for the change, is the best strategy to create new, update the existing, replace the existing or do something else? How many of these changes can they manage in a given period of time and what are the costs?
Mr J Dubray posted on his blog an excellent article on “Understanding the Costs of Versioning an API (or a Service)“. After reading it, I wanted to add a few thoughts of my own, along with a couple alternate strategies, not covered by Mr J Dubray.
The Knot is when the old API and all the Client Consumers are replaced with a newer version. This requires that the API be suitable for all clients and can require a higher level of complexity in API development and testing. When a new version of the API and Clients are released, the old version is replaced as part of the SDLC, requiring additional operations expenses, particularly if there are many clients.
Although every environment and API is different, this is generally the highest cost approach. After the initial development of the API and the clients, updates can be calculated by adding the cost of the API development, the Cost of testing the API and the cost of any additional infrastructure, deployment and operations. This is then added to the cost to update a client, test a client and the infrastructure and operations costs of this multiplied by the number of clients. If the number of devices or clients is 10, the multiplier is 10.
A note on testing. If testing is done through the UI of each client and not using an Automated testing tool similar to SOAPSonar (independent of client) the formula would actually be different. By Validating the API independently, for far more than just functional unit testing (performance as well), the API testing need only occur once. When automated API testing tools like SOAPSonar are not used, testing of the API is done as part of a far larger client testing process. QA issues identified (say performance for a given client), require a decision on if it is the client or the API that should be fixed, a fix possibly influencing other clients. Besides the additional time taken, the formula would then include API testing, repeated for every client.
Point to Point is when a new API is developed for each client or group of clients. This is perhaps the most common scenario due to simplicity and speed, however can lead to considerable complexity as the number of API’s grow. A SOAP API may remain in production, serving a client or business partner, while a new REST API is created for a mobile device. Multiple REST API’s may even be created for different Mobile Devices (1 for each client), or a new SOAP API created for a new Business Partner. Each API and Client pair is managed on its own independent SDLC.
The costs of the approach can be calculated by simply summing up the API development + The API testing + API Operations, infrastructure and life-cycle + Client development +Client testing + Client Operations, infrastructure and life-cycle and this then times by the number of API/Client pairs that need to be developed or upgraded. While its quick and easy to add a new API/Client pair, given the simplicity of the API, a change to a backend system, can require a number of API/Client pairs to be updated. Small changes to the supporting systems having a ripple effect.
Compatible Versioning is when all the clients use the same API and only the API is changed. For this to work, the API needs to remain “backward” compatible with all the clients. This can make for complex API’s and significant testing of the API, but greatly reduce the Client development and operations. Since there is no development or operations changes on the client side, the cost calculation is different. This strategy can however be the slowest.
With BYOD and Opendata, Open Government (gov 2.0), OneAPI etc a growing strategy is to develop the API and treat it similar to you would the entire application. Many organizations are now splitting API development from client development into 2 distinct teams. Managing its SDLC independent of the any clients that your organization may develop and that perhaps encourage other parties to develop their own clients. Social media companies like Twitter are great examples of this approach, but others are rapidly following. This is a very light model, with costs being the cost of developing, testing and deploying the API only. Testing the API obviously requires some tool like SOAPSonar, as testing via the client is not possible.
Client development and testing can be greatly aided by using a tool like CLOUDPort to create a virtualized mock service for developing and testing the client against. Free run-times virtual API being distributed instead of managing labs and data integrity.
The appliance Model makes use of ESB like functionality (software or hardware based) appliance or API gateway. The initial API remains the same, but through automated polices and rules, a customized virtual API can be created for a group of Clients. The appliance becoming a control point for changing API structure (SOAP to REST, Active Directory to SAML, Node removal or format changes etc). Depending on how appliance like the appliance is, this can be done without coding very rapidly. Appliances can help manage the versioning, leaving older versions in place for older clients, while newer versions are created, or supporting changes to the API, without effecting the clients. Additional functionality like security, encryption offload etc can also reduce operations and deployment costs.
Conclusion
An organization need not pick the same strategy for all API’s being deployed and mixing strategies is common. The prices for each component, varying in amount per organization, API and strategy. The intent with this post, is to aid in making the right decision on strategy by reviewing all the options.
Resources are however limited. Mr J Dubray highlights in is graph of costs, a point of inflection were he says “at this point the costs become unbearable, and every change in services become tactical.” I do not believe it is just for a single strategy, or necessarily just cost related, but a clear trend is there. At some point, organizations realize the cost, complexity and manageability cannot be supported by the exiting process, tools and workforce. Some strategies result at this point occurring earlier, and other later. This is represented as the strategy variance.