API Registry and Service Registry

[Based on a post originally appearing at http://asanka.abeysinghe.org/2014/07/api-registry-and-service-registry.html.]

Registry acts as a core component in Service Oriented Architecture (SOA). Early SOA reference architecture named the registry as a service broker for service providers to publish service definitions, allowing service consumers to look up and directly consume the services.

Figure-1 SOA triangle

With the evolution of SOA, Registry started to provide more value, such as lifecycle management, discovery, dependency management, and impact analysis. The Registry became the main management, control and governance point in such architecture. It also became a vital component within the SOA Governance layer of the overall architecture. As a product, the Registry started providing three distinct functionalities – repository, registry and governance framework.

Repository functions to store content such as service artifacts, configuration and policies, the Registry to advertise them for the consumers to access, governance to build management control and policy based access over the stored artifacts and to connect people, policies and data.

image

Figure-2 SOA 2.0

Changing Role of Registry

With changes and challenges taking place in businesses as well as the technical architecture of the enterprise, the role of the registry has changed. Even while the technical definition of a service is based a standard (e.g. WSDL, WADL), the business definition of a service can vary from organization to organization. Therefore, a customizable definition for a registry artifact became a requirement. For example WSO2 Governance Registry contains RXT (Registry ExTensions) to define services. RXT provides a customizable service definition as well as sets the behavior of an artifact when an artifact is imported and operated in the governance runtime.

When it comes to governance, registry became the core design-time governance controller in the enterprise. Features such as Discovery and UDDi compatibility became more nice to have features than the practical usage of them. Having said that, runtime wiring based on environment metadata has emerged as a practical replacement for discovery.

In the modern architecture, Services implement business functionalities of an organization, APIs are interfaces for services that allow consumers to consume business capabilities.

During the last decade Services were developed using various service development standards, programming languages and frameworks. Services were designed and developed – and funded – in silos in each business/organization unit. This led to duplicate services in the same organization which violates the core SOA concept of reusable shared services. Services are more technically driven, designed and implemented by enterprise developers. If we look at a service catalog, more than 80% of the services perform some kind of a CRUD (Create/Read/Update/Delete) operation. Data Services is another common industry term used to describe the CRUD services. The remaining services implement business logic with the help of business rules and CRUD services. A small remaining portion of utility services exist to provide functionalities such as computations and validations.

Figure-3 service types

The technically driven nature of the services lead to unhappy consumers. As a result, new service implementations were introduced to the market by duplicating as well as avoiding reusability. Some enterprises started implementing wrapper services in front of the actual services and added an additional burden of maintaining a new service layer.

Emerence of APIs

Complex, rapidly changing business requirements for consumer apps have changed the expectations of the services. Consumers increasingly look at APIs to be:

  • RESTful
  • JSON based
  • Secured with OAuth
  • Follow WEB API design

Unfortunately most of the existing services are not compliant with these expectations leaving a huge gap between the implementation and demand. In a consumer driven market, APIs that do not meet the demand will not have value and will not survive for long. To meet the demand, technical teams started to write wrapper services in front of the actual services. But this created a huge maintenance issue as well as slowing down the time to go-to-market.

Using APIs as the service interfaces for consumers to invoke service functionality, resolves the issues we identified above, mainly converting technical driven services into business friendly APIs as well as implementing the common reusable services for the enterprise and meet the demand/expectation of the service consumers. There is much more value proposition from APIs to the enterprise. Lets now look at them in-detail.

Figure-4 service and APIs

A pure API layer might be not enough to cater the demand. A mediation layer might be required based on the gap between the services and API. As I explained earlier APIs are light-weight interfaces that do not represent any implementations. Traditional Facade patterns came to the API architecture as API Facades, providing a solution to this problem. Introducing a mediation layer with the API Facade will take care of protocol switching (transports/message formats) as well as security bridging. In addition, it helps to convert the traditional backend services into modern web-api designs by using techniques such as service chaining (light-weight orchestration). More information can be found in this blog. Having said that, API-ready backend services can be directly exposed through the API management layer as an API without going through any additional mediation layer.

There are two main usages of APIs, internal and external. External APIs create an eco system for a business to expose their functionality to their customers and partners in a consumer friendly, secured and governed manner. Internal APIs again help to resolve the broken SOA pattern and provide reusable common functionalities across business units. This allows business units to promote and “sell” their services across the enterprise and maintain and manage by the business unit itself.

Design-time Implications on the Registry

I hope the information provided above is insightful and has helped you identify the usage of the registry and a clear differentiation between services and APIs. Lets now look at the key discussion points of the service registry/API registry. Looking at a reference architecture will be more helpful to identify the concept.

Figure-5 service and API registry reference architecture

Service definitions will be defined in the service registry, the service registry will maintain the additional metadata about the services and catalog detailed technical definition of the services. Usually service definitions is defined automatically when services are deployed to the service containers, but in some enterprises this happen as a manual process by importing the service definition from various catalogues or service containers. Once the services are defined the registry will create the dependencies, associations and versions of these services and metadata.

We spoke about having a mediation layer to bring non-API ready services into API ready, proxy services defined in the mediation layer. This will also go as service definition in the service registry.

While technical definitions of the services and proxy services are defined in the service registry, consumers of the APIs require a place to lookup the services. This is where the API registry will come in to the picture as the consumer facing API catalogue.

Target audience of the APIs are application developers, hence an API registry requires and supports publishers to cater to their expectations such as social features and the ability to subscribe and get an access token.

From an API governance point of view, API publishers should be able to secure the APIs by providing access control to the APIs and resources inside the APIs. Some APIs might require additional control with entitlement and workflow support to get approval for subscriptions before providing an access token.

Runtime Implications

We already discussed the functional requirements of the two registries, lets now look at the runtime view. Deployment will depend on the nature of the APIs. There are three categories.

  • Internal only APIs
  • External only APIs
  • Internal and external APIs

To facilitate the first category deployment can combine the service registry and API registry to run in the secured network (LAN). But this will provide two different views for the API consumer and the Service developer. External only APIs require the API Registry to expose externally (in DMZ) and service registry to run in the secured network. Internal and external APIs require an external (public) API registry as well as an internal (private) API registry. The internal API registry can combined with the service registry.

Figure-6 deployment patterns

Conclusion

I hope the information described above helps you identify the difference between services and APIs as well as the benefits of separately architecting a service registry and a API registry. Having two registries helps fulfill the requirements of the API consumer as well as service developers, in addition, this allows you to decouple services and APIs by having an individual life cycle and versions for each of them.

Asanka Abeysinghe
Vice President of Solutions Architecture
Blog: http://asanka.abeysinghe.org

WSO2 Expands Bay Area Presence

We outgrew another office. This is becoming a trend! Our Palo Alto office served us well for the last couple of years but with our accelerating hiring we needed more space.

The new location, 787 Castro Street in Mountain View, is just down the road from our old office, but provides us all the local amenities of the coveted and vibrant Castro Street corridor. The office forms a lively and productive environment for our growing team, as well as giving us more room to grow.

Keep a lookout for a public open house invitation towards summer’s end as we get settled in.

Really curious? You can see a few snapshots of the space and this week’s small dedication ceremony on my Google+. You might even be inspired to take another look at our Careers page as well.

-Jonathan Marsh
VP of Strategy

WSO2Con Insights – BarclaycardUS Optimizes Backend Services and Performance Across 10 Distinct Environments with WSO2 ESB

As one of the world’s largest and most respected financial services companies, with partnerships that include over 60 best-in-class companies and brands, BarclaycardUS is dedicated to making the purchasing experience simple and rewarding for its customer community. A key part of serving those customers is working with backend service providers using multiple protocols at very high volumes, explained Alex Brown, BarclaycardUS group lead, in his presentation at WSO2Con 2013 US.

For BarclaycardUS, the solution has been to integrate to integrate WSO2 Enterprise Service Bus (WSO2 ESB) with its existing service-oriented architecture (SOA), and leverage REST APIs with the ESB to boost performance and monitoring across 10 distinct environments.

Different Partners with Different Domains

BarclaycardUS’ partnerships involve handling credit cards for large companies such as Apple and LL Bean, and specializing in the backend services for these types of businesses. Additionally, BarclaycardUS works with different vendors to conduct its credit checks, rewards and fulfillment.

“This creates an interesting perspective, since we have to integrate with a lot of different partners with unique and different needs,” Brown said.

AlexBrown-Barclaycard1According to Brown, the company has to accommodate applications and services relying on SOAP, REST, Android and Apple iOS mobile operating systems, Voice XML, and OFX, along with many different APIs. In the wake of increased cross-domain orchestration, BarclaycardUS realized the need for an ESB to serve as a common backbone for connecting these different services.

After exploring various offerings on the market, BarclaycardUS decided on WSO2 due to its ease-of-use and comprehensive middleware stack, which represented endless possibilities for growth.

“The other open source platforms we looked at during the time didn’t have that, and WSO2 was very complete and robust and supported all the modern protocols, which was a big advantage for us,” Brown explained.

Strengthening Services with WSO2 ESB

Today, Brown noted, “We’re leveraging the ESB and different parts of it, and we have a lot of different use cases.” At WSO2Con he reviewed three of those use cases to highlight how WSO2 ESB is enabling BarclaycardUS to optimize business services.

Prepaid Platform is a new service the company has rolled out, which works as a mobile application available on iPhone and Android application stores. It supports balance inquiry and mobile application and origination, Brown noted. The mobile bill payment platform also is leveraging external prepaid vendors.

“To do that, we had a mobile app that was talking in REST to the ESB, and behind the scenes AlexBrown-Barclaycard2 we had to orchestrate to many different systems,” Brown explained.

Core Domain Services: are hosted in many locations and pull data from data sources and different vendors. As a result, company wants to gain control over what is happening and have a cohesive view of service APIs in the organization, Brown explained. The company’s goal is to have customers, accounts, devices and some of these core domain services facade with each other. Working with the ESB, the company has an effective intermediary to do this.

“In the ESB, behind the scenes, it’s talking to many different things like partners, existing services we have, and different vendors—this is what we’re heading towards and is in production now,” Brown said.

Account Aggregators is a new service that is currently in development and will go live soon, Brown noted. It will address the challenge of screen scraping, programmatic collection of visual data, which can often be a heavy process that requires aggregators to login, pretend to be a customer, go through and load Web resources. BarclaycardUS didn’t want these aggregators to take valuable processing away from its regular customers. With the WSO2 ESB, BarclaycardUS can have its aggregators support the OFX standard used by banks and boost performance.

“In the ESB, the aggregators are coming in and are authenticating themselves, and I’m loading all of this data at exactly the same time. So the aggregators were able to get the info off of our website, scraping it in a minute or two—this takes less than a second, around 900 milliseconds—we’ve loaded 100% of that customer’s information and given it to the aggregator,” Brown said.

He added, “We’re also using the throttle, so if they’ve indicated they won’t go above a certain threshold but there are multiple aggregators, we need to make it so they don’t accidentally go above that threshold.”

Investing in the Future

In addition to WSO2 ESB, BarclaycardUS takes advantage of WSO2 Governance Registry and WSO2 Business Activity Monitor. Looking ahead, Brown noted, the company plans to further enhance existing services and the use of coesb-logo-h42mposite services.

“The goal is to make this a cohesive API,” Brown greg-logo-h42explained. “We’re finding that we have a lot of services that had to be handwritten, which involves talking with bam-logo-h42our backend service providers and developers, and taking the time to test and deploy.”

BarclaycardUS plans to integrate the WSO2 Identity Server into its system to implement OAuth for RESTful services, which will be important for mobile applications. Additionally, the company is looking at the potential to leverage the WSO2 Complex Event Processor to help manage business operations and events streams and the WSO2 API Manager to gain insight into metering and monitoring of different service consumers.

“It sounds like we’re going to get exactly what we need with WSO2 in 2014, it’s a very cool product we’re going to be playing with more.”

For more information about how BarclaycardUS works with different backend service providers across 10 distinct environments using the WSO2 Carbon enterprise middleware platform, see Brown’s WSO2Con 2013 full presentation.

WSO2Con-US-2013-Powering-an-enterprise-with-messaging-and-APIs

WSO2Con Insights – Accenture Solves IT Integration Challenges Using the Cloud-Enabled WSO2 ESB to Support the Accenture Cloud Platform

As the world’s largest independent technology and outsourcing services provider for nearly 25 years, Accenture has extensive experience in helping global clients navigate through the
complex cloud choices in the market. Igor Mameshin, a manager within the Accenture Cloud Platform (ACP), explained in a presentation at WSO2Con US 2013, how the company is using WSO2 technology to help address today’s cloud demands.

Mameshin explained that, as companies use hybrid cloud integration—combining private computing resources and public services with integration touch points between the two environments—new and more complex risks arise.

For Accenture, the solution was to build a cloud-based infrastructure that accelerates the successful adoption and integration of cloud services, without compromising quality, standards or security. The Accenture Cloud Platform, powered by WSO2’s enterprise middleware platform, Mameshin explained, provides a rich portfolio of application adapters that simplify the connectivity to external systems such as ERP applications like SAP, Salesforce.com, social platforms and other integration endpoints.

The Service Integration Challenge

“Governance was a challenge in the past, but the cloud introduces new and even more complex risks. Service interoperability is a critical problem to solve,” Mameshin observed. Without consistency to this solution, IT users take over the issues, and a company loses control over defining any IT infrastructure within its organization. Multiple business units in the enterprise go to multiple public providers. “Thus, you never know what goes on within the enterprise,” he said.

IgorMameshin-AccentureTo address integration challenges around connectivity, system configuration issues, application customizations, non-functional requirements, reliability, scalability, and security, Accenture realized that the solution could not rely on existing on-premises integration tools. The resulting solution was the Accenture Cloud Platform, a cloud-enabled middleware platform based on a SOA, to better implement the interoperability between the cloud and on-premises environments for its customers.

By acting as both a governance entity and a cloud broker, the Accenture Cloud Platform allows an enterprise to provision services in public cloud providers. Accenture currently works with Amazon Web Services, Azure, NTT Communications, and Terremark. A proof of concept was developed to explain the integration between Salesforce.com and SAP ERP using a cloud-based implementation of WSO2 Enterprise Service Bus.

Integration Approach Overview

Accenture has employed WSO2’s Enterprise Service Bus as a cloud-based integration backbone to pass messages from system to system, Mameshin explained. This has provided agility and flexibility to adapt to future change and growth. Moreover, it provides location transparency, transformation, routing, and protocol conversion, and it has helped with security issues, reliable messaging, and monitoring and management.

The platform has a management portal that provides a self-service interface where customers can engage with the service catalogue and pre-integrated multiple services, and order cloud services through this front-end. WSO2 ESB is installed on one of the virtual machines that Accenture has provisioned in the entity public cloud. IgorMameshin-Accenture2

Within WSO2 Enterprise Service Bus is WSO2’s Salesforce.com Certified Force.com Developer (SFDC) mediation library, which contains a set of ESB templates and offers operations that simplify a customer’s interactions with Salesforce.com. Supported operations include log in, log out, create, update, Query and QueryMore. On the Salesforce.com side, Accenture used SOAP API, REST API, and the WSO2 Salesforce.com Connector to work with these APIs. A new feature of the WSO2 Developer Studio integrated development environment (IDE) allows for developing integration flows when integrating via the Salesforce.com connector.

Mameshin noted that an advantage of WSO2’s application-level integration approach, which uses real-time integration based on the native application’s integration frameworks and APIs, is that it preserves the application’s data integrity. It also allows for real-time integration between Salesforce.com and SAP, he said, and provides application-level security and an audit trail.

In implementing WSO2 Enterprise Service Bus to create Accenture’s cloud platform, Mameshin concluded, “We established that WSO2 ESB is a powerful and stable product—it did the job quickly and correctly. It has been working for three months without needing to re-boot to fix any issues.”

For more information about how Accenture uses WSO2 products to power the Accenture Cloud Platform, see Mameshin’s WSO2Con 2013 full presentation.

WSO2Con-US-2013-Using-WSO2-ESB-to-integrate-Salesforce-com-and-SAP-ERP-Accenture (1)