By Asanka Abeysinghe
Vice President – Solutions Architecture, WSO2
There is no finish line in a digital transformation journey. As you adapt to one technology, the need to change to another one is on the horizon. The goal isn’t so much to reach a certain level in leveraging digital technology as it is to systematically develop the capacity to continuously adapt to the new possibilities technology provides. Agility and rapid application development are key indicators of an organization that is prepared to sustain digital transformation. At the same time, enterprises need to address their organization’s culture: internally transforming the team to generate innovative ideas while in parallel externally transforming themselves to provide a better digital experience to consumers.
In this white paper, we will look at a pragmatic and repeatable way to plan, build, and deliver innovative ideas with an iterative approach, spanning both technical and non-technical elements of a digital enterprise.
Let’s start with a real-world example: NASA's Hubble Space Telescope, which was launched April 24, 1990 and offers the best real-world iterative engineering project we know. The catalyst for the project began more than a century earlier in 1918 when the Hooker Telescope went into operation, providing astronomer Edwin Hubble the tool to discover the expanding universe. The concept of a space telescope first emerged in 1923 and then evolved to the vision of a large, orbital telescope in 1969. By 1981 the Space Telescope Science Institute had been built as the astronomical research center for NASA’s largest, most complex, and capable orbiting telescope in honor of Edwin Hubble. In 1990 Hubble Space Telescope deployed into orbit; three years later scientists on the shuttle Endeavor installed two corrective lenses on the Hubble telescope. Since then, five more missions have served to replace parts, upgrade the computer, and provide other enhancements, and the evolution continues. This is a perfect example of an iterative approach.
As we can see with the Hubble, the key to an iterative approach is “thinking big, but acting small” and making steady, and self-correcting progress toward the end objective.
Figure 1: Timeline of hubble mission (https://www.skyandtelescope.com/astronomy-news/hubble-readies-for-full-operation/)
The initial architecture of Hubble left room for innovation to take place in the future by keeping both the overall vision and areas of uncertainty in mind. That mindset helped the Hubble team’s scientists to be innovative and build a product—the stunning results for which the telescope is famous. Each release in the Hubble “release” history has provided a significant increase in value of the product.
The same concept can be applied to technology adoption. Building a successful digital enterprise requires changes in technology to be adopted in an iterative manner. To this end, organizations need to consider some key factors. For instance, an enterprise’s business and IT objectives should be closely aligned first; this needs to be followed by empowering employees to adapt to change and thus build a digital workforce. The next key step involves picking the right technology to help the enterprise create an innovation platform. The following section explains these components in detail.
Digital transformation requires an alignment between business and IT. However, technical and business requirements and their respective stakeholders are often imperfectly aligned. Therefore it is important to adopt an iterative process.
Generally, the business side will aim to provide customers with a significantly different digital product at once, which could result in lengthy project timelines. Moreover, the business executive may have a concept for a new product (using this term broadly), but it can be a challenge to think through and communicate all the required details that an implementation team needs to fully realize the idea. Therefore, what is eventually delivered might not meet the business executive’s expectations; end users may not find the product attractive, or the environment in which the product was conceived has changed, limiting its value.
In a traditional waterfall-based development process—where requirements are developed, but cannot be changed or validated until the entire implementation is complete—it takes a long time to see results and assess whether the concept or implementation succeeded or failed. WIth a built-in resistance to change, the waterfall-based approach creates frustration when project teams invest a significant amount of time and effort to deliver against requirements and environments that turn out to be invalid. Those are failures that an organization cannot afford if they are to keep up with today’s pace of innovation.
Business and technical leads working on a digital transformation project have already realized that the primary challenge of this exercise is to create a digital workspace comprised of proper processes, systems, and an efficient and productive workforce. Digital brings changes, and by nature, many people are reluctant to change. Engaging, empowering, and entrusting the workforce helps to overcome that challenge. Applying the same iterative approach to the change involves moving people to digital projects where they are most productive.
An iterative approach to create a digital workforce starts with building a small team by picking a skilled and smart set of people to deliver a strategic project for the organization, and one that provides a digital experience for end-users. Enterprises often name this team the digital-core, a-team, or dream team. Let’s refer it as digital-core in this instance.
Figure 2: Digital-core
The digital-core team should directly report to the technical and business sponsors of the organization's digital transformation strategy. He or she can be the chief technical officer (CTO), chief information officer (CIO), or the newly identified role digital transformation officer (DTO). The successful release of the digital-core team's first project provides the required visibility for this team as well as incentivizes them to implement the next concept. Digital-core will become the advocates and evangelists of the digital transformation strategy and they will encourage and adopt more people to the new process.
Figure 3: Digital teams
If an organization operates in a podular structure, bringing a digital-core team and teams around that are easy. There are several reasons for this - pods are responsible for delivering some business functionality based on the key business domain of the organization; pods provide a fundamental concept of digital transformation, i.e. plan, build, and run an idea; and pods are independent and self-operated, and hence changing and implementing something new can be done more easily.
Figure 4: Podular organization
Source: ‘The Connected Company’ by Dave Gray (https://www.xplaner.com/connectedco/)
Being podular creates the foundation to establishing a digital workforce, but how do you go beyond the digital-core? Many organizations are often stuck at this stage - either moving slowly or just remaining stagnant. Designing and executing a proper evangelizing and onboarding program helps to increase the portfolio of the digital workforce. Running workshops, seminars, and having an internal Wiki to find information will help to educate people about the digital strategy. Providing sandbox environments, samples, and conducting hackathons will boost the interest of technical people. On top of these activities, building a digital platform and bringing the pods to operate on the platform to plan, build, and run innovative ideas are some key points that will be discussed in detail in the latter part of this paper.
Evangelizing the innovation platform internally will make the business and technical stakeholders aware of the platform. Offering a proper onboarding program to the platform, introducing training programs and sample applications, and last but not least, enabling innovation platform capabilities via APIs will help to bring more people into the digital workforce. Strategic leads of digital transformation are required to work closely with the human resources department when implementing these changes and building the digital workforce.
This section will explore how you can take an iterative approach to the technical aspects of digital transformation. It will explain how you can be iterative at a high-level in the innovation platform and at a low-level in innovation projects. Let's explore details of the innovation platform first as having an understanding about the platform will help to apply the iterative implementation approach.
Figure 5: Strategic innovation platform
The innovation platform framework is common for most enterprises while the specific needs can vary based on the business domain and size of the organization as well as the IT budget. However, to build a successful innovation platform, the company's business strategy and digital strategy should be closely aligned. There are four aspects that need to be considered when developing the high-level innovation platform strategy; these include internal and external consumers of the platform, current IT infrastructure, and future IT plans. These areas should align with the business and digital strategy as well as cross functions in each sector. For instance, consider an organization that has legacy applications, data, messaging and integration infrastructure as well as people working in those areas who can put in the current IT infrastructure. Plans to incorporate new applications, such as mobile apps, web apps, and IoT solutions, architecture paradigms like microservice architecture (MSA), and technology enhancements like containers and cloud can be added to future IT strategies. Internal consumers and external users consuming the current applications as well as the target audience in the future IT strategy can be included in the internal and external consumer boxes. Moreover, apart from who is and who will be using the applications, their behavior, usage patterns, and security requirements too need to be captured.
Figure 6: Gartner’s Pace-Layered Application Strategy Source: How to Develop a Pace-Layered Application Strategy Refreshed: 24 August 2016, Luis Mangi, Dennis Gaughan
Gartner introduced a pace-layered architecture by realigning the concept of the system of systems. This conceptual structure is an excellent base architecture framework to consider when building your innovation platform. If you ask the question why? The pace-layered architecture is about reusability of current systems, bringing new systems to be innovative, and implementing iteratively.
There are many architecture representations for the digital enterprise. We took the most popular layered architecture pattern to explain this in line with the system of systems concept. Having a holistic view of the architecture approach will help to look at how iterative architecture can be used to implement the same.
Figure 7: Architecture view - digital platform
The system of automation provides an environment to run tests of the architecture layers, and various automated processes associated with the software development lifecycle, such as continuous integration/testing, environment provisioning, and auto scaling. This layer includes the traditional platform as a service (PaaS), container as a service, and container orchestration technologies. In addition, devops scripting, such as Puppet, Ansible, Vagrant, and system level monitoring tools too are a part of this layer.
A Source System of Record (SSoR) represents the physical data storage layer that represents file systems, relational databases (RDBMS), no-SQL, and cloud storages. The system of records is the layer that provides access to the data stored in SSoR. Data access is provided in various channels, such as data APIs, functions, application software, and even database programs like stored procedures.
The system of integration is the most important layer in the digital enterprise that represents the business logic and interaction with internal and external data and processes. Components are represented in the MSA, and service-oriented architecture (SOA) resides in this layer. Implementers of the digital enterprise will spend most of their time architecting and implementing the system of integration. Let's deep-dive into this vast architecture layer.
The system of engagement contains the numerous delivery channels that end-users, i.e. the consumers of the digital enterprise, interact with. These can vary from traditional Web and mobile applications to modern smart devices, gas stations to connected vehicles, and AI systems. As identified, the digital experience is the most important factor in digital transformation, and digital architects have to pay more attention to this layer as well as innovate and evolve this layer based on consumer feedback and demand.
There are two common iterative approaches because one method will not be applicable or practical for each enterprise. Organization culture and immediate business needs have to be considered when picking one method over the other. However, a group can find its own way or stick to a mixed mode after establishing a steady start using the techniques described in this paper. Let's look at two popular methods we discovered when building innovation platforms for small and medium-scale enterprises (SMEs) as well as large firms.
Figure 8: Vertically iterative
The vertically iterative approach mainly introduces different architecture layers based on the priority of business requirements. SSoR and a system of record already exist in some format in most enterprises. Bringing the system of engagement and directly connecting with the system of record layer is an approach for a business that’s required to give a quick consumer experience by using existing data and logic. Thereafter, incrementally introducing the rest of the layers to bring in the system of integration next and the system of automation thereafter is a typical pattern we observed. This approach is a good fit for SMEs because the scope of the business and technical functionality are limited.
For example, consider an organization that’s looking to replace all its paper-based forms with an online form. They can build a web module by directly connecting with the databases and start accumulating data. Then they can introduce integration in the next iteration to distribute the collected data to other systems. The third iteration can include adding a set of data-APIs by properly decoupling the web module and the data layer.
Figure 9: Horizontally iterative
The most standard iterative approach is to implement a small portion of each architecture layer and introduce more capabilities using an incremental plan. The scope of each iteration is decided based on the project’s onboarding to use the innovation platform and the priority of requirements. This approach adds significant value because architects will have the liberty to create a properly layered architecture and separate the concerns from day one. As a result, applications with rich functional capabilities, including the quality of services, such as security, reliability, and high-availability will emerge from the innovation platform. Let’s take the same example of providing digital forms instead of the paper-based forms. Enterprises can reuse the existing data models and data repositories while introducing a few new data storages if required. These include building a data-API by exposing the information in the data repositories; tightening data security by adding data entitlement to the data API; avoiding duplication of data by introducing the integration layer; incorporating proper information sharing by connecting to other systems; and implementing the web module by calling web APIs exposed based on the data APIs.
Figure 10: Innovation platform and projects
The other dimension we have to look at is the link between the innovation platform and the projects built on it as well as how both evolve iteratively. Most organizations start with an innovative idea and convert it to a project. The first iteration of the first project will not focus too much about building a platform, but rather deliver it as a successful consumer experience that delivers on-time. Once the first project is in production, the next step would be to identify the reusable capabilities, construct, and release the first version of the platform. Maintaining a version for the innovation platform and projects separately helps to manage and govern the iterative approach and manage dependencies.
Figure 11: Project to platform
Separating the digital-core (i.e. the platform team) and platform project teams help to bring more agility and innovation to the digital enterprise. Even though these two teams are separated there should be some level of synergy between them to achieve success.
Implementation of the above iterative models can be achieved by using agile methodologies like Scrum by creating scrum teams, converting business requirements to user stories and architecture spikes, which implements in less than three month sprints. Scrum has been successfully used by SMEs as well as larger enterprises who have delivered positive results. However, an organization looking at a heavy enterprise architecture practice can explore models like SAFe (Scaled Agile Framework https://www.scaledagileframework.com/) or TOGAF (The Open Group Architecture Forum https://www.opengroup.org/).
Basic principles of modern software development like continuous-integration (CI), test automation and continuous-delivery (CD) are required to be in place for an organization to be successfully iterative. Less manual work on high activities enables the digital workforce to focus on building and delivering innovative ideas. Frequent releases are an important characteristic of the iterative model; automation for development, test and release processes will enhance rapid application development and continuous delivery. Moreover, modern architecture paradigms like MSA help organizations to be podular and plan, build, and run local business functionalities and applications. Edge technologies, such as containers and cloud increase agility, lean nature, and accessibility for pods to develop and deliver innovative ideas. Containers and cloud help to rapidly and dynamically build environments that align with the iterative approach.
WSO2, an open source technology provider for agile digital transformation, offers iterative architecture friendly middleware. Internally, WSO2 uses iterative architecture to build its middleware products, thus making it compatible to create applications and solutions using the same methodology. And when providing consultancy, customers too are encouraged to adopt the same. Open interoperability or ability to seamlessly connect with many systems is the primary advantage that’s offered by WSO2’s middleware products. As a result, an enterprise can become completely agile by utilizing existing IT assets and building an innovation platform via reuse. This way, changes are incremental with minimal impact to the business. WSO2 products are designed to support an API anywhere concept, which allows accessing functionality developed through a business API as well as managing and controlling middleware runtime using a system API. This helps to share business capabilities across pods or business units and integrate through continuous integration, continuous delivery, and automated test frameworks. In addition, the exposed APIs can override and reformat based on API consumer needs.
WSO2 products provide architecture flexibility to implement innovative solutions using traditional architecture patterns such as SOA, event-driven architecture (EDA), Web-oriented architecture as well as modern architecture paradigms like MSA, cloud-native, and container-native architecture. WSO2’s product portfolio covers five technology areas that are primarily raw materials to build a digital enterprise. Integration, API management, identity and access management, and analytics include the complete set of middleware capabilities required to become digital. Each product’s rich functionality provides the flexibility for architects to incorporate middleware capabilities in an incremental way.
All WSO2 products are 100% open source and licensed under the business-friendly Apache 2.0 license. Open source is an ideal model for an iterative architecture. The cost model is opex-based compared to the typical capex model that comes with proprietary software. An opex model helps organizations to take financial risks when implementing digital projects and enables them to be innovative. Open source is about running known middleware because you can access the source code, enabling you to reshape the middleware runtime and fit it into the required shape of iterations of your innovation platform and digital projects.
Figure 12: Iterative process - holistic view
Digital transformation is all about transforming internally to provide a platform to build innovative ideas, and transforming externally to offer better consumer experiences. Better user experiences and a faster go-to market of these digital products are key requirements for an enterprise to maintain its competitive edge. To this end, an iterative architecture approach is the way forward. Once you quickly release your product to end-users, a feedback loop will be created. And when you move to the next iteration of the product, you can incorporate this feedback to deliver a better and improved product. This cycle will continue and your product will evolve rapidly.
For more details about our solutions or to discuss a specific requirement