Why application delivery executives need to think about lifecycle integration, not just lifecycle management
For many years, software delivery has been treated as an ancillary business process; a business process that, though costing the organization a considerable amount of money, does not have the structure, rigor, or focus of other enterprise business processes such as supply chain management, financial management, and even talent management.
But as many organizations look to software as a key business enabler, the practice of creating, manufacturing, and maintaining that software becomes even more important, especially as the very nature of that software becomes more complex with the advent of mobile, cloud and open web technologies. Not only does the software created form a competitive advantage; the ability to deliver it faster and cheaper than competitors has real business value. After all, what often matters is not who has the next idea for a new product or service, but who delivers that product or service to market first.
Software delivery is not only a multi-billion dollar industry, it also is the secret sauce for the majority of business processes. Business innovation is enabled or hindered by a business’s ability to deliver software. Software time-to-market, innovation, and quality are becoming the business variables that define a company’s ability to compete. Compared to other traditional business processes, software delivery is often a poorly assembled collection of immature disciplines.
Portfolio planning, project management, requirements, design, development, test, and deployment individually have well-defined processes and tools, but collectively do not effectively integrate, collaborate, or flow. Work is defined numerous times throughout its life as it moves from the planning stage to definition, development and test, as each group redefines the project artifacts for their own needs and tools. Spreadsheets, email, and wikis are used to glue together processes, but not only are these tools an overhead, often they create another system of record to be kept up to date and integrated.
Process movements such as Lean Startup, DevOps, and Agile have driven organizations to re-evaluate their development practices with the desire to increase its cadence and feedback. Though many organizations have adopted Agile and are trying DevOps, both these initiatives focus on engineering teams and their practices.
Business Agility requires work to flow through engineering, management and customer processes in a seamless and integrated way. But this is far from the situation we have today.
The following two sections outline the realities of what we have today:
Henry Ford and the industrial revolution taught us specialization of labor and departmental hierarchies are the best way of increasing efficiency and focus. The more complex a problem becomes, the more complex the organization becomes to solve it. As IT groups grew in size, so did their processes, management structures, and hierarchies. Each discipline over time was taken out of the developer’s purview, creating separate groups such as requirements, design, and testing. Agile methods cited this separation as a key reason why development projects failed and required cross-functional teams to be created. Even with everyone on the same team, the reality is different roles approach a problem in different ways, applying different practices and tools.
Even within an Agile team, developers and business analysts will use disparate tools, each encouraging a different way of working. Traditional test tools will describe problems from a test perspective, while development tools will view them from a developer’s point of view. These tools will fragment the process of software delivery by introducing different vocabulary, artifacts, and process steps. Process improvement is generally at these functional unit levels, rather than across the entire software development and delivery process.
Imagine a factory were each step in the production process is optimized, but the product is still low in quality and far too expensive. That was the experience of automobile manufacturers prior to the Lean manufacturing revolution. The traditional manufacturing models practiced by Henry Ford were ill-equipped to manage the process variance, product complexity, and product flexibility required for modern automobile manufacturing. The adoption of Lean methods meant a holistic view of the end-to-end process, allowing an organization to reduce waste and increase value at the enterprise level rather than department or job level.
It also led to the creation of clear process ownership, architecture, automation, and measurement - concepts still eluding the software development industry.
Within software development, we still have:
Modern business is all about data based decision making. From day-to-day operations to corporate and regulatory compliance, companies rely on both complex instrumentation and reporting capabilities. The process of software delivery, like other business processes, requires traceability and reporting, but unlike other processes does not have a consistent, agreed-upon process or data model. Unless the organization is building a safety critical system, there is typically no agreement on the reports necessary to describe the flow, impact, productivity, quality, or value being generated in the software delivery process.
This is an interesting conundrum. Are there no agreed-upon reports because we don’t know what to measure? Or can we not report upon what we’d like to measure?
As we’ve previously noted, the tools used within the software delivery lifecycle are isolated from each other and there are an abundance of manual processes where automation should prevail. The resulting manual process for creating status and traceability reports takes time and requires the involvement of people. On large projects or projects that are spread out geographically, the amount of time required ultimately reduces the value of the information. For example, a common situation is the use of a defect spreadsheet when working with an outsourced testing partner. The defect spreadsheet is sent at the end of each working day, but often versions are updated by different teams at different times, resulting in many meetings. These meetings often start with the question ‘which version of the spreadsheet are we working from?’ and lead to discussions about differences in each version.
Now, let’s look at connecting the software delivery lifecycle.
As software increases in value and importance, the obvious next step is to treat it as a key business process. By approaching software delivery in the same way as other processes, such as sales, procurement, and distribution; you approach software delivery in a different way and concentrate on not only the processes that it comprises, but also its value, reporting, and analytics. This also results in an end-to-end or holistic view rather than concentrating on each discipline in isolation. Ultimately, the value of the process is not in each discipline, but instead the result: software that is used by customers or the business.
Software delivery comprises many processes including the SDLC, project management, demand management, quality management, operations, and service management. For many organizations, these processes have guidelines, templates, tools, and artifacts. In fact, many processes include many of the same artifacts, such as defects, requirements and tasks, and for Agile stories and epics. Unfortunately for many organizations, there is no overall process model or formal description of how these processes interact.
In fact, even with organizations striving for Continuous Delivery, the one “continuous” thing is the delivery of the final work product: the code and application. There is still a lot of discontinuity in the flow of project artifacts from one functional discipline to the next.
This situation is made worse by the fact that software delivery is increasingly a collection of suppliers providing code, services, and APIs for inclusion in the product. With the advent of web services and API driven development, often supply chains are loosely coupled with limited control and no consistent process or ALM tool set. Another example is the trend to outsourced testing. Often these testing groups are not a part of the regular stand-ups because of organizational boundaries and the reality of time zones, but their information must be included in the stand-up to determine the real status of the project.
If you ask a group of IT project managers what the key metrics are for any software organization you will be greeted with an array of very different measures ranging from the tactical, such as number of defects to the strategic, such as change state over time. Not only do application development professionals need to define the right measures, they also need to see how the data changes over time. For many tools, temporal- or time-based data is difficult to find, with those tools focusing on the immediate flow of work.
The element lacking here is not necessarily the desire to manage the application development and delivery process, but the ability to do so. Organizations are prevented from having a holistic lifecycle view because their tool infrastructure is comprised of disparate products intended to maximize the efficiency of one (or two) of the functional teams within the broader organization.
For many organizations, the promise of ALM implies the adoption of one tool, or one tool suite. This allows normalization of information across different disciplines and supports common reporting and analytics. Vendors add to this idea with clear marketing and sales collateral describing how ‘all your development problems will be solved when you move to Tool X’. However for many organizations, the reality is much more complex than any one tool can solve. Add to that emerging platforms such as mobile, cloud, and open source and your tool landscape will always be complex.
Heterogeneous tools stacks are the reality. Integration among them is required.
Still, “integration” isn’t simply a point-to-point connection between two systems. The kind of integration providing the pan-organizational visibility that underlies true process improvement requires:
Enabling this Software Lifecycle Integration across disciplines and tools will provide the kind of infrastructure necessary for organizations to automate and report on their key business process of software development and deployment.
For many organizations, the lack of an integrated software delivery practice means the difference between project success and failure. It is time to apply Lean thinking to the practice of software delivery and make the creation and maintenance of software flow from idea to implementation, removing disconnects and enabling real-time collaboration. It is time to create a discipline focused on connecting the end-to-end practice of software delivery. That discipline must provide the materials necessary to connect the practice of software delivery; creating integration architecture, process, and measurement approaches for software delivery professionals to deliver software faster and remove the waste plaguing the practice.