Stream Step

If the CEO or GM of a business unit producing software using agile in a modern development, operations and QA structure, were to sit in on a release coordination meeting, they'd wonder what decade they'd been transported back to. They'd sit in a room full of folks -- expensive staff, folks like developers, MBA project and product management, QA and IT support folks, a marketing person or two and some extras -- and they'd watch them all gather over a set of spreadsheets and post-it notes and . . . basically decide the future of the company by planning the release steps of its products. They'd see these meetings happen weekly or more as product development and release details are communicated and coordinated using very old -- and poorly suited -- techniques and technologies indeed.

So, yes, they'd think they'd fallen backwards in time, as they wouldn't recognize the painful, time-consuming, staffing hours consuming sink-hole of a practice that defines most release coordination today. They'd see their multi-million and billion dollar business managed at the grass-root level by developers, MBAs and DBAs working in real-time on multi-layer spreadsheets. Using two-hundred dollar tools to attempt to manage billion dollar processes.

They'd see the DevOps gap in terms of salaries paid to expensive talent to sit in a room while a RC or a project manager or equivalent goes through the meeting by making notations in spreadsheets and post-it notes and a notepad for follow-up to get answers that weren't available during the meeting. He'd see IT explain why they couldn't provision everything development wanted and development explain why the release delayed because QA and IT had coordination problems. He'd take the 10 to 20 staff headcount and the process and see that the core of his business, producing product to be sold and installed and counted as revenue, was really more a sort of wonky professional coincidence of deliverables than scientific process.

The CEO would leave the meeting in a state of shock, probably even wondering if he needed to report the state of "DevOps" in his organization to the board, because frankly the DevOps gap is material information on the state of the business. That is, if the CEO ever found out how release management works in an organization that isn't DevOps savvy.

Most enterprise development teams have challenges deploying their n-tier web applications into test and production environments.

Let's look at some of these challenges:

1. Shorter release cycles - the time available between deployments is reduced and deployment problems can create significant project delays. Also, when problems do occur it usually takes some of the best developers to resolve them, further reducing development team effectiveness.

2. Multiple development teams - bringing development teams and their application components together in one environment for system integration tests can take hours and even days of deployment work. What's more the tooling that development teams use to deploy to pre-production environments isn't available for production environments due to compliance and change management rules.

3. Increased infrastructure complexity - many pre-production environments are relatively simple, but as the application components come together into a system-wide environment the complexity grows. In these environments configuration issues which were not evident in local environments can emerge as significant performance or functional problems.

Solving these challenges requires a shift in thinking about the relationship between development and operations teams and the use of DevOps automation. I've done some brief, educational videos on these topics here.

Post it Notes = Agile

Many of us with experience on agile development teams have used "post-it notes" and "card walls" to manage requirements and development work. These tools work well for teams that are collocated and relatively small. But as agile methodologies have matured, teams have gotten larger and more distributed. The market has responded with great tools like Rally (http://www.rallydev.com). Now we can collaborate on agile development processes without needing to be in the same physical room - reviewing the same physical cards on the wall. And of course, now that these agile processes are embodied in software we are seeing value beyond managing the cards themselves.

Spreadsheets = Deployment Coordination

In a similar way - today's deployment processes are primarily coordinated with spreadsheets and conference calls. These tools have worked well - but they are starting to show signs of strain as the rate of deployments increases. (This is especially true in organizations that are adopting Agile development methodologies.) There are 2 key issues with spreadsheets for deployment coordination:

*1. Manual automation - it may seem like a contradiction in terms but most deployment automation scripts are manually launched at the command line by a system admin or deployment engineer. During this fundamentally manual process, a momentary typo in a script parameter can create hours of downstream trouble-shooting and repair work. If you have had any exposure to n-tier application deployment you know that there has to be a better way to do this. One customer we worked with had 2 system administrators assigned to each production deployment - one to type in the commands and another to review and verify each command (letter by letter) before it was executed. Now that's expensive.

*2. Which spreadsheet? - when the list of activities to be coordinated grows beyond a handful teams frequently need to change and update the spreadsheet. Managing the versions of the spreadsheet and tweaking it to accommodate unique steps for this release becomes a full time activity. Checking it into version control is one strategy - but is the one in version control the latest copy? Does the operations team have access to version control? Probably not.

Both of these challenges call for a solution that is lightweight and easy to use - but gives structure to the process and allows you to execute your existing automation.

Enterprise application development teams are wrestling with more of everything – more systems, more services, more projects and more environments.  And less time to do it in - as cycle times for development are accelerating with increased testing automation and virtualized lab environments. All of this is making the release process more complex and error prone than ever before.  Today, most enterprises use spreadsheets, checklists, bridge lines and emails to coordinate complex release processes.  Release automation is typically low level and command line driven - in the form of scripts which are manually run or scheduled by release engineers or developer.  This patchwork of tools is failing to deliver as the number and frequency of deployments increase.

Release Management Challenges
• Process errors due to lack of end-to-end process coordination
• Delays due to cross-organizational hand-offs and lack of communication
• Expertise linked to a few key individuals
• Lack of metrics to drive improvement objectives

Impacts
• Increased cost – Inefficient processes result in reduced development throughput for business objectives
• Increased risk - Reduced development and testing time negatively affects quality driving production outages and slowdowns

The goal of this Blog is to discuss the challenges and solutions - we have seen in the sphere of Application Release Management - and hopefully deliver some value to the larger development community.