Stream Step

If the CEO or GM of a business unit producing software using agile in a modern development, operations and QA structure, were to sit in on a release coordination meeting, they'd wonder what decade they'd been transported back to. They'd sit in a room full of folks -- expensive staff, folks like developers, MBA project and product management, QA and IT support folks, a marketing person or two and some extras -- and they'd watch them all gather over a set of spreadsheets and post-it notes and . . . basically decide the future of the company by planning the release steps of its products. They'd see these meetings happen weekly or more as product development and release details are communicated and coordinated using very old -- and poorly suited -- techniques and technologies indeed.

So, yes, they'd think they'd fallen backwards in time, as they wouldn't recognize the painful, time-consuming, staffing hours consuming sink-hole of a practice that defines most release coordination today. They'd see their multi-million and billion dollar business managed at the grass-root level by developers, MBAs and DBAs working in real-time on multi-layer spreadsheets. Using two-hundred dollar tools to attempt to manage billion dollar processes.

They'd see the DevOps gap in terms of salaries paid to expensive talent to sit in a room while a RC or a project manager or equivalent goes through the meeting by making notations in spreadsheets and post-it notes and a notepad for follow-up to get answers that weren't available during the meeting. He'd see IT explain why they couldn't provision everything development wanted and development explain why the release delayed because QA and IT had coordination problems. He'd take the 10 to 20 staff headcount and the process and see that the core of his business, producing product to be sold and installed and counted as revenue, was really more a sort of wonky professional coincidence of deliverables than scientific process.

The CEO would leave the meeting in a state of shock, probably even wondering if he needed to report the state of "DevOps" in his organization to the board, because frankly the DevOps gap is material information on the state of the business. That is, if the CEO ever found out how release management works in an organization that isn't DevOps savvy.

Most enterprise development teams have challenges deploying their n-tier web applications into test and production environments.

Let's look at some of these challenges:

1. Shorter release cycles - the time available between deployments is reduced and deployment problems can create significant project delays. Also, when problems do occur it usually takes some of the best developers to resolve them, further reducing development team effectiveness.

2. Multiple development teams - bringing development teams and their application components together in one environment for system integration tests can take hours and even days of deployment work. What's more the tooling that development teams use to deploy to pre-production environments isn't available for production environments due to compliance and change management rules.

3. Increased infrastructure complexity - many pre-production environments are relatively simple, but as the application components come together into a system-wide environment the complexity grows. In these environments configuration issues which were not evident in local environments can emerge as significant performance or functional problems.

Solving these challenges requires a shift in thinking about the relationship between development and operations teams and the use of DevOps automation. I've done some brief, educational videos on these topics here.

Post it Notes = Agile

Many of us with experience on agile development teams have used "post-it notes" and "card walls" to manage requirements and development work. These tools work well for teams that are collocated and relatively small. But as agile methodologies have matured, teams have gotten larger and more distributed. The market has responded with great tools like Rally (http://www.rallydev.com). Now we can collaborate on agile development processes without needing to be in the same physical room - reviewing the same physical cards on the wall. And of course, now that these agile processes are embodied in software we are seeing value beyond managing the cards themselves.

Spreadsheets = Deployment Coordination

In a similar way - today's deployment processes are primarily coordinated with spreadsheets and conference calls. These tools have worked well - but they are starting to show signs of strain as the rate of deployments increases. (This is especially true in organizations that are adopting Agile development methodologies.) There are 2 key issues with spreadsheets for deployment coordination:

*1. Manual automation - it may seem like a contradiction in terms but most deployment automation scripts are manually launched at the command line by a system admin or deployment engineer. During this fundamentally manual process, a momentary typo in a script parameter can create hours of downstream trouble-shooting and repair work. If you have had any exposure to n-tier application deployment you know that there has to be a better way to do this. One customer we worked with had 2 system administrators assigned to each production deployment - one to type in the commands and another to review and verify each command (letter by letter) before it was executed. Now that's expensive.

*2. Which spreadsheet? - when the list of activities to be coordinated grows beyond a handful teams frequently need to change and update the spreadsheet. Managing the versions of the spreadsheet and tweaking it to accommodate unique steps for this release becomes a full time activity. Checking it into version control is one strategy - but is the one in version control the latest copy? Does the operations team have access to version control? Probably not.

Both of these challenges call for a solution that is lightweight and easy to use - but gives structure to the process and allows you to execute your existing automation.

One of most challenging things about improving a complex release management process is getting started. Here is a great article on improving release processes from http://www.cio.com - http://www.cio.com/article/print/440101 The authors point out in section 3:

"3. Get lightweight processes in place. Test them early and review them regularly."

"If there is one single guiding principle in engineering (or reengineering) a process, it is to do a little bit, review your results and then do some more. Repeat this cyclic approach until you get the results you want.

Lightweight processes are those that do not require lengthy bureaucratic approvals or endless meetings to get agreement. They usually require only the minimum acceptable level of inputs and outputs. What they lack in bulk and bureaucracy, they make up for in response to change and popular adoption!

Underpinning this approach is the thorny issue of documentation. You need to record what you did and how you did it. Otherwise, what do you review and how do you improve?"

As part of this effort the authors put in place a commercially available portal product - based on a wiki-portal approach.

But as we have found with many of our customers - Portals are generic tools for documentation and sharing. Release management has a specific set of challenges which are onlypartially covered by the portal approach.

1. Documentation - a documentation portal is only as good as the updates to it. If people forget to update it then it quickly loses currency and usefulness. In the day-to-day release process - portals can drift out-of-sight and out-of-mind - resulting in the checklist entry "Make sure you update the portal with the DB dumpfile location."

2. Process control - portals are primarily stateless repositories of lists and documentation. They don't enforce or assist with process control. And portal workflow modules, where they exist, suffer from the core problems of workflow - brittleness and maintenance complexity. In fact, "coded workflows" are antithetical to the notion of a "lightweight process".

3. Metrics - related to the lack of process control, portals don't produce metrics that assist in targeting subsequent improvement work. At best managers can use forms to measure inputs to the process in the form of requests for release activities - but the underlying process detail is lost from view. Frequently this detail - what happened and when - is where the real process improvement opportunities reside.

4. Automation - finally the goal of release management is to deliver flawless releases as efficiently as possible. Automation is a key part of achieving this goal. Portals just don't engage release automation - it's not what they were designed to do.

Conclusion

Putting in place a portal for your release team is a great first step to improving your release processes. If you don't have a way for your team to share release documentation then a portal can make a huge difference. But if like many companies today, you already have a release team portal and yet you still see room for improvement in your release processes, you should consider a lightweight release management solution like StreamStep - SmartRelease. A solution purpose built for today's release management challenges - Release Management... Simplified.