- The Challenge: Development stopped for 7 days during every Content Management System (CMS) release cycle.
- The Discovery: The CMS architecture did not allow for code development while new code was being tested and released.
- The Improvements: They cleaned up existing Quality Assurance (QA) environments, set up dedicated Development Environments and created a new Code Branching Strategy.
- The Results: They reduced the “code freeze” from 7 days down to 2-3 hours—a 95% improvement.
- What’s Next: Completely eliminate the code freeze and conduct a 5S of the development service ticket system.
Every 45 days the Web Development Team of The Nature Conservancy in Arlington, VA conducts a new release of their Content Management System (CMS) but that entailed 7 days of lost productivity. The new releases allow the organization to make quick and easy updates to their public-facing website. Unfortunately, each release resulted in a week-long “code freeze.”
While the team tested the new code and moved it into the live production environment, they had to stop all code development. The reverse was true as well—if they found bugs during the testing, the testing had to be stopped.
Tadeo Reyes, Senior IT Business Analyst, and his manager decided to tackle the problem. When Tadeo started the project, he didn’t know what improvements he could expect but he knew there had to be a better way.
Tadeo connected with the Web Development Team and found that there were a lot of merge conflicts and errors in addition to the long code freeze. Tadeo conducted Process (Gemba) Walks with the web developers and the CMS vendor. He used Fishbone Diagrams, 5 Whys and Process Maps to get to the root cause of the long code freeze.
What he found surprised him:
- The development and testing environments were set up such that some environments were not being used at all and others were being used in unintended ways
- The web developers were not able to test and ensure Quality Assurance of their own work
- The QA environment hadn’t been used since the previous vendor left the project
- The code branches were overpacked and cluttered and the current branching did not allow for quick changes or fixes
The web developers were initially reluctant to change the structure. They had ideas on how to best set up the environments and change the code branching structure, but they were already bogged down trying to fix errors caused by existing merge conflicts. They didn’t know when they would find the time to work on changes—or if this was a priority for the business.
At the same time, Tadeo was feeling pressure from the business side to not make things slower than they already were, but he found a way to bridge the gap. He was able to communicate the business needs to the development team and vice versa.
Tadeo worked closely with the Head Web Developer and together they came up with a list of improvements:
- Clean up the Quality Assurance (QA) Environment so it can be used for QA
- Reassign the Development Environment to be used strictly by developers
- Update the branching strategy to break out development code branches by feature for easier parsing of approved code
- Implement a better code-branch naming convention
- Implement a dedicated Release Branch for continuous development integration
With these improvements in place the developers are now able to test their work in their own QA Environment without halting development. The new code-branching structure allows them to quickly create a new Release Branch. This means development and testing don’t have to stop for long in their own environments or branches. With these changes the developers are now able to make fast fixes directly without affecting the development and testing in progress.
Tadeo and the team were able to reduce the code freeze time from an average of 7 days down to 2 – 3 hours—a 95% improvement. Before they made the improvements, about 16% of the release cycle time resulted in lost development time. Since implementing their fixes, they cut the lost time to less than 1%.
The improvements not only give the developers more time to work on new features but also allow them to focus on QA and the work quality for every release.
Tadeo is determined to eliminate the code freeze time completely so the developers don’t have to stop development at all. And he is already in the middle of his next endeavor: A 5S project on the service ticket system to streamline and reduce the development backlog. Along with a constant dialogue with leadership, the team collaboration was one of the greatest discoveries and he is looking forward to more of what they can accomplish together.
Tadeo Reyes is a Senior IT Business Analyst at the Nature Conservancy in Arlington, VA. Tadeo is working with Web Development and Marketing and leads efforts that are focused on the public-facing website of The Nature Conservancy. He is a Certified Lean Six Sigma Green Belt.