Telecommunications Company Reduced Software Bug Fix Lead Time By 40% With GLSS
GLSS Green Belt Eduardo Torres did a great job of cutting waste out of the process of fixing software bugs. The use of software is growing fast, and with no known way to guarantee new software is error-free, rapidly fixing bugs found is critical. Eduardo not only cut nearly 40% of the process time, but also cut the variability in half, greatly improving reliability!
Eduardo Torres is a Senior Project Manager and Lean Six Sigma Green Belt with expertise in the Telecommunications Field. For his Green Belt Project, he decided to tackle the long lead time for software bug fixes â reducing the total lead time from 25 to 15 days!
THE CHALLENGE
Eduardo knew that software bug fixes were taking far too long to complete and that he needed to reduce software bug fix lead time.
His goal?
Reduce software bug fix lead time from 25 to 15 days. This would save his team money and time that could be spent on developing new features, ultimately increasing customer revenue and boosting his teamâs morale!
To achieve this, he needed to first measure his process to find out where there were bottlenecks and delays. Time for DMAIC!
THE DISCOVERY
Eduardo developed a Data Collection Plan and set out to measure every step in his process. Luckily, his ticket request system made this easier.
After a month of data collection, he discovered that the average request ticket lead time was 25 days â not good! Requests could take anywhere from 3 to 55 days to complete, so he knew that his process had too much variation and could be improved.
Eduardo first turned to a Fishbone Diagram to brainstorm possible root causes, bottlenecks and issues within his process.
He then used a Swimlane Map to outline his entire process and a Value Flow Analysis to determine the lead time for individual steps. He found some parts of the process were taking an alarming amount of time.
Assigning a request ticket to a software developer took 2 days. Fixing the software bug took 15 days. Delivering the fix to his parent branch took 3 days. And to have the fix tested and delivered to their official branch took 5 days.
Needless to say, if he wanted to reduce the entire process to just 15 days, these steps needed some trimming.
THE IMPROVEMENT
Eduardo decided to weigh his possible solutions by using a Solution Selection Matrix.
He was able to determine which solutions would have the greatest impact and require the least amount of time and effort to implement.
What did he find?
By removing unnecessary code reviews, cross-training developers, and using Kanbans for ticket assignments, he would be able to remove a non-value adding steps, reduce the time it took to fix software bugs and speed up the ticket assignment process.
He removed code inspections for all software fixes that were less than 250 lines of code. His research showed that for 91% of these fixes, no further correction was required. This removed an unnecessary review that saved the team valuable time and allowed developers to process requests faster.
He cross-trained developers so that his team was more capable of handling different kinds of requests, increasing the amount of tickets that could be processed.
Lastly, he implemented a Kanban method for ticket assignment. This allowed tickets to be prioritized and assigned to developers faster, reducing the time tickets spent waiting to be addressed.
In the end, software bug fix lead time reduced by 39%, from 25 to 15 days, and variation within the process decreased by almost half!
Eduardo learned some valuable lessons from his project:
- Extra reviews provide a false sense of security and can hurt a process
- Continuous communication is key to avoid surprises and variation
- Giving credit to his team members for good ideas is easy, free and boosts morale
Weâre excited to see Eduardoâs project create real value for his team and customers. We canât wait to see what process he takes on next!
EXECUTIVE SUMMARY - BUSINESS CASE
Lead time improvement for software bug fixes will result in cost savings improved use of resources. Lead time improvement could translate into monetary benefits because we will be able to utilize development and testing resources to work on more features, which could increase customer revenue. This aligns with our “Best In Class’ mission.
EXECUTIVE SUMMARY - ROOT CAUSE ANALYSIS
Data showed bug fix ticket assignments and fix deliveries to the official branch took too long. Too much time was spent in software building in multiple process steps and unnecessary code inspections.
EXECUTIVE SUMMARY - SOLUTIONS IMPLEMENTED
- Use Kanbans for bug fix ticket assignments
- Remove code inspections for LOC < 250
- Software deliveries directly to official branch
- Conduct cross-training for the build and delivery to official branch
PROJECT RESULTS - GRAPHICAL DIS PLAY OF IMPROVEMENT
Software bug fix lead time has reduced from 25 days to 15 days
Key Takeaway: The project was a great success! Customer perception changed in a good way!
DEFINE PHASE - PROJECT CHARTER - PROBLEM STATEMENT
The current total lead time for software bug fixes is an average of 25 days. This results in missing milestones by test teams and delivery delays to customers.
DEFINE PHASE - PROJECT CHARTER - GOAL STATEMENT
Decrease the software bug fixes lead time from 25 days to 15 days by 4/1/2019.
DEFINE PHASE - PROJECT CHARTER - SCOPE
- First Process Step: Fix request ticket created by customer
- Last Process Step: Fix request ticket built in the official branch
- In Scope: Ticket assignment process, code development process
- Out Scope: Hardware additions, official build load delivery and test processes done by external teams
DEFINE PHASE - PROJECT CHARTER - BUSINESS CASE & BENEFITS
Lead time improvement for software bug fixes will result in an cost savings and improved used of resources. Lead time improvement could translate into monetary benefits because we will be able to utilize development and testing resources to work on more features, which could increase customer revenue. This aligns with our ‘Best In Class’ mission.
DEFINE PHASE - PROJECT CHARTER - TIMELINE
| Phase | Planned | Actual |
| Define | September 1 | September 3 |
| Measure | September 17 | September 28 |
| Analyze | October 21 | November 1 |
| Improve | November 21 | December 1 |
| Control | December 14 | December 13 |
DEFINE PHASE - PROJECT CHARTER - TEAM MEMBERS
| Position | Person | Title | Time Commitment |
| Team Lead | Eduardo Torres | Lean Six Sigma Green Belt Candidate | 40% |
| Sponsor | Phil D. | Director | 15% |
| Team Member | Bharati R. | Release Lead | 20% |
| Team Member | Rashmi M. | Software Builder | 20% |
| Team Member | Sundar E. | Software Developer | 20% |
| Team Member | Errol D. | Software Tester | 20% |
Key Takeaway: Excellent buy-in â project is worth doing.
Â
DEFINE PHASE - SIPOC
Key Takeaway: The scope of the project is from the moment a bug fix ticket is created until the bug fix is built in the official branch.
DEFINE PHASE - AS-IN DETAILED MAP SEGMENT
Key Takeaway: The team learned how long it takes to provide a fix to the customer! This provides a bigger picture of the impact of the work they do.
MEASURE PHASE - DATA COLLECTION PLAN
Key Takeaway: Since this is a lead time project, most of the measures turned out to be continuous measures of different segments of time. The discrete measure is to determine if a process step is really needed.
MEASURE PHASE - BASELINE DATA - PROJECT Y - BASELINE LEAD TIME OF SOFTWARE BUG FIXES (AUGUST 2018-SEPTEMBER 2018)
Key Takeaway: Lots of variation in the process. The average time to deliver bug fixes is 25 days which is very high!
MEASURE PHASE - BASELINE DATA - PROJECT Y - SOFTWARE BUG FIXES LEAD TIME BASELINE LEAD TIME OF SOFTWARE BUG FIXES (AUGUST 2018 - SEPTEMBER 2018)
Key Takeaway: Target delivery is 15 days or less, so the bug fix delivery process is clearly not capable. 75th Percentile is 33 days!
ANALYZE PHASE - FISHBONE DIAGRAM
Key Takeaway: The biggest areas to analyze further were code inspections, delivery to parent branches, and assignment of bug fix tickets.
ANALYZE PHASE - MAP SEGMENT SHOWING ANALYSIS
Key Takeaway: Process analysis showed a potential for removing unnecessary steps that are not adding value or causing bottlenecks.
ANALYZE PHASE - VALUE-ADDED FLOW ANALYSIS
Key Takeaway: Biggest opportunities to remove steps are code inspections and delivery to parent branch. Opportunities for time reduction is the assign tickets to software developers step.
ANALYZE PHASE - VALUE-ADDED FLOW DATA
Key Takeaway: Split of the baseline data based on the ticket state. Deliver to parent branch time is all wasted time. Removing code inspection saves average 3 days in the lead time.
ANALYZE PHASE - ROOT CAUSE ANALYSIS
Key Takeaway: Data confirmed all the hypotheses except the code inspection not needed for LOC > 250.
ANALYZE PHASE - ROOT CAUSE HYPOTHESIS RESULTS
Key Takeaway: Data confirmed all the hypotheses except the code inspection not needed for LOC > 250.
IMPROVE PHASE - SELECTED SOLUTIONS
Key Takeaway: The effort required to cross-train software builders (development tasks) will not allow to meet the schedule. Code removal buy-in showed that sometimes some processes are a false sense of security.
IMPROVE PHASE - "TO BE" MAP SEGMENT
Key Takeaway: Process changes including remove unnecessary inspections (for fixes of 250 lines of code or less), removing bottlenecks, Cross-training to reduce hand-offs and use of Kanban methodologies.
IMPROVE PHASE - RUN CHART SHOWING IMPROVEMENT
Key Takeaway: Improvements had an impact of 39% reduction on software bug fixes lead time while cutting variability in half! This gave much faster response time with greatly improved reliability!
IMPROVE PHASE - IMPROVED - SOFTWARE BUG FIXES LEAD TIME
Key Takeaway: Data with improvement solutions shows an average lead time decrease from 25.8 days to 15.6 days and 75th percentile is 19 days from 33 days.
IMPROVE PHASE - IMPROVED - VALUE ADDED FLOW DATA
Key Takeaway: Improvements showed on step #1 (from 2 days to <1 day) by using Kanban and on step #7 (from 3 days to <1 day) by software cross-trained developers to deliver directly to the official branch.
CONTROL PHASE - PROJECT CLOSURE - LEASSONS LEARNED
- Continuous communication to avoid surprises
- Open-minded attitude due to we will challenge status quo
- In some cases some processes give a false sense of security like code inspection for bug fixes
- Be sure to give credit to everyone who came up with good ideas – credit is free!
CONTROL PHASE - PROJECT CLOSURE - CUSTOMER IMPACT
- Customer perception changed in a good way.
CONTROL PHASE - PROJECT CLOSURE - FINAL CALCULATIONS
- Effort savings of 46 People Month that is translated in $384K in savings
- Team member morale increased, more time to work on features
CONTROL PHASE - SIGN-OFF FROM PROJECT SPONSOR
The Platform Software Team did a great job! We are moving on the right direction to be "Best In Class" - I'm offically giving my stamp of approval for documented improvements. - Phil Diguglielmo, Director
Key Takeaway: Ended up with resource effort savings and the VP is thrilled!
CONTROL PHASE - PROCESS CONTROL PLAN
Key Takeaway: The bug tracking tool has the data needed to collects this information. Weâll only need the release lead to gather the data.
CONTROL PHASE - MONITORING & RESPONSE PLAN
Key Takeaway: Responding to the last 4 items will help ensure that the first item (our primary goal) remains low.
KEY WORDS
- Code Inspection: The most formal type of review, which is a kind of static testing to avoid the defect multiplication at a later stage
- Development Branch: Software repository created under a Parent Branch and used by software developers to make code changes
- Development Build: Build one or more software components in the Development Branch
- Lines of Code (LOC): Software metric used to measure the size of a computer program by counting the number of lines in the text of the program’s source code
- Local Branch: Software repository created under the Official Software Repository
- Local Branch Build: Build all software components in the Local Branch
- Official Branch: Official software release repository
- Official Branch Build: Build all software components in the Official Branch
- Parent Branch: Higher level repository
- People Month: Number of resources / month
- Private Patch Testing: Testing of software code changes done to fix specific bug or issue
- Sanity Testing: Testing so no further issues are introduced due to these changes
- SW Bug Fix Lead Time: The amount of time (in Days) it takes from the moment the customer creates a bug fix ticket (ticket state Created) to the moment the bug fix is built into the official SW branch (ticket state Ready For Testing (RFT))
- Ticket Assigned Lead Time: The amount of time (in Days) it takes from the moment the customer creates a bug fix ticket (ticket state Created) to the moment the bug fix ticket is assigned to a SW developer (ticket state Assigned)
- Ticket Code Committed Lead Time: The amount of time (in Days) it takes from the moment the SW developer is ready to deliver the code (ticket state Ready to Submit) to the moment the SW developer delivered the code to parent branch (ticket state Code Committed). Development testing is done and Code Inspection is approved in the Ready to Submit state.
- Ticket Ready for Testing Lead Time: The amount of time (in Days) it takes from the moment the SW developer delivered the code to parent branch (ticket state Code Committed) to the moment bug fix is built into the official SW branch (ticket state Ready For Testing (RFT))
- Ticket Ready to Submit Lead Time: The amount of time (in Days) it takes from the moment the bug fix ticket is assigned to a SW developer (ticket state Assigned) to the moment the SW developer is ready to deliver the code (ticket state Ready to Submit). Development testing is done and Code Inspection is approved in the Ready to Submit state.
- Ticket State
- Created: Ticket created by a customer
- Assigned: Ticket assigned to a developer
- Ready to Submit: Development testing is done and Code Inspection i approved
- Code Committed: Code delivered to Parent Branch
- Ready for Testing (RFT): Code built in Official Branch and ready for verification
PROJECT STORYBOARD
