Question: How big is the problem?
Answer: We don’t know.
This is how my initial conversation with one nonprofit organization began recently. They use a database to track all consumer encounters. Similar to an electronic medical record, the database keeps track of consumer interactions and treatment plans. It also connects to payroll and tracks time spent on individual projects and the corresponding grant. Data is sent to state-wide funding agencies to ensure that time spent relates to how funding is allocated.
So, what’s the problem?
A gut-level quality check told leaders that the data was off—by a lot. End-of-month reports generated from the database showed consumer activity to be less than what leadership knew it must be. Given the staff on hand and the number of appointments per day, there is no way the reports were right.
How wrong was it? No idea.
What’s the impact of wrong data? If the organization can’t show they are serving the population they say they are, there is a risk of losing funding or being accused of fraud.
Enter Toyota Kata. Improvement Kata is a four-step routine that builds on discovery as teams strive to reach their goals. Kata is Japanese for “model” and the four steps of the model are:
- Understand Your Direction and Set a Challenge
- Grasp the Current Condition
- Set the Next Target Condition
- Conduct the Experiments to Get There
Improvement Kata is designed to become a teachable, daily practice that emphasizes scientific thinking and problem solving. This practice forms the foundation of a Continuous Improvement or “Kaizen” Culture.
Step 1 of Improvement Kata: Understand Your Direction and Set a Challenge
The leadership of the agency wasted no time defining exactly the direction the organization needed to go: 100% complete and accurate data collection that exactly matched the services staff are providing to consumers each day.
Step 1 was complete!
Step 2 of Improvement Kata: Grasp the Current Condition
The hard part was answering the second question, where are we now? What does the process for data collection look like? What is the current rate of errors compared with services provided?
I worked with leadership and we observed and interviewed all 19 staff and asked them the following questions:
- How did staff enter data after seeing a consumer?
- What fields did they use?
- Which grants were selected?
- Was data entered after each encounter or in a batch at the end of the day/week?
To the extent possible, we observed data entry activity in real time after a consumer encounter.
We created a process map of how data was entered and noted the variation by staff member.
The process owner also started to do some random views of consumer charts to understand the type and extent of errors. He categorized his record reviews into a chart similar to the one shown below.
Now that the current condition was understood, it was time to tackle Step 3 of the Improvement Kata.
Step 3 of Improvement Kata: Setting the Next Target Condition
The intent of the target condition is to define a future state in terms of both what the process will look like (if you’re looking down from the ceiling tiles three months from today and can’t see any numbers) and how it will be measured. Where do we want to be next?
As we discussed the target condition with the process owner, the leader expressed the importance of his colleagues in leadership taking on more responsibility for measuring and keeping track of accuracy. He also wanted staff to understand what accurate data collection looked like and to fix errors before they submitted their time sheets.
We documented both the current and target condition for each of the types of errors:
Steps 1-3 of the Improvement Kata (Setting a Challenge, Understanding the Current Condition and Setting Your Next Target Condition) were complete. These steps are also known as the planning stages in that they are a precursor to experimentation.
Step 4 of Improvement Kata: Conduct the Experiments to Get There
The next step was to understand the obstacles between the target and actual conditions and finally begin to experiment. As a coach, I could now start using the Coaching Kata questions for our coaching cycles.
As the process owner began to populate his Obstacle Parking Lot, he was struggling to come up with the facts and data that were preventing him from operating according to the target. He had lots of assumptions (our brains don’t wait for actual data; they just fill in the blanks and jump to conclusions). His assumptions about why staff weren’t accurately reporting consumer interactions included: they don’t care, they don’t have time, they don’t know how to do it.
But he didn’t know any of this for sure. Time to go-see and find out more. Here is a list of obstacles similar to what he came up with:
- Staff hours don’t add up to scheduled time.
- Staff report hours in the “none” category which doesn’t connect to a specific grant.
- Staff are not comfortable asking consumers certain questions and so charts are incomplete.
- Staff selecting wrong grant/service—not sure why.
- Staff are running weekly performance reports but not fixing errors.
- Supervisors are making mistakes too.
- Time sheets are late—indicating that data entry is batched.
- Same staff are making repeated errors even after instruction.
- Staff don’t know they are making mistakes.
Not every obstacle had facts and data and there are some assumptions noted in there, but we decided to get started on some experiments.
What one obstacle do you think is keeping you from operating according to the target condition?
Staff doesn’t realize they are making mistakes.
Coaching Question: “What is your next step?”
The learner decided to email all 19 staff and tell them to run their own weekly performance report showing some of the errors that occurred. He also asked them to print it out and attach it to the time sheet.
Coaching Question: “What did you expect?”
Errors will decrease by 80% because they will see their errors and self-correct after reading performance report.
Fast forward to one week later and our next coaching cycle:
Coaching Question: “What happened in your last experiment?”
Of the 19 staff, only 7 printed their performance report. Some handed it to me with errors that were still uncorrected.
Coaching Question: “What did you learn?”
It didn’t work as I expected, and there are obstacles I’m not aware of.
The learner continued to work through obstacles and tried the following experiments, one at a time, documenting what happened and what he learned after each one.
- Discuss errors at a staff meeting.
- Provide training to specific errors to specific staff.
- Discuss errors with managers and let them know how important coaching is.
- Track errors by type and by staff name to get more information.
- Send individual errors to staff with screenshots and email.
- Mass email to managers identifying errors.
- Email error report with names to all staff.
Each experiment led to learning and some improvement, but the learner still hadn’t cracked this process or found a way to get 100% accurate every time.
In my coaching, I encouraged him to explore more detail around the obstacles.
Who was making errors? Can we get more specific about the types of errors they were making and how often?
Up until this point, he was tracking error rates using a form similar to the one shown above that I had created. As he dug further into the obstacles, he created the below chart to track errors by type and by person. *Not their real names.
Each X indicates a count of one error for that person over the review period. When I first saw this tracking sheet, I was skeptical. I’d rather see a Run Chart that shows progress over time. However, my job was to focus on the learner’s development, and I know that self-directed learning is more effective than being told what to do.
So, as the learner continued to understand more about the types of errors and who was making them, he decided that his next experiment would be to send this chart out via email to his colleagues indicating the errors they had made. He opted for all names to be included.
The feedback he got from the experiment—not from me—is that staff couldn’t tell if these were active errors that still need to be corrected or if they were cumulative errors over time.
This realization made him decide that his next experiment would be to create a new Excel worksheet inside the same workbook so that he could track errors by week—each user would have a clean slate each week and new errors.
I showed him how to highlight all the X’s to get a count of total errors. Showing the data this way had an impact on the learner and staff and he reached his first target condition.
The tool created by the learner had multiple benefits:
- It was easy for him to fill out when he was reviewing charts
- He didn’t have to convert his findings to percentages
- It was easy for staff to read and see errors
- The learner created the form himself, learned from feedback what didn’t work and then changed it.
This learner is continuing to develop as a problem solver.
I just left a coaching cycle. Here’s what he had to say:
This process has given me so much confidence! I saw these problems with the data every day and I was managing it and fixing it the best way that I could. But when we started measuring the problem, making it visual, defining it and creating a structure for looking at the problems and experiments, it helps me understand it on a much deeper level. I have so much more confidence when I speak to our board, executive director, colleagues and staff because I know the facts around the problem and can speak to it specifically.
Toyota Kata is made up of two practice routines. One for the learner (improver) called the Improvement Kata (1. Challenge; 2. Current Condition; 3. Target Condition; and 4. Experiment) and one for the mentor called the Coaching Kata (see card above). The goal of the routine is not the stated challenge (Step 1), but to develop the problem-solving and scientific-thinking capability of the learner.
Mission Accomplished!
