The Secret to Surveys: Go to the Source

If you have ever read a Lean book, blog or article, you’ll know that “value” is defined by the customer. While often referenced, we don’t spend enough time thinking about what this really means.

What Do Customers Truly Want?

Henry Ford reportedly once said, “If I’d asked my customers what they wanted; they’d have said a faster horse.” While perhaps not terribly empathetic, Henry had it right. Sometimes customers don’t know what they truly want. They may be hampered by their own assumptions, not realizing that there’s a better solution out there somewhere.

Or there may be different people or separate groups within one customer who want different things. I’ve been at several customer meetings where the Purchasing and Engineering representatives disagreed about our supplier performance. A perfect response or solution for one was awful for the other. Subsequently, one group loved dealing with us as a supplier—and the other…not so much.

Sometimes customers don’t know what they truly want. They may be hampered by their own assumptions, not realizing that there’s a better solution out there somewhere.

What Drives Customer Feedback?

In these instances, it doesn’t take long to come to the conclusion that you can’t please everybody all the time. As the adage goes: “Cheap, fast or good. Pick two.” Inevitably, this leaves someone disappointed. When those disappointed people are the ones who fill out your Market Feedback Analysis (MFA), your QMS-complaint customer survey or your employee satisfaction survey, you can rest assured you’ll be receiving some less-than-stellar feedback.

That feedback can be damaging; sometimes severely so. Companies can lose customers, people can lose jobs and, in some extreme cases, entire shops go out of business. Customer feedback is serious business, and should be treated that way.

Customer feedback is serious business, and should be treated that way.

As I write this, we’re in the thick of shopping for Christmas and, as a lot of people are doing, we’re completing the bulk of our shopping online. Not only is it more convenient than going out to stores, it’s way more efficient, much less aggravating, and often less expensive. So what’s not to love?

Part of my process (being an engineer, I like to say that I can’t help it) is to do a lot of research on products before I buy them, which I also do online. Part of that research is reading the reviews that people leave for products. If there seems to be an emerging pattern—poor workmanship, inferior packaging leading to damage, long delivery times, what have you—I’ll move on and look elsewhere. No sense in following others down a path of inevitability, right?

The Trouble With Customer Reviews

Here’s the issue: Reviews are suspect. Heck, lots of reviews aren’t even real. There’s literally a whole industry built around writing and posting fake reviews (both positive and negative) on sites like Amazon and Google. My friend Mark Graban posts and tweets about this issue as it exists on Amazon Books, where it appears to be pretty rampant. This is more than just disheartening, it’s problematic. At an extreme, it undermines the trust within a social network and makes everything, no matter the source, questionable.

At our firm, we get these “offers” probably once a month. These folks claim that by posting many 5-star reviews, we will enjoy “higher Google rankings” and “a guaranteed 50% increase in website traffic.” These reviews are all fake, of course, and we never take them up on their offers, but honestly—if we did, how could anyone tell?

Another method some employ is to pay—or offer rebates to—actual customers for good reviews. We recently had a roof repair done at our house by a local company. I’d done my research (which included reading online reviews), and they checked out. They did the work satisfactorily—they were on time, did the work pretty quickly, and did an overall good job. I wasn’t necessarily “wowed,” but I was certainly satisfied, and grateful that I didn’t have to find time to perform the work myself.

When it came time to pay the bill, the owner offered me a $50 discount in exchange for a “5-star Google review.” I politely declined. I didn’t challenge him on it—I was pretty sure I wasn’t going to convince him that doing such a thing was mildly unethical—but I remember thinking, “Wouldn’t it be better to ask me how I actually felt about how the job went, then take that feedback and somehow incorporate it to make your processes better?”

Wouldn’t it be better to ask me how I actually felt about how the job went, then take that feedback and somehow incorporate it to make your processes better?

True Customer Feedback Makes You Better

That’s the soul of Kaizen after all and, in the end, will only help his company to be better and thus provide consistent 5-star service. As consumers merely looking at online ratings, though, we can’t tell the difference. As I write this, his company has 105 5-star reviews on Google, and an overall rating of 5/5. Undoubtedly, this helps him win more business when, as I did, people look him up and read the online reviews.

This example is notably less severe than having a bunch of flat-out fake reviews, and I believe that if his company were actually performing “bad” work, those reviews would get posted with some regularity. My belief is that they do good work (I actually spoke to people I know personally before hiring him, so I felt confident that I was making a good choice), and that he’s just a local guy trying to build up his business.

While some reviews may not be fake or false, they may be incorrect. Consider this Google review of a local paving company (redacted):

As you can tell, “Heather” never removed her review as the owner requested—since I could still find it. Overall, this company’s reviews are positive—an average score of 4.6 out of 22 reviews is pretty good for a small, local company that only a smattering of people will use once in a lifetime. But what if the scores were different? With only a couple more 1-star reviews their overall score would drop quite a bit and, when viewed on the front page of Google, may cause potential customers to move past their company without any investigation.

In this particular case, it’s pretty clear that Heather either confused them with another company, or she was being duped by someone merely claiming to be from this company in and effort to steal her money (this has been known to happen around here with paving contractors). Either way—the real company is paying the price for this bad review, even though they haven’t done anything wrong.

What’s the Story Behind the Data?

Don’t get me wrong: data is important, but it’s only part of the story. The other, perhaps more important part, is what’s behind the data. This is often hard to know, and can take a fair bit of investigation. For instance, an employment candidate may have poor reviews in his/her file, or receive a bad referral, but what is the data behind those? Is the employee really “bad” or were they merely subject to the outcomes of bad processes? And what about the reviewer who gave the marks? Do they actually know what they’re talking about and can we trust them and their report? What are their motivations?

Data is important, but it’s only part of the story. The other, perhaps more important part, is what’s behind the data.

Different than metrics and KPI’s, ratings are subjective. They include all the conscious and unconscious biases from the rater, and can’t be reliably treated as “absolutes.” They’re inherently suspect for that reason—even more so when you consider that people purposely try to manipulate them on occasion (both positively and negatively).

Within the industry, we deal with ratings all the time. Customer survey feedback and employee satisfaction survey results can be skewed by bias and individual experiences. Maybe a purchasing agent at the customer was having a bad day, so instead of a bunch of “5’s” they gave you all “4’s”. Other times, motivations are pure, yet still yield uncertain results. What might be a “4” for me could be a “2” for someone else. Ultimately, it’s relative. Ratings can reveal patterns and shifts over time, but aren’t necessarily finite.

Get the Story Behind the Data

What to do? Short answer: go to the Gemba. Go see things for yourself. Talk with the people there who are doing the work and ask them about it. They are the absolute best place to go to learn about what it is they do—after all, they do it every day! They can describe the data behind the numbers on the chart—not the perception (which is a conclusion of data) but the data itself. Anyone can give feedback that says “your on-time delivery needs to be better” without having any data, but it takes intimate knowledge of the process to know what is affecting the outcome.

Talk with the people there who are doing the work and ask them about it. They are the absolute best place to go to learn about what it is they do.

Staying in contact is key. We see marked differences at clients who take the time to work with their customers and suppliers versus those who keep everyone at arm’s length. This difference is apparent in their feedback, and in their growth rates (spoiler—staying in touch is better!).

In a world that moves fast—where data can be easy to come by—it still remains vitally important to stay connected on a personal level. Yes it takes more time than simply reviewing data, but the benefits far outweigh the investment.