Misleading Metrics

For data pros, defining performance metrics is an important job that requires using the right data and some trial and error to deliver accurate measures. Since the subject of defining metrics is boring to a lot of people, I thought I’d illustrate some points by poking fun at one of the most abused consumer metrics relied on during the holiday shopping season: customer satisfaction.

Although I try to support local businesses when I shop for a gift, there are times when I need to shop online for a product that is not available at my local stores. Since I cannot test drive the item myself, I sometimes rely on customer reviews to tell me if the product is worth the purchase. For online customer reviews, I am truly an Amazon review junkie. Their reviews help enrich the shopping platform Amazon is hosting, but as informed consumers know, you have to understand what data makes up a metric and how it was obtained before you can trust it.

Let’s take a look at one type of Amazon customer review you will find, the Free Lunch Customer Rating. This is where the customer received a free product (or sample) in exchange for giving a positive customer review or rating. I love these reviews – they give the item a perfect score (i.e. 5-star rating) and proceed to tell you how the product changed the world. Truly hilarious. Some reviewers even disclose that they received the item free in exchange for a positive review, which is very noble but enough for me to read before moving on. While Amazon has taken steps to disallow these exchanges, the ratings are still on Amazon and other retail sites. These reviews skew the number of positive responses and make it difficult to find ratings from the unbiased reviewers. So that we can learn what not to do with our own metrics, let’s overanalyze what these free lunch ratings measure:

Their formula:

Since most of the reviews came from the customers who are happy taking the free lunch, the average satisfaction rating is skewed toward those perfect ratings. This produces a satisfaction metric that is close to 100% or 5 stars. Sounds amazing!

A more trustworthy formula:

The satisfaction of customers who actually paid for the product is what shoppers really care about, so obviously we would be better off eliminating those free lunch reviews.

Product sellers who publish their own satisfaction ratings have many mathematical tricks and dishonest practices they can employ. While the example I showed seems to point out the obvious, it highlights many of the considerations we need to make when defining our own business performance metrics:

1. What key question(s) should the metric help answer? The more specific, the better. For example, business stakeholders that use a metric may be trying to measure operational efficiency of an assembly line or sales performance for a particular product. Before worrying about the data, we should focus on the process or performance area we need to measure and form the key question we want to answer using the metric. With Amazon customer reviews, the original intent was to give consumers the ability to measure the average satisfaction of shoppers who purchased the same item.
2. With a key question in mind, efforts should be taken to collect all of the data that can provide a complete picture of the performance being measured. Nothing more nothing less. For example, the efficiency of a production line should include data from all phases of assembly including time for restocking and preparation, not just the production run. You need all of the data elements or ingredients needed to reflect a complete and accurate measure.
3. What population of data are we measuring against? In the Amazon customer review example above, the customer population included a group that skewed the average because they received a free product. We know that businesses want the metrics they use to reflect the true health of the business, not inflated numbers that use data like the reviews submitted by our free lunch reviewers.
4. What is the goal? For each metric you define, you may want some benchmark or goal to compare it against. If your metric is too broad, a goal may not be easy to set for it. Key Performance Indicators (KPI’s) are typically paired with a goal that targets the same scope as the metric. A metric without a goal to compare against is just a metric, and that might be all you need. But if measuring against a constant goal is your objective, you should define your metric with a tight scope that is easy to assign a goal or target value.
5. Are we measuring on a level playing field? When a metric is being measured, are there times where it is expected to be much higher or lower than other time periods? For example, if production output is always much lower during PM hours, then it might be appropriate to split the output performance metric into two distinct metrics (e.g. AM vs. PM units produced). The same principle holds true for variables that are not time periods. Having more than one metric for distinct conditions also enables you to set the appropriate goals, each adjusted for the different conditions (e.g. morning and evening hours, seasons, holidays, etc.).
6. Can you back it up? This is where the good metrics hold their ground and the others get thrown to the wolves. Presenting business performance metrics will inevitably expose weak areas, which is a good thing. Unfortunately, questions will be raised about dreadful performance metric scores as well as those that show supernatural scores. Those Amazon free lunch reviews I read through seemed to be perfect scores submitted over a limited time. Seemed fishy to me. To maintain credibility for any metric, you need to show the underlying data as well as the specific definition and ingredients used to make up that calculation. Having this detail at the ready not only validates your metrics, it keeps expectations about them in check. If business users disagree with a metric score or how it is calculated, you can always take them with you back to the drawing board and publish a metric with an agreed upon definition.

Businesses that make good use of metrics can be confident when answering questions about the performance of their organizations. The people who define the metrics have to be careful to use the right ingredients or they will wind up losing credibility with what they report.

I wish you and your family a wonderful holiday season, full of gift purchases made using informative and objective customer reviews. If you shop online using Amazon or Yelp and use customer reviews, you should check out Fakespot which will analyze the reviews for you. And please remember to FIRST support your local small businesses! They do it better.

Data Lens – Misleading Metrics