What Do Your Rating Scales Really Mean?

Posted By Debbie on February 20, 2013

I’ve noticed on a few sites that a scale they give you doesn’t really mean what it turns out to mean.

Take eBay. They found that their feedback system of positive, neutral, and negative wasn’t really doing the job. They created 4 criteria, and let people rate 1 through 5 stars. So if you ask me to rate how someone’s shipping speed was from 1-5, it’s a sort of Likert scale. eBay shows the average score for each criterion, and you can be in trouble for having anything under a 4.6 average out of 5 for any criterion. That means you’re in trouble for getting under a 92 on a test. WOW.

Most people will give a 4 out of 5 even if they were very happy. 4 out of 5 feels like the seller was great. Most people would reserve 5 for such amazing service it’s nearly off the chart.

But what most people don’t know is that eBay penalises you for getting anything lower than a perfect score of 5. You’re in super trouble if you get 1s or 2s. But think about what I said about about how any average under 4.6 starts to get mean trouble for eBay sellers. That means that giving someone a 4, which feels like a perfectly good score, actually lowers that seller’s average, bringing them closer to a 4.0 average… and closer to being on eBay’s poop list.

Then is a 5-star scale the right thing?

I’m not sure a 5-star scale is the right way to approach this if you have decided that 1 and 2 are equally bad, 3 and 4 are almost equally mediocre, and 5 is the only good score. It’s almost positive, neutral, and negative all over again just with more granularity and more for sellers to stress over.

I also saw this on eHarmony.

I also saw what I felt was rating scale abuse on eHarmony when I was trying it in 2011. They asked you what you wanted in a partner for a particular quality. For example, is it important that the person have a certain level of education? You pick what level of education, and then there was a slider from 1 to 7.

It turns out that rating something 1 through 6 lead eHarmony to mostly ignore whatever that preference was. When I said having a college degree was of importance 6 out of 7, I got guys who never went to college. When I slid it to 7, I only got people who graduated from college.

To the user, this appears to be binary, not a scale.

I either get guys who match my preference, or they get filtered out completely. On/off. Binary.

Behind the scenes, perhaps a scale is being used. Maybe they weight people with a college degree more than people without because of your preference. But couldn’t this still be achieved with a three-point scale? Let’s say I tell the dating site I want a guy with a college degree. It could then ask me to pick how important it is to me to have a partner with this quality:

  • Not important at all.
  • Somewhat important.
  • It’s a 100% must-have.

This could still be enough info to let the dating site weight people. And it’s enough to let me put my foot down on mandatory qualities where I need to. If he has to have no kids, be of a certain religion, or be a certain ethnicity, this scale of 3 should be enough for a good user experience AND behind the scenes data crunching experience. I mean, how differently will you rank someone who never got a college degree when I say having a college degree is 3 on a scale of 7 vs 4 on a scale of 7?