Follow

My God, it's full of stars...

Why is 4 out of 5 stars good for a restaurant but near-catastrophic for an Uber driver? Why do hotels and movies have two kinds of ratings but books and consumer electronics only have one?

I mean, a rating is just a rating, right? If I like it, I give it more stars. If not, less. I've made a small study of star ratings, which is to say I've read the work of marketing analysts, and it turns out it's not that simple.

If you've ever driven for Uber, or you know someone who has, then you know how hard they fight to get five stars from every rider. Why? Why do fractions of a star below perfection matter for rideshares but not for, say, food trucks or hair salons?

To start, you have to understand that consumers occupy a position of near-total uncertainty. We have no idea which coffee maker is best for us, or whether it's worth it to drive across town for a higher-rated mechanic?

In the old days, consumers knew the price of a good or service and that was about it. Clever marketers would spend a lot of time designing branding and packaging the signaled certain cues in an attempt to nudge people to purchase, but unless they happened to know someone who had personal experience with the product, there was little to go on but looks and price.

If a shopper was contemplating a large purchase, such as a car or computer, they might consult a trusted expert or magazines like Consumer Reports, which made a living testing home appliances and financial services the like -- products people were willing to research before buying.

But here, as everywhere, the internet cut out the middle man. Consumers could start sharing our experiences directly with each other. If we had a bad experience with a doctor or lawn service, we could post it on sites like Google, where others could do the same.

That meant that star ratings, ostensibly a simple measure of quality, aggregate numerous experiences, and so indirectly measure a great many variables besides quality -- for example, consistency.

Human are biologically risk-averse. They like consistency, which is why McDonald's is perennially popular. We want to know what we are getting and will often choose a mediocre service that is consistently mediocre over one that is typically extraordinary but occasionally bad.

In other words, above a certain level, quality starts to matter less than other considerations, and those considerations vary by product.

The reason Uber drivers need 100% 5-star ratings is because a ride from point A to point B is effectively fungible. The biggest determinants of quality -- traffic, weather, speed limit -- are out of the control of the driver.

That didn't use to be the case. We starting tipping taxi drivers because in the days before Google maps, local knowledge of shortcuts and alternate routes varied by driver, and a good one really could get you where you were going faster.

That's largely not the case anymore. So, though taxis have not been fully automated, an important first step has already happened. Urban transportation was commodified, and there is nothing more ripe for automation than the repetitive production of a commodity.

What does that have to do with star ratings? As much as they are risk-averse, humans also dislike choosing at random. Where they don't have a reason, they will invent one (and rationalize it after the fact.) However, they're also lazy, and the reason they invent will be the easiest one possible.

The easiest way to pick between Driver A and Driver B is star rating, because it's staring right at us in the app. We may know there is no real, measurable different between a driver with 98% 5-star ratings and one with 97% 5-star ratings, but what other determinant is as obviously available?

But what is true of Uber rides is not true of books or restaurants, where two of the same type will still be very different. Books are not commodities, not because they're art but because there's no accounting for taste.

Readers understand that it's not simply a matter of picking the book with the higher rating. For one, how do you compare the latest Tom Green tear-jerker to the latest installment of a space opera series, where both have roughly identical star ratings?

Not every book is trying to be Shakespeare. Some are trying to be nothing more than a fun summer read, and if that's what we enjoy (or think we will), then we want a book that does a good job of that, not one short-listed for the Booker Prize.

This is why hotels have multiple ratings on travel sites. The first, determined by the site, establishes the overall quality of the hotel. Five-star hotels are supposed to be the epitome of luxury. 2-star motels are a cheap place to crash for the night.

The second, customers reviews, determine how well the property met that expectation. You can have a 4-star 5-star hotel, for example, where they didn't bring you fresh towels every day, and a 5-star 3-star hotel, where they did.

Online movie rental apps are getting at something similar when they include both an aggregate critic rating as well as an audience rating. The critic rating hints at what kind of movie it's trying to be and the audience rating lets you know how well it achieved that.

In my world, NUMBER of reviews matters more than star rating, which is why I'm constantly urging not to feel bad about being honest. Above a certain level, it really doesn't matter much how many stars you give it. Really.

What's matters is that you add your rating to the total number, because readers use number of ratings as a proxy for popularity.

A book with more reviews is assumed to have engaged more people, so readers consistently tend to pick, say, a 3.6-star book with 800 reviews over a 4.4-star book with 80.

It's unfortunate, in a sense. As a class, readers tend to pride themselves on their independence of thought, yet when choosing a book, they effectively follow the crowd. But complaining about it does nothing to change the reality.

@rickwayne people give way too much weight to ratings. Ever watch Black Mirror? But until we have a more objective measurement of customer satisfaction/dissatisfaction, what can we do? (Given enough volume of ratings, most things tend to average out to 3.9-4.2, unless they are REALLY great or REALLY suck.)

@dajbelshaw @hollyjahangiri

To be fair, it's not original to me. I read it in a marketing study a number of years ago.

@rickwayne @hollyjahangiri

It's a good insight. Also, looking forward to reading your new book (I haven't read your work before!)

@dajbelshaw @rickwayne My new book?? Or Rick's? I don't have a NEW book out. I've been writing online, lately. But now that I'm retired from the day job, no excuses - time to write a NEW book. I'm betting you haven't read my old ones, either. 😂

Unless you have children/grandchildren/nieces/nephews, you may want to read some of my writing on Medium or my own blog, while I get back to work at writing a NEW book.

@dajbelshaw @rickwayne I had not seen that. It's unfortunate, as I've had good experiences with products from both of those.

There's another nasty one out there to beware of, and Amazon does little or nothing to help authors affected by it - see the second scam mentioned here ("The Goodreads Extortion Scam"):

victoriastrauss.com/2021/05/14

@rickwayne dip into Net Promoter Score and social media listening tools and...

Geez. Big money to be made. And kind of damned if you do, damned if you don't.

@rickwayne I simply don't engage with that foolishness. I decline to give stars to any site or app for any reason whatsoever.

@fossil You're free to do that of course, but if you could ever sully yourself, it helps small producers compete with the big guys.

@rickwayne No, it's actually totally fucking meaningless, made so by Uber, WordPress and the like.

@rickwayne Oh, God, you don't want to know about just how complicated rating systems get. And part of it stems also from misuse of them by management.

When you see anything other than 5/5 as catastrophic, you've got ridiculous management somewhere behind the scenes. The same kind of boneheads who use the expression "give it 110%" without irony. To that crowd anything less than perfect is failure, so your "five star" system is really a "like/dislike" system.

1/n

@rickwayne On top of that you have the problem that no two people agree on what the various star levels mean. Some people think "3/5" is "fine" while others would think it means "mediocre" and still others would think it means "bad" and so on.

2/n

@rickwayne Then you get the manipulators. *THEY* think the product is mediocre (say 3/5 in their rubric) so when they see the average rating is 4.2/5, say, they get morally offended and vote 1/5 not because that's what they really believe but instead because they want to force the average rating lower.

3/n

@rickwayne When I set up ratings these days, I offer only two (sometimes at customer insistence three) ratings: good/bad (or good/OK/bad). Further instead of having it be an overall "what's your feeling on everything in aggregate" rating (which has its own major issues) I'll divide it by fields: helpfulness/timeliness/accuracy/friendliness say, or whatever is appropriate for the domain, with a score for each.

4/n

@rickwayne Because aggregate ratings have an added problem over and above what rating systems always have: what if, say, I order a product and the product was fine, the sales staff were helpful, but the logistics fumbled the delivery and took two weeks instead of two days. An aggregate rating will be artificially low because of irritation even in a reasonable customer, pulling down ratings on the parts that were just fine. This is hard to t ease out of analysis.

5/n

@rickwayne If, however, each major field is independently rated you can focus in more quickly on what went wrong and where you need work. Your customers are happier and your life is less stressful.

6/6

@zdl

More dimensions are helpful with some products, sure. You see it on some hotel sites now, where you can rate cleanliness and a couple other factors separately. Audible does the same with audiobooks. You can rate the story separately from the audio production, which is important.

But adding more dimensions does drive the number of responses down, so there's a trade off. In some cases, it will be worth it. (I'm not sure books need multiple dimensions, for example.)

But yes, great points!

@rickwayne Even books I can see at least three dimensions:

Quality of writing (important for the authors).

Quality of presentation—binding, artwork, typesetting, etc. (important for printers).

"Perceived value" (important for publishers).

That's a matrix with three thumbs-up/down choices. That's not going to drive away too many.

If you go insane and have 57 different dimensions, well, yes, only crazy people will respond. :D

@rickwayne As an example, Taobao here on each order solicits customer feedback (and literally pays the customer for it!). Each item in your order from a given shop is rated for authenticity (three levels) to help combat fraud, and then the overall order is rated (sadly out of 5 stars where 4 is considered bad by the public...) on vendor helpfulness, logistics speed, and overall satisfaction.

(I don't like the overall satisfaction rating for reasons I gave, but the rest is good.)

@zdl

From the business side, it's very hard even to get people to click a single rating, even without text, which is why Amazon recently recently dropped the text requirement. (They used to force you to write at least 20 characters or something.)

There's probably some cultural differences in there. The Japanese, for example, love their 8-dimensional spider charts and will flash them up on TV commercials. That wouldn't happen here or in most European countries.

@rickwayne There are tangible benefits to doing the reviews here. Taobao gives you points that you redeem for (admittedly small) reductions in transaction fees.

It turns out that my Psychology 101 courses were dead on in the behavioural segment: you get the behaviour you reward. Almost everybody spends the fifteen seconds or so per transaction to review. 😀

Sign in to participate in the conversation
Mastodon

Server run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!