Decision Mechanics

Insight. Applied.

Slot machine hack

By

slot machine

Wired has an article on a slot machine hack that seems to make use of data analytics.

The hack relies on a weakness introduced in the pseudo-random number generator used by the machines from Austrian gaming company Novomatic. Teams in the casino video the machines using their cellphones. The videos are then sent to St Petersburg where they are analyzed—there are no technical details available on the “back room” analysis techniques.

Once the analysis has been performed, the casino team receive a vibration via a custom phone app that prompts them to press the spin button in 0.25 seconds–the normal reaction time of a person.

It’s not a perfect method, but, statistically, it provides enough of an edge for the slot machine players to consistently come out ahead in the long term.

One fix would be to use a true random number generator, such as a device that exploits the true randomness of quantum physics. However, updating all the existing slot machines would clearly be expensive.

Personal rating dystopias

By

Black Mirror’s “Nosedive” episode portrays a future society, frighteningly like our own, in which people rate each other as a consequence of all kinds of trivial social interactions. Your overall rating is public and determines your job prospects, housing options, social invitations, etc—causing people to obsess over improving them.

As in most public policy decisions you control behavior by tweaking the incentives.

This terrifies me—because I can see it happening. I’ve since been informed that Uber pretty much operates along similar lines.

Now, it has to be said, I’m not a huge fan of social media. And, I’ve railed against the pointless tyranny of personal ratings in the past. To say the least, the world portrayed in the show isn’t my kind of thing.

So, imagine my horror to read in the Wall Street Journal that

Beijing wants to give every citizen a score based on behavior such as spending habits, turnstile violations and filial piety, which can blacklist citizens from loans, jobs, air travel

My concern is that we know that data science is a bit of an art form. False positives appear all the time when profiling potential terrorists. Recommendation systems run the gamut from bloody obvious to downright bizarre. Many corporations can’t begin to make sense of their own data lakes. Basically, it’s a work in progress.

Yet, here we are…on the verge of disenfranchising people on the basis of scores that, I can guarantee you, will be fundamentally flawed.

Visualization pioneer, Hans Rosling, dies aged 68

By

Gapminder screenshot

Hans Rosling has died this week, aged 68. He become famous due to a TED talk he gave in 2006. In the talk he brought international health statistics to life via animated visualizations. He used motion to help viewers make sense of the data.

The software used in his talk was later released as Gapminder and can be used by anyone wishing to display their own data using time-based animation.

Superconsumers

By

The Economist’s Schumpeter column recently cautioned that

Research shows that firms ignore passionate consumers at their peril

These “superconsumers” represent 10% of all customers, but are responsible for 30-70% of sales, an even greater share of profits and almost all consumer insights. In addition, they influence their social circle with their unbridled enthusiasm.

The article offers advice to companies wishing to engage with their superconsumers. First, identify them through data analytics—they are easy to find. Second, reward them—e.g. through loyalty programs.

I’ve written previously about companies that focus on their best customers, as opposed to trying to appease unhappy ones. Superconsumer research seems to support such a strategy.

Idiosyncratic Rater Effect

By

performance review cartoon

A colleague of mine cautions that performance ratings say more about the marriage of the person doing the assessing than the performance of the person being assessed.

Turns out she may have a point.

Most people have some experience with performance appraisals. Maybe as part of an annual salary review. Or even just completing a customer satisfaction survey. It’s become a pretty ubiquitous process over the past decade or so.

Unfortunately it’s process that is flawed. Deeply flawed.

It turns out that we’re really bad at rating others. In fact, we’re so bad at it that someone has given the phenomenon a name—the Idiosyncratic Rater Effect.

And it’s well documented.

It was first shown in a study from 1998, published in Personnel Psychology. This was followed by a 2000 paper in the Journal of Applied Psychology. Then, in 2010, the results were confirmed by yet another paper in Personnel Psychology.

Each study found that the idiosyncrasies of the person doing the rating accounted for over half of the variations in ratings. The three studies put the number at 71%, 58% and 55%, respectively.

No other factor accounted for more than 20% of the score.

As performance management expert Marcus Buckingham put it in an interview

Most of what is being measured is the unique rating tendency of the rater. Thus ratings always reveal more about the rater than they do about the ratee.

Later in the same interview he highlighted the data quality problem facing organizations who rely on these flawed assessments, saying

We all need to grapple with that problem [the Idiosyncratic Rater Effect] as we move into the big data world of the future. We need to figure out how we put good data in.

Once this data ends up in the system, it’s sliced, diced and combined with other data to produce a range of derivative conclusions…all based on a false premise—i.e. that one person is able to effectively judge the performance of another.

There has been some suggestion that the Idiosyncratic Rater Effect can be minimized by averaging across many assessments. But this, amongst other things, assumes that the idiosyncrasies exhibited by raters are independent. That’s a big assumption.

Research on the Idiosyncratic Rater Effect has largely focused on HR. However, the same problems plague customer satisfaction surveys. Developers of iPhone apps have long suffered the vagaries of the App Store review process, for example. In what turns out to be a brutal demonstration of the Idiosyncratic Rater Effect a number of leading iOS app developers made a hilarious video of themselves reading one-star reviews received by their apps.

Business consultant and author Frederick Reichheld has argued for customer feedback to be boiled down to one question—or “The Ultimate Question” as he calls it in his book of the same name. This question is

How likely is it that you would recommend us to a friend or colleague?

In his research, satisfaction scores for this question, when compared with those for other questions, consistently showed the strongest correlation with repeat purchases or referrals.

Amazon is the latest company to say it will overhaul its controversial performance assessment rating system. This follows shake-ups at major organizations such as Goldman Sachs, the Pentagon, IBM and GE. Accenture has ditched ratings altogether.

Others have argued in defense of performance evaluations—looking at ways to “minimize” bias. This strikes me as an attempt to push on with something that is fundamentally flawed because no-one has a better solution.

We need to focus on linking individual performance directly and transparently to well-defined organizational goals. This isn’t easy. It’s been problematic when attempted in the US school system. But, if performance-related management is going to work, and gain credibility, it’s going to need a radical overhaul.