Decision Mechanics

Insight. Applied.

Microsoft “FedExNet”

By

Cessna 208B Caravan N775FE FedEx

People, in my experience, tend to find it hard to get their head around many “big data” concepts. It’s only when they attempt to implement initiatives, and are frustrated by the basics, that they start to “get it”.

One of the most basic things that people seem to misunderstand is the challenge of moving data around. Most big data tutorials assume that you already have terabytes (or petabytes) of data on your cluster. However, how does it get there in the first place? If you have hundreds of terabytes of data in traditional storage how do you get in onto your Spark (for instance) cluster?

There’s no magic answer. Moving that amount of data is painful—plain and simple.

Microsoft has recognized this and introduced an Azure Import/Export service. Basically you snail-mail them your hard disk(s) and they upload the data to distributed storage via their high-speed secure internal network.

It’s a start, but is trailing Amazon’s solution to this problem by, oh, around 100 petabytes.

2017 Big Data Landscape

By

big data landscape 2017 infographic

Matt Turck has published the 2017 edition of his annual review of the big data landscape. It include an infographic showing the key players in various sub-fields.

This year’s main finding was that

Big Data provides the pipes, and AI provides the smarts.

Social rating and machine learning

By

I recently blogged about the risks of social rating systems. Machine learning adds another dimension to this.

In her book “Weapons of Math Destruction”, Cathy O’Neil highlights the umaccountability of algorithms used to make decisions that have a significant impact on peoples’ lives. The details of these algorithms are often undocumented—for commercial or security reasons—making it difficult to challenge their conclusions.

Unaccountable big data algorithms serve to amplify the risks posed by social scoring.

Personal rating dystopias

By

Black Mirror’s “Nosedive” episode portrays a future society, frighteningly like our own, in which people rate each other as a consequence of all kinds of trivial social interactions. Your overall rating is public and determines your job prospects, housing options, social invitations, etc—causing people to obsess over improving them.

As in most public policy decisions you control behavior by tweaking the incentives.

This terrifies me—because I can see it happening. I’ve since been informed that Uber pretty much operates along similar lines.

Now, it has to be said, I’m not a huge fan of social media. And, I’ve railed against the pointless tyranny of personal ratings in the past. To say the least, the world portrayed in the show isn’t my kind of thing.

So, imagine my horror to read in the Wall Street Journal that

Beijing wants to give every citizen a score based on behavior such as spending habits, turnstile violations and filial piety, which can blacklist citizens from loans, jobs, air travel

My concern is that we know that data science is a bit of an art form. False positives appear all the time when profiling potential terrorists. Recommendation systems run the gamut from bloody obvious to downright bizarre. Many corporations can’t begin to make sense of their own data lakes. Basically, it’s a work in progress.

Yet, here we are…on the verge of disenfranchising people on the basis of scores that, I can guarantee you, will be fundamentally flawed.

Highwaynet

By

AWS Snowmobile

There are a lot of advantages to storing your big data in the cloud. Start-ups can get going without having to set up servers and data centers.

However, what about organizations that already have data centers? How do you move petabytes of valuable data to a cloud provider? You rarely see much coverage of this—because it’s a difficult problem.

There are no clever technical solutions. Amazon’s solution extends one of the oldest electronic data transfers strategies around—the sneakernet.

First, they introduced the Snowball—a suitcase of SSDs that can hold 80 terabytes (TB) of data. You fill it up at your data center and ship it to Amazon.

“80 TB?!”, I hear you scoff. “I’m not moving my kid’s blog.”

OK. Well, fear not—Amazon has your back. Let me introduce you to their Snowmobile—an 18-wheel truck that’ll hold 100 petabytes of data.

Drive up, fill it with your cat GIFs, tear down the highway and upload it to AWS. Now you’re running in the cloud.

Clarkson must have had something to do with this.