Decision Mechanics

Insight. Applied.

  • Services
    • Decision analysis
    • Big data analysis
    • Software development
  • Articles
  • Blog
  • Privacy
  • Hire us

Microsoft “FedExNet”

April 22, 2017 By editor

Cessna 208B Caravan N775FE FedEx

People, in my experience, tend to find it hard to get their head around many “big data” concepts. It’s only when they attempt to implement initiatives, and are frustrated by the basics, that they start to “get it”.

One of the most basic things that people seem to misunderstand is the challenge of moving data around. Most big data tutorials assume that you already have terabytes (or petabytes) of data on your cluster. However, how does it get there in the first place? If you have hundreds of terabytes of data in traditional storage how do you get in onto your Spark (for instance) cluster?

There’s no magic answer. Moving that amount of data is painful—plain and simple.

Microsoft has recognized this and introduced an Azure Import/Export service. Basically you snail-mail them your hard disk(s) and they upload the data to distributed storage via their high-speed secure internal network.

It’s a start, but is trailing Amazon’s solution to this problem by, oh, around 100 petabytes.

Filed Under: Big data Tagged With: Azure, data egress, data ingest, data transfer, export, import

Highwaynet

December 5, 2016 By editor

AWS Snowmobile

There are a lot of advantages to storing your big data in the cloud. Start-ups can get going without having to set up servers and data centers.

However, what about organizations that already have data centers? How do you move petabytes of valuable data to a cloud provider? You rarely see much coverage of this—because it’s a difficult problem.

There are no clever technical solutions. Amazon’s solution extends one of the oldest electronic data transfers strategies around—the sneakernet.

First, they introduced the Snowball—a suitcase of SSDs that can hold 80 terabytes (TB) of data. You fill it up at your data center and ship it to Amazon.

“80 TB?!”, I hear you scoff. “I’m not moving my kid’s blog.”

OK. Well, fear not—Amazon has your back. Let me introduce you to their Snowmobile—an 18-wheel truck that’ll hold 100 petabytes of data.

Drive up, fill it with your cat GIFs, tear down the highway and upload it to AWS. Now you’re running in the cloud.

Clarkson must have had something to do with this.

Filed Under: Big data Tagged With: Amazon, AWS, cloud, data transfer, Snowball, Snowmobile, upload

Search

Subscribe to blog via e-mail

Subscribe via RSS

Recent posts

  • Self-driving car from 1958
  • Sentient AI
  • Is functional programming more effective than object-orientated programming?
  • Assumptions can ruin your k-means clusters
  • # The Forer effect

Copyright © 2023 · Decision Mechanics Limited · info@decisionmechanics.com