People, in my experience, tend to find it hard to get their head around many "big data" concepts. It's only when they attempt to implement initiatives, and are frustrated by the basics, that they start to "get it". One of the most basic things that people seem to misunderstand is the challenge of moving data around. Most big data tutorials assume that you already have terabytes (or petabytes) of … [Read more...]
2017 Big Data Landscape
Matt Turck has published the 2017 edition of his annual review of the big data landscape. It include an infographic showing the key players in various sub-fields. This year's main finding was that Big Data provides the pipes, and AI provides the smarts. … [Read more...]
Social rating and machine learning
I recently blogged about the risks of social rating systems. Machine learning adds another dimension to this. In her book "Weapons of Math Destruction", Cathy O'Neil highlights the umaccountability of algorithms used to make decisions that have a significant impact on peoples' lives. The details of these algorithms are often undocumented---for commercial or security reasons---making it difficult … [Read more...]
Personal rating dystopias
Black Mirror's "Nosedive" episode portrays a future society, frighteningly like our own, in which people rate each other as a consequence of all kinds of trivial social interactions. Your overall rating is public and determines your job prospects, housing options, social invitations, etc---causing people to obsess over improving them. As in most public policy decisions you control behavior by … [Read more...]
Highwaynet
There are a lot of advantages to storing your big data in the cloud. Start-ups can get going without having to set up servers and data centers. However, what about organizations that already have data centers? How do you move petabytes of valuable data to a cloud provider? You rarely see much coverage of this---because it's a difficult problem. There are no clever technical solutions. Amazon's … [Read more...]