Tableau have just announced that Tableau Public Premium will now be free. Tableau Public is a cloud-based service that makes is easy to analyze and visualize large data sets. Working with Tableau is akin to working with a spreadsheet.

The premium services adds the following features:

• support for larger data sets—up to 10 million rows
• a storage limit of 10 gigabytes
• the ability to restrict access to workbooks (the previous free tier could only be used to host public data sets)

The TIOBE Index (of programming language popularity) for April 2015 notes that:

Another interesting move this month concerns Scala. The functional programming language jumps to position 25 after having been between position 30 and 50 for many years. Scala seems to be ready to enter the top 20 for the first time in history.

Scala is the language in which the hugely popular Apache Spark large-scale data processing engine is written—which, no doubt, explains the sudden burst of enthusiasm for the language.

I’ve been listening to extracts from “Birth of a Theorem: A Mathematical Adventure” on BBC Radio 4’s Book of the Week. It’s Cédric Villani’s account of the years leading up to his award of the Fields Medal—the most coveted prize in mathematics.

We rarely get a chance to see the creative process at work. All we get to see is the final result—wrapped up in a neat little bow. We don’t see the blind alleys that were navigated to get there. Villani takes us on a journey through his unproductive lows, his breakthroughs and everything in between.

It’s the same with business decisions. We see the rationalised, justified final decision. Not the messy process of negotiation and estimation that resulted in the decision. If we’re to get better at making decisions, understanding the process is key.

A recent survery of data scientists by CrowdFlower found that, when it comes to challenges

Dirty data is the #1 hurdle…

My own experience leads me to agree. Access to good quality data remains a huge problem. Organizations would do well to invest in improving the quality of their data before boosting their analytics capabilities. Garbage in, garbage out.

One error I see made regularly is not to appreciate that it’s cheaper to fix problems “upstream”—e.g. at the point the data is collected. Better tools, UX and training can significantly improve the quality of the data entering your analysis ecosystem.

Too many organizations see cleaning as a process that occurs centrally, late in the collection pipeline. By then it’s often too late to fix the problem, and all you can do is discard the data. Even worse, you may not notice the problem and use the inaccurate data in your modeling.

Many errors can only be identified in context—and as the data moves further from its origin, context disappears. For instance, if a skate park employee enters the ages of a bunch of customers as being 80, a simple iPad app could ask for confirmation, based on statistical profiling.

However, if the same data were, instead, cleaned by an analyst at HQ a week later, it would be impossible to tell if that was an error or the reunion of the 1955 Olympians. And, discarding unusual, but accuate, data is going to reduce your ability to spot emerging trends, or niche markets.

If you are interested in learning about Apache Spark, a popular large-scale data processing engine, the Introduction to Apache Spark Workshop videos from the 2014 Spark Summit are a good place to start.