Decision Mechanics

Insight. Applied.

Big data and narrative

By

As the amount of data we collect continues to explode, attention needs to shift to making sense of it. Tools like Hadoop and Spark allow us to analyse these huge datasets, but they don’t really make sense of it. Senior managers want insights. They want to have their business illuminated by the data. At present, this is done by data scientists analyzing the data and weaving it into a report.

However, this isn’t scalable. Data scientists become the bottleneck. It’s hard to get real-time insights when someone has to plough though the data and then write it up.

Unsurprisingly, there are people working on this problem. Narrative Science have a tool called Quill that they call an “automated narrative generation platform”.

If we are going to start harnessing the power of big data as part of our day-to-day business, tools of this kind will be essential. I presume there is still a way to go before they get even close to the insights a data scientist can provide, but we need to start somewhere.

What’s hot at the 2015 Strata+Hadoop World conference?

By

It’s always refreshing to see data scientists turn the tools on themselves. Gives the field credibility.

Benedikt Koehle looked at the frequency of ngrams in the abstracts for the 2015 Strata+Hadoop World conference. He concluded, as a result of this analysis, that

2015 will be probably known as the “Spark Strata”

He also notes the resurgence of interest in R, at the expense of Python.

I also notice that the word “APIs” is very prominent—paralleling the growth of interest in other development-related areas.

Free on-line data science courses from Stanford

By

Stanford has a number of free on-line courses in session that might be of interest to data scientists. They started last week, but you can still join.

Statistical Learning

The Statistical Learning course is an introductory-level course in supervised learning with a focus on regression and classification methods. The course uses R, and the text book, “An Introduction to Statistical Learning with Applications in R”, is currently available as a free download.

Mining Massive Datasets

The Mining Massive Datasets course teaches algorithms for extracting models and other information from very large amounts of data, with an emphasis on techniques that are efficient and scale well. The course textbook, “Mining of Massive Datasets”, is currently available as a free download.

Machine Learning

The Machine Learning course provides a broad introduction to machine learning, data-mining, and statistical pattern recognition.

Microsoft to acquire Revolution Analytics

By

Microsoft have already shown R some love by supporting it in their Azure Machine Learning service. But, today they announced an acquisition that borders on infatuation—they are acquiring Revolution Analytics.

Revolution Analytics is a commercial provider of R software, support and services—with strength in the big data area. They are the developers of the open-source RHadoop packages that allow analysts to use R when managing and analyzing data with Hadoop.

Joseph Sirosh, in charge of Machine Learning at Microsoft, said:

We are making this acquisition to help more companies use the power of R and data science to unlock big data insights with advanced analytics.

I’m looking forward to seeing where Microsoft are going with this.