As the amount of data we collect continues to explode, attention needs to shift to making sense of it. Tools like Hadoop and Spark allow us to analyse these huge datasets, but they don’t really make sense of it. Senior managers want insights. They want to have their business illuminated by the data. At present, this is done by data scientists analyzing the data and weaving it into a report.
However, this isn’t scalable. Data scientists become the bottleneck. It’s hard to get real-time insights when someone has to plough though the data and then write it up.
Unsurprisingly, there are people working on this problem. Narrative Science have a tool called Quill that they call an “automated narrative generation platform”.
If we are going to start harnessing the power of big data as part of our day-to-day business, tools of this kind will be essential. I presume there is still a way to go before they get even close to the insights a data scientist can provide, but we need to start somewhere.
