Insight. Applied.
Microsoft is providing technology for tonight’s Iowa caucus that should
facilitate [the] accuracy and efficiency of the reporting process
It also gives the company, which is seeing strong grow in its cloud services, to showcase its cloud and mobile offerings in a high stakes setting.
Data Informed have published interview with Doug Cutting on the future of Hadoop.
He makes a number of observations.
On MapReduce…
MapReduce is on its way to being legacy.
On important Hadoop developments…
I think Kudu is very exciting; a new storage engine that offers a lot of low-latency, random-access capabilities that HDFS doesn’t while still permitting the fast analytics that you can do on the flat files in HDFS.
On deployment…
…providing a vendor-neutral cloud, so you don’t have to be locked into an Amazon or a Microsoft or a Google but retaining the option to move your data and operations, we think is important.
On hardware developments…
it’s pretty clear what the next hot hardware area is: memory technologies that are coming out that give you orders of magnitude faster access to persistent storage and also, combined with that, hardware that lets you access that memory over a network without involving the remote CPU, so basically every machine on a cluster can have micro-second-level access to all the memory in that cluster. And that’s going to be a game changer.
The NFL is fitting players’ shoulder pads with data sensors. This will allow the real-time collection of metrics such as position, speed and acceleration.
In last year’s Pro Bowl sensors were installed in footballs to measure the length of throws. Surprisingly, given “Deflategate”, they have chosen not to expand this idea.
In addition to being used for the “game stats” that are popular in the US, the data will be used by coaches and be available to the public—for a fee. Will be interesting to see what the market can do with it.
While NFL players are clearly “high value assets”, falling prices for sensor technology and computing resources mean we are going to see similar approaches deployed in more “mundane” areas—such as ranching.
Jan Mulkens recently published an article on Microsoft’s recent rush to enhance it’s data science offering. And, as he illustrates, they have been very busy this year.
He highlights a number of their flagship data science initiatives.
Other significant data science activities at Microsoft this year, in my opinion, include:
I’m looking forward to what they come up with in 2016.
Twitter have an interest in detecting anomalies in their service. Anomalies could be down to user engagement, spamming or technical issues. Regardless of the reasons, it’s something they want to know about when it happens.
To aid detection of anomalies in their time series data they have developed, and open-sourced, an anomaly detection package for R. Their algorithm is based on the Generalized ESD Test and can detect both global and local anomalies. The package is also capable of detecting anomalies when seasonal and trend factors are present in the time series data.