Insight. Applied.
By editor
In his recent keynote Google CEO Sundar Pichai said
In fact, when I look at where computing is headed, it’s clear to me that we are evolving from a mobile-first to an AI-first world.
Devices like Amazon’s Echo and Google Home are “always-on” intelligent assistants designed to sit quietly and await our bidding.
As the devices recede into the background it’s the AI-powered services that will come to the fore.
By editor
Microsoft have a new conference aimed at data scientists. The first Microsoft Data Science Summit will be held between 26-27 September 2016 in Atlanta GA.
The conference will cover
…the latest Big Data, Machine Learning, Artificial Intelligence, and Open Source techniques and technologies.
By editor
David Smith, R Community Lead at Microsoft, talks about how they are using R.
He covers both how it is being integrated into the product line and how it is used internally to analyse operational data.
By editor
There are now three Spark APIs for working with large volumes of data
Which one should we use? Good question. Jules Damji provides a pretty comprehensive answer in an article on the Databricks blog.
RDD was the original API for working with large volumes of data. The first thing to note is that the RDD API is not being deprecated. It has an important role to play. RDDs make sense when working with unstructured data, such as media or text streams. They are also the best approach if your problem fits neatly within the functional programming paradigm.
However, for the majority of data science tasks, it is likely that the DataFrame and Dataset APIs will be more appropriate. Dataset is a strongly-typed API, whereas DataFrame is untyped. A DataFrame can be thought of as a Dataset of generic (untyped) objects. From Spark 2.0 onward the Dataset and DataFrame APIs will be unified.
Datasets imposes more constraints on the structure of the data. They are not as flexible as RDDs. However, those constraints allow the API to have higher-level functionality and support enhanced compile-time checks and significant run-time performance optimizations.
So, at the risk of oversimplifying, use the Dataset API unless it’s making you jump through hoops. If it is, feel free to use the RDD API. It’s not disappearing anytime soon.
It should be noted that the Spark libraries (such as MLlib) are still being updated to work with the Dataset API, so, in the short term, RDDs may still make sense even when working with structured data.
By editor
Learning Tree International just published my article on using Microsoft Cognitive Services to perform face recognition and emotion detection.