Spark 1.5.0 has now been released—and it’s a significant one for the data science community. Databricks, in their announcement blog post, state
Another major theme of this release is data science: Spark 1.5 ships several new machine learning algorithms and utilities, and extends Spark’s new R API.
Improvements of note include better coverage for the pipeline API and an MLlib API for SparkR.
