Decision Mechanics

Insight. Applied.

Common use cases for graph databases

By

Neo4j have an informative whitepaper highlighting the top 5 use cases for graph databases.

They highlight the following application areas

  1. Fraud detection
  2. Real-time recommendations
  3. Master data management
  4. Network and IT operations
  5. Identity and access management

A similar article on Data Informed also highlights the role graph databases can play in managing the Internet of Things.

RDDs, DataFrames and Datasets

By

There are now three Spark APIs for working with large volumes of data

  • RDD
  • DataFrame
  • Dataset

Which one should we use? Good question. Jules Damji provides a pretty comprehensive answer in an article on the Databricks blog.

RDD was the original API for working with large volumes of data. The first thing to note is that the RDD API is not being deprecated. It has an important role to play. RDDs make sense when working with unstructured data, such as media or text streams. They are also the best approach if your problem fits neatly within the functional programming paradigm.

However, for the majority of data science tasks, it is likely that the DataFrame and Dataset APIs will be more appropriate. Dataset is a strongly-typed API, whereas DataFrame is untyped. A DataFrame can be thought of as a Dataset of generic (untyped) objects. From Spark 2.0 onward the Dataset and DataFrame APIs will be unified.

Datasets imposes more constraints on the structure of the data. They are not as flexible as RDDs. However, those constraints allow the API to have higher-level functionality and support enhanced compile-time checks and significant run-time performance optimizations.

So, at the risk of oversimplifying, use the Dataset API unless it’s making you jump through hoops. If it is, feel free to use the RDD API. It’s not disappearing anytime soon.

It should be noted that the Spark libraries (such as MLlib) are still being updated to work with the Dataset API, so, in the short term, RDDs may still make sense even when working with structured data.

ScaleR package now available as part of free Microsoft R Client

By

The ScaleR package provides functions for performing scalable and extremely high performance data management, analysis, and visualization in R. It was only available to those who had a Microsoft R Server license—until now.

With the introduction of the free Microsoft R Client for Windows tool you can now work with the full set of ScaleR functions without having to part with a cent.

Of course, there’s a catch—there are constraints. Specifically

[…] the data to be processed must fit in local memory, and processing is limited up to two threads for ScaleR functions.

However, this allows you to prototype your analyses using the free client and push them to SQL Server or Hadoop using Microsoft R Server when you need to scale.

Microsoft R Server documentation is now online

By

The complete Microsoft R Server documentation is now available on MSDN—and is publicly accessible.

It includes comprehensive details of the RevoScaleR High Performance Analytics package. RevoScaleR includes the following analysis functions

  • rxSummary (basic summary statistics)
  • rxLinMod (linear modeling)
  • rxLogit (logistic regression modeling)
  • rxGlm (generalized linear modeling)
  • rxCovCor (covariance/correlation, with convenience functions, rxCov, rxCor, and rxSSCP)
  • rxCube and rxCrossTabs
  • rxKmeans (k-means clustering)
  • rxDTree (classification/regression decision tree modeling)
  • rxDForest (classification/regression decision forest modeling)
  • rxBTrees (classification/regression boosted decision tree modeling)
  • rxNaiveBayes (Naive Bayes classification)