If you are interested in learning about Apache Spark, a popular large-scale data processing engine, the Introduction to Apache Spark Workshop videos from the 2014 Spark Summit are a good place to start.
Those damn users
A picture is worth a thousand words. The above photograph, tweeted by @stuartice, illustrates why we need to think about the wider system when solving problems. This is especially true when, as in this case, the wider system involves sentient actors who may not share our goals.
Similar effects can be seen in any situation in which decision-makers are trying to save people from themselves. If we don’t consider the reactions stakeholders will have to a decision, we’re bound to make the wrong one.
Common Core echo chambers
Aankit Patel, a data consultant at NYC Department of Education, has published an interesting article on the Common Core State Standards (CCSS).
The CCSS are nationwide US academic standards for English and mathematics. They were introduced in response to:
- American students’ declining scores on international tests
- differing standards being applied by state education departments
- the prospects for low-skilled workers in the modern workforce
It’s proved to be controversial, with much of the debate being driven by politics.
Patel analyzed 41,000 CCSS-related tweets from 21,000 users during November 2014. He ran a number of network analyses on the data to obtain insights into the nature of the debate.
One of the interesting results was the diagram shown above. This is based on a “who follows who” analysis. It shows liberal (red) and conservative (blue/green) groups fiercely debating in their own echo chambers. There’s very little cross-pollination in the debate. While the debate isn’t a simple one of liberal supporters against conservative opponents—liberals are more split—there’s seem to be little evidence that stakeholders are receptive to dissenting views.
If this analysis of tweets reflects the wider debate, it’s difficult to see anything constructive emerging from it.
If you have tools, everything looks chartable
Data and charts seem to be inseparable. If there’s data, the temptation to visualize it can be irresistible. We have all these amazing tools that produce amazing graphics with the press of a button—and showing the data feels honest.
But it’s often a “cover your ass” strategy. The job of a data scientist is to make sense of data…draw insights from it—not summarise it in a technical picture.
Sarah Slobin, a graphics editor at the Wall Street Journal, came to a similar conclusion while working on a story recently.
Data visualization is a worthy craft. We can tell important stories with data. We can illuminate complex issues and expose lapses. We can inform and delight readers and make good apps. We need to remember that behind the data are stories and inside those stories are people and those people are connected to the statistics in a way that we never will be, regardless of how badass we are with our tools, how rockstar-smart our code is, or how facile we are in manipulating the information.
Data science with F#
R is great for doing data analysis, but it can be frustrating as a programming language. It doesn’t feel familiar to developers—it pays greater homage to its statistical heritage—and there’s little consistency across the community-built packages.
F#, on the other hand, is first and foremost a programming language. And, as it’s a .NET language, it has access to a enterprise-grade tooling and libraries. As a functional language, it’s a natural fit for scientific and big data computations.
If you are interested in exploring F# as a data science language there’s a great place to get started—the data science section of the F# site.
As well as learning how to use F#, you’ll learn how it integrates with most of the popular data science tools, such as:
- Excel
- R
- MATLAB
- Python
- Mathematica