The agenda for Spark Summit East 2015 (18-19 March) in New York City has just been published. I’ve listed the topics being covered and highlighted the ones that piqued my interest.

Developer stream

• Beyond SQL: Spark SQL Abstractions For The Common Spark Job
• Streaming Big Data Analytics with Team Apache: Spark & Spark Streaming, Kafka and Cassandra
• Spark User Concurrency and Context/RDD Sharing at Production Scale
• Power Hive with Spark
• Spark Application Carousel: Highlights of Several Applications Built with Spark
• GraphX: Graph Analytics in Spark
• Experience and Lessons Learned for Large-Scale Graph Analysis using GraphX
• Towards Modularizing Spark Machine Learning Jobs
• Spark Streaming—The State of the Union and the Road Beyond
• Using Spark and Elasticsearch for real-time data analysis
• Accumulo and Spark: Geospatial processing with more distribution, less shuffle

Applications stream

• Spark Plugs Into Your Car
• When Spark meets Baidu
• Plot all the data—Interactive visualization of massive datasets
• Real-Time Recommendations using Spark
• Estimating Financial Risk with Spark
• Spark’ing an Anti Money Laundering Revolution
• Recommendations in a Flash: How Gilt Uses Spark to Improve Its Customer Experience
• Graph-Based Genomic Integration using Spark
• Geospatial and Temporal Analysis and Visualization
• SILK: A Spark Based Data Pipeline to Construct a Reliable and Accurate Food Dataset
• Finding Shoe Stores in more than 100k Merchants: Using Apache Spark to group all things!

Data science stream

• Spark Infrastructure for Lumiata’s Probabilistic Graphical Model of Medical Science
• Un-collaborative filtering: Giving the right recommendations when your users aren’t helping you
• Distributed Graph-Based Entity Resolution Using Spark
• Functionality and Performance Improvement of SparkR and Its Application
• Practical Machine Learning Pipelines with MLlib
• Streaming machine learning in Spark
• HeteroSpark: A Heterogeneous CPU/GPU Spark Platform for Deep Learning Algorithms
• Multi-modal big data analysis within the Spark ecosystem
• Visualizing big data in the browser using Spark
• Interactive Scientific Image Analysis and Analytics using Spark
• Next-Generation Genomics Analysis Using Spark and ADAM

Big data creates opportunities for all sorts of new services—good, bad or just wacky.

It’s difficult to be sure which category the Luce X2 TouchTV from Rhea Vendors falls under.

They have a snack-vending machine that could, in principle, access your medical records via facial recognition. Cross-referencing this with your purchase history would determine if you’d been naughty and, if so, no treat would be vended.

Of course, a passing healthy-looking student would always be happy to help you stick it to the man. But do you really want a vending machine vetoing your lunch?

Seriously, though, the increasing availability of data is going to present us with all manner of vexing opportunities. And the technology is moving faster than our understanding of the consequences.

CrowdFlower have produced a thought-provoking infographic that considers what’s going to be hot in data science during 2015.

It’s interesting that four of the eight topics point to the importance of domain expertise in data analysis:

1. Integrated data scientists—people who are in the thick of the problems, as opposed to external analysis who just see numbers
2. Rich data—focus on what is relevant, as opposed to throwing everything into the bucket
3. Everyone loves data—user-friendly tooling allows people to analyse their own data
4. Data scientist with a BA—recognising that you need more than technical skills to make sense of data

I regularly see data scientists struggle to derive value from data as they fail to grasp the needs of the business. There’s a belief that context-free analysis of the numbers will magically reveal incredible insights. Tragically, such a belief often leads to disappointment and the rejection of an important capability.

Business leaders need to be intimately involved with their data, guiding analysis and making sure it bring insights to real-world problems.

Let’s hope that happens in 2015.

Read more in the CrowdFlower blog post.