Decision Mechanics

Insight. Applied.

R now has native pipes

By

The latest release of R introduces an F#-style pipe operator.

Pipes have been available in R for a while, via the magrittr package. However, with the release of R 4.1.0, pipes are supported in base R.

The syntax has changed from that used in magrittr. The magrittr pipe operator is %>%. The new base R pipe operator is |>.

So, instead of

df %>% head()

we have

df |> head()

R 4.1.0 also introduces a lightweight, lambda-style function syntax, which can be usefully combined with the new pipe operator when piping to an argument other than the first one.

iris |> {\(x) lm(Sepal.Width ~ Sepal.Length, data = x)}()

Why is everything so complicated?

By

Ever expressed frustration that stuff is unnecessarily complicated? If so, a new paper published in Nature might have an explanation.

Apparently, when solving problems, we have a natural inclination to add things. For instance, when asked to make a design symmetrical, people are more likely to do so by adding features, rather than removing them.

You can see evidence of this in a range of technical domains.

  • UX designers add more cues to address usability issues when they should make the the design less "noisy".

  • Software developers add more code to fix problems instead of simplifying existing code.

  • Data scientists chase R-squared improvements by adding more features.

Simplification is a crucial component of effective decision science. Approaches such as confrontation analysis strip away extraneous details and focus your attention on your immediate dilemmas.

It’s not all doom and gloom. You can "nudge" people to simplify using appropriate cues. For instance, maybe you could display prominent "lines of code" counts, or focus on predictive R-squared when modelling.

Given our instinctive biases we must be vigilant in the search for creeping complexity. Science is showing us that it’s more work to keep it simple.

As President Woodrow Wilson explained,

If I am to speak ten minutes, I need a week for preparation; if fifteen minutes, three days; if half an hour, two days; if an hour, I am ready now.

Photo by Gayatri Malhotra on Unsplash

The company you keep…

By

weird combination

Researchers have demonstrated how objects can be hidden by exploiting correlation bias.

Computer vision systems, like other machine learning technologies, rely on correlation and context. They learn that certain things often appear together–like mice and keyboards. But you don’t expect to see a platypus wielding a chainsaw, for example (…you’ve no idea how much I wanted to find that splash image…).

Artwork can be challenging, for example, as it can depict items that have no business being in the wider context.

This opens up another angle for hacking AI systems. The researchers were able to make a computer vision system fail to recognise a STOP sign by placing it next to fruit—a suitably bizarre combination.

Another example of the brittleness of "intelligence as correlation". The state-of-the-art in machine learning is leading us down the latest AI blind-alley. This one may be a little darker than before.

Photo by celery soup on Unsplash.

The rise of functional programming

By

Yes, I know functional programming has been around since the 1950s. And, yes—Excel users are functional programmers. There’s nothing particularly new to see here.

Or is there?

There’s been sustained hype around functional languages for a few years now. Yet they have failed to break into the mainstream. Languages such as Haskell, Clojure and F# remaining frustratingly niche.

The top ten programming languages rankings in recent surveys by Stack Overflow and GitHub are devoid of functional programming languages.

Except that they aren’t.

Functional programming is on the rise, but through approach, techniques and style—not language choice.

Mainstream languages are hybridizing—adding functional capabilities to their (predominantly object-oriented) features. C# has LINQ. Java has added Lambda Expressions. JavaScript has many features that support a functional approach—which is actively encourage React, for instance. Rust, Kotlin, Swift…and a host of other modern languages…have a distinctly functional flavour to them.

Functional programming languages have a lot going for them. They encourage (or enforce) functional design principles. And they often bring desirable features with them, such as powerful type inference. However, the real revolution in functional programming is happening through traditional enterprise languages—gradually introducing practising software developers to the ideas without forcing them into a "paradigm shift".

Why does functional programming matter?

Functional programming brings with it skills and concepts that are valuable when building modern applications.

One benefit for data scientists is that functions look like…well…functions. Mathematical functions to be more specific. The closer the code is to the equations you are using, it easier it is to make sure you have the correct implementation. There’s a reason that Excel and R draw heavily on functional ideas.

Today’s applications are increasingly asynchronous. Mobile…web…microservices…functions-as-a-service (FaaS). Asynchronous software is hard to write—and even harder to debug. Functional programming encourages practices such as immutability and isolation of side-effects that make it easier to reason about asynchronous code.

There’s a view, admittedly controversial, that the object-oriented programming (OOP) paradigm that’s dominated software development for over 30 years has retarded progress.

Alan Kay, who coined the term "object-oriented", has long criticised mainstream OOP.

I’m sorry that I long ago coined the term “objects” for this topic because it gets many people to focus on the lesser idea. The big idea is messaging.

He has also expressed his dissatisfaction with major OOP languages.

Java is the most distressing thing to hit computing since MS-DOS.

I invented the term "Object-Oriented" and I can tell you I did not have C++ in mind."

It’s been called the "trillion dollar disaster". Its "three pillars" are a combination of the unremarkable (encapsulation) and positively harmful (inheritance).

OOP’s singular focus on "nouns" has given us an industry of convoluted patterns in an attempt to impose some order–leading to CommandExecutor classes with executeCommand methods.

As we build more complex, asynchronous systems, OOP is failing to scale effectively.

The tools used in functional programming—map, filter, reduce, copy-on-write, etc—are simpler and more generic than OOP patterns. And, as functional programming has a mathematical foundation (lambda calculus), it opens the door to more advanced tooling for automatically diagnosing problems in our code.

Unlike OOP, functional programming encourages us to separate our data from our calculations. This makes our code easier to test and aligns perfectly with today’s data-centric workflows. Data pipelines fit snugly into functional designs.

Can I use functional techniques without adopting a functional programming language?

Yes! That’s the point. Absolutely.

Functional programming is a set of concepts and techniques. It’s a way of thinking about and writing code. Sure, functional languages make it easier to apply those ideas, but you can apply them in any programming language.

And, as mentioned previously, modern languages are increasingly becoming hybrid languages—supporting the use of multiple paradigms.

Focusing on the concepts, rather than language features, will allow you to take your functional programming skills with you as you move from language to language during your career.

I’d argue that it’s worth learning at least one functional-first language as it helps consolidate your ideas, but it’s not essential.

Just start writing functional code. Today. Your customers will thank you for it.