Handy VS Code extension (from Microsoft) for viewing and cleaning data.
The Trolley Problem
The trolley problem is a classic ethical dilemma that asks what you would do if you had to choose between saving one person or saving many people from a runaway trolley. For example, would you pull a lever to divert the trolley from hitting five workers on the track, but instead hit one worker on a different track?
This problem is important to generative AI because it illustrates the challenges of programming machines to make ethical decisions that may involve human lives. For instance:
- How should a self-driving car decide who to save or harm in a crash scenario?
- How should a medical robot prioritise patients in an emergency?
- How should a military drone distinguish between combatants and civilians?
Different people may have different moral values and preferences, so there is no clear-cut answer to the trolley problem. Moreover, AI systems may not have all the relevant information or context to make the best decision. Therefore, it is crucial to ensure that generative AI systems are aligned with human values, transparent in their reasoning, and accountable for their actions.
Counting votes using Excel
Excel strikes again. There were only two outcomes and the spreadsheet got it wrong. You’d be better flipping a coin.
Accuracy vs precision
I recently conducted an inter-rater reliability study for a client. There was some confusion about what this measures.
Inter-rater reliability measures agreement. It’s a measure of precision, not accuracy. As anyone who’s been on social media knows, it’s possible for everyone to be in complete agreement, yet utterly wrong.
The following diagram summarises the difference between precision and accuracy.
Precision indicates whether the assessments are similar. If multiple people access the same service, and give the same ratings, your survey is precise. There isn’t a lot of variability in precise ratings.
However, it doesn’t mean they are accurate. Systemic bias leads to high precision, but low accuracy. Racism, sexism, domain specialism, etc. can cause biased, but precise, scores. Accuracy measures whether assessments are correct on average. It’s harder to gauge in complex domains, as it’s not always clear what is being assessed. However, it’s more important than precision.
We’d rather be generally correct than specifically wrong. Of course, we want ratings that are both precise and accurate. Inter-rater reliability only addresses the less important half of the problem.
It’s not because we have insufficient data…
In 1998, Neil Postman wrote critically about the Age of Information.
If there are children starving in the world—and there are—it is not because of insufficient information. […] If there is violence on our streets, it is not because we have insufficient information. If women are abused, if divorce and pornography and mental illness are increasing, none of it has anything to do with insufficient information.
Replace "information" with "data" and he could be writing in 2023.
Our obsession with data correlation doesn’t allow us to intervene in our system to improve things. Countless graphs watching the world burn.