I recently conducted an inter-rater reliability study for a client. There was some confusion about what this measures. Inter-rater reliability measures agreement. It's a measure of precision, not accuracy. As anyone who's been on social media knows, it's possible for everyone to be in complete agreement, yet utterly wrong. The following diagram summarises the difference between precision and … [Read more...]