Intra Rater Reliability Puns

Hi all,

I'm working on a project where I am reporting on the intra-rater reliability of a viewer who has watched video analysis twice and has repeated coding on specific movements and the frequency and duration of when they occur (e.g. Over 4 innings, the athletes made 27 high intensity throws, 14 sprints that lasted a total of 23 seconds, etc...). I am reporting the standard error of measurement (SEM) and Coefficient of variation (CV%). However, certain movements occurred very infrequently during the 4 innings analyzed (like 0.25 +- 0.5 jumping events), which caused the SEM and CV% to be above the acceptable limits of 10% due to the small sample size.

I need to run a post hoc analysis on these variables to determine how many instances I would have needed to view before they would have had acceptable SEM and CV%. It's expected that the amount of time it would have taken me to watch for the appropriate number of jumps or slides to get an acceptable SEM and CV% would have been unreasonable (10+ hours) within the scope of this project, but I just need to be able to report that I did my due diligence.

Any suggestions on how I can calculate the threshold needed to give me acceptable (<10%) SEM and CV%?

Thanks for the help.

Hello everyone,

I was wondering if the Cohen's kappa statistic can be used as a measured of intra-rater reliability ?

For example, consider the case of one rater performing at two separate times points the rating (binary) of the same set of objects, both ratings are done at sufficiently separated times so that the rater is supposed not to recognise the objects. Could Cohen's kappa be used as a measure of agreement between both ratings ? Even though it seems to me there is no independence between both set of ratings ...

Thanks for your input !

The experimental question was whether practice with rating voices on a variety of dimensions makes one more reliable at rating those voices. 10 recordings of different voices were rated on 8 variables (pitch, loudness, etc) once per week for 15 weeks. The same 10 recordings were used each week but they were presented in a random order.

I thought Cronbach's Alpha might be able to measure whether reliability increased over time (e.g., compare alpha from the first 3 weeks vs. to alpha for the last 3 weeks). However, that treats the repeated nature of the ratings as if they are independent, which they are not. Is there a better way to approach this that avoids the problem of running multiple independent analyses (e.g., doing simple correlations between each week for each rating of track)?

I often see subtle misuses of interrater reliability metrics.

For example, imagine you're running a Search Relevance task, where search raters label query/result pairs on a 5-point scale: Very Relevant (+2), Slightly Relevant (+1), Okay (0), Slightly Irrelevant (-1), Very Irrelevant (-2).

Marking "Very Relevant" vs. "Slightly Relevant" isn't a big difference, but "Very Relevant" vs. "Very Irrelevant" is. However, most IRR calculations don't take this kind of ordering into account, so it gets ignored.

I wrote an introduction to Cohen's kappa (a rather simplistic and flawed metric, but a good starting point to understanding IRR). Hope it helps + I welcome feedback!

I recently published a blog on the risks of relying on inter-rater reliability (IRR) when assessing data quality in machine learning.

I wanted to share here because I think it may be useful for folks learning about ML and the difficulties of getting good training data for models (and evaluating that data for quality).

For those that have direct experience evaluating data with IRR, curious to hear about your experiences (what worked, what hasn't worked etc).

Also want to know how many of you answer the logic problem correctly :)

Does anyone else feel like IRR is often relied on too heavily for assessing data quality? Now that so many ML use cases involve highly subjective tasks like content moderation, sentiment analysis, etc, I think we need to reconsider how to think about the relationship between IRR and data quality.

Recently wrote a blog on this subject, but curious for this community's thoughts on the matter too. Any examples of instances where an over-reliance on IRR has caused problems for you down the road?

In my study of peer ratings, I want to measure consistency ( the extent to which reviewers consistently apply a scoring rubric to writing evaluation). After a while of reading, I was pointed to intra-inter rater reliability in assessment and ICC. But I am not sure how researchers entered data into SPSS or whatever else. Specifically, in my study:

• Each writing will receive only 4 ratings
• Each rater, on average, rates 2 writings only. Which means one rater doesnt rate the same set of papers as another.

I do know that before ICC, an analysis of variance should be carried out but my knowledge only extends to two-way ANOVA tests. Since each student evaluates only a small, unique subset of all of the papers, there will be a reviewer-paper interaction. I'm not sure with the circumstance above, 2 way ANOVAs can be applied.

Every youtube tutotial I've been able to find only calculate agreement and the raters seem to rate the same set of data. Could anyone please offer advice on which direction I should take? Greatly appreciated.

Hello stats people!

I am wondering what test would be most appropriate to compare how similar two raters code for durational behaviours while observing.

Eg if two people watch a panda bear for an hour and look for behaviours and Coder A records the panda sleeping for 30mins, eating for 17mins and playing for 13mins, and the Coder B records the panda sleeping for 27mins, eating for 18mins and playing for 15mins, how would one compare the statistical agreement of that?

Thanks in advance for any insight!

One apparently confused redditor has made the following claims about the attractiveness assessments used in research into preferences:

> https://cdn-images-1.medium.com/max/2000/0*aiEOj6bJOf5mZX_z.png > Look at the male messaging curve. > > Now again look at the woman's curve. > > http://cdn.okcimg.com/blog/your_looks_and_inbox/Female-Messaging-Curve.png > Why would men be messaging women they mostly find attractive while women seem to be messaging men they on average find unattractive? > > Here's a break down of how this works: > > Let's say there are 3 ice cream flavors: A B C, and subjects are to each rate them 1 - 5. And this happened: > > Subject 1 > > A 1 B 3 C 5 > > Subject 2 > > A 5 B 3 C 1 > > Subject 3 > > A 1 B 5 C 1 > > Subject 4 > > A 1 B 5 C 3 > > So our results are: > > 5 1s 3 3s 3 5s > > 3 good flavors > > 8 less than good flavors > > The subjects would be rating 80 percent of ice cream flavors less desirable. Yet they each still individually PREFER ice cream flavors that are on average rated as less than desirable by the group. > > Black pillers along with LMSers deliberately ignore the messaging curve while pretending that women all have the same tastes and judge 80 percent of men as unattractive and so the 20 percent that remains must all be the same guys. > > The messaging curve easily debunks that and reveals what's really happening. > > The power of stats.

Side-stepping the utterly questionable (aka wrong) math and implicit assumptions involved in interpreting the sum count of all <5/5 ratings on 3 ice cream flavors as subjects overall rating "80 percent of (three!) ice cream flavors less desirable," let's focus on the crux of this post: that the ratings are too "variegated" to be reliable.

First, I'll elaborate on something I mentioned here in response to this redditor's concerns. An excerpt:

> The argument you're trying to make is that some subgroup or diffuse heterogeneity precludes any statistical analyses. Except f

So, I'm looking at the formula for Cohen's kappa for inter-rater reliability.

Why is it so complicated? Why not just take the difference between number of observed agreements vs. number of agreements expected by chance, and then decide if that difference is big enough?

What does the kappa formula do that goes beyond that? Or is it a way of doing that comparison and controlling for. . .??

I have 52 items and participants (experts) have to decide whether each item belongs in 1 of 3 levels, which I have coded as 0, 1, 2. These items were given to 10 participants and I want to calculate some sort of measure of agreement/reliability between them. Initially I used fleiss's Kappa but now I am a bit confused as to whether that is the correct statistic to use. Further, the difference between agreement and reliability is confusing to me.

Thanks in advance!

I'm working on a paper. Two raters are taking discourse units and categorizing them. How can I calculate inter rater reliability? What's a satisfactory number? Also, is there an easy way to do this is Google sheets or Excel?

Hey,

I have some data where two different raters are rating 12 of the same cases on a scale consisting of 10 items. The items range from 0 to 2, but the majority of cases are a 0. For example, it would not be unusual for a single case to be rated with nine "zeroes" and a single "two".

This is causing issues when I run inter-rater reliability analyses, I've used both Kappa and ICC and get the same problems, one item I have one rater who rated all ten items as "zero" whereas the other person rater nine "zero" and one item as "one". This has caused both Kappa and ICC to say there is zero agreement. However, there is agreement because on the other nine cases we agreed and rated it as zero! I'm wondering if there is a way to deal with this type of data for inter-rater reliability analyses?

I'm working with 2 other coders (we might add one or two more - we're still in the training stage) to code conversations between counselors and clients. We have 25+ codes we are using for counselor statements. Any given counselor statement could be given multiple codes or no codes.

What is the best test of inter-rater reliability in this situation? A colleague wants to use Cohen's kappa. My understanding is that that is only appropriate for comparing 2 raters, and when the categories are mutually exclusive and exhaustive. She is suggesting that we view each code as a "yes" or a "no" to make it fit the "mutually exclusive and exhaustive" criteria. I have less statistics experience, but in my research I've come across Kippendorff's alpha as a measure of inter-rater reliability for qualitative research like ours.

Has anyone here worked with Krippendorff's alpha? What are the weaknesses? What is the best measure to use in what we're doing?

I'm also super pumped about this project, so if you want any additional details or have other pointers, I am very happy to talk about that.

We're starting a new project where one objective is to compare different protocols for training Research Assistants (RAs). The hypothesis is that the novel training protocol will lead to greater inter-rater reliability than the standard training protocol. The problem is that we're not sure which statistical test to use when comparing protocols.

We initially thought to measure intraclass correlation (ICC) and run a t-test between ICCs, but realized each group provides only one ICC point-estimate, so no standard deviations, no distribution of ICCs, so no t-tests. (Correct?) We've been doing some reading and are also considering Krippendorff’s alpha, but have never used it before and are not sure that would be right, or how to compared between protocols.

Specific details: Participants report three qualitative responses. RAs rate each response on three variables (C, R, U) using a 1–5 Likert scale. There will be 6 total raters in a counter-balanced cross-over design: 3 do the novel training first, 3 do the standard training first; they rate items; then each group does the other training and rates more items. (i.e. control for order-effects). We want to see which training protocol results in the greatest internal reliability.

Which statistical methods would you use to test for differences between training protocols on a measure of inter-rater reliability?
Would you recommend any design changes to address this question better?

First time here, so I apologize in advance if there's something inappropriate about this question. For the life of me I haven't been able to find the answer to this question. I'm planning a retrospective chart review with 2 abstractors who are dividing the charts to review. I'm planning a pilot study to help improve the inter-rater reliability. My question is how do you calculate the number of charts that need to overlap between the abstractors in order to calculate inter-rater reliability? How would you calculate inter-rater reliability in that case?

Hi everyone!

We have 20 participants in this study, and each participants performs a certain task which is video recorded. this task need 10 steps to be performed completely.

2 independent raters watch these videos and rate each step from 0-5 (0= did not do the step and 5= perfect performance)

For example , rater 1 would watch the first video and give his rating on each step performed by participant 1. Later rater 2 watches the same video and gives his rating on each step by the same participant. It would end up to look something like this:

now I have 400 readings , 200 for each rater (10 for each participant by each rater)

What would be the better test to test inter-rater reliability?

Thanks!

So it's been a while since I had a stats class that covered this. One of the analyses in my current research involves independent raters rating recurrence plots (example) qualitatively based on which of 3 different visual patterns they see.

What test would you suggest I use to measure rater agreement? I know that Cohen's kappa is a thing but I wanted to double check and see if there are any assumptions required that I need to meet, or if there are other tests I need to run to check for things like if the raters are behaving like true independent witnesses, etc.

Thanks! Any help is appreciated

Hi everyone I am looking to work out some inter-rater reliability statistics but am having a bit of trouble finding the right resource/guide.

In our study we have five different assessors doing assessments with children, and for consistency checking we are having a random selection of those assessments double scored (double scoring is done by one of the other researchers - not always the same one due to logistical restrictions).

As far as I can tell from looking into it one way to calculate whether there is consistency among the researcher and double scorer is through calculating a Kappa statistic using SPSS syntax. However as it is we have about 50 separate variables so manually calculating Kappa for each researcher pairing for each variable is likely to take a long time.

I'm wondering if anyone knows of any faster way or better way of doing this?

Any help would be super appreciated!

Hello, would like some input on if I'm using the right statistics for assess inter-rater reliability for some survey data.

A survey was sent out to 12 people to rate a number of items on 3 different criteria - each criteria on a 7 pt Likert scale. All the items were rated by all the 12 respondents.

Is Fleiss' kappa what I need to compute to assess agreement between the 12 respondents? Thanks!

Hi all,

I'm reading this paper:

Ifantidou, E., & Tzanne, A. (2012). Levels of pragmatic competence in an EFL academic context: A tool for assessment. Intercultural Pragmatics, 9(1). http://doi.org/10.1515/ip-2012-0003

In the methodology, the author is examining exam scripts written by students and rated by both authors. They note that "the t-test statistical analysis performed on indpendent samples did not present a statistically significant difference between the mean scores of the two authors-cum-raters, which confirms the inter-rated reliability of the specific process of assessment" (p. 61).

My question is: is this a reliable measure of inter-rater reliability? Generally, when reading papers in TESOL or education-related fields, researchers use Cronbach's alpha to confirm inter-rater reliability. However, that varies on a scale. A Cronbach's alpha of .98 would be very reliable, whereas a Cronbach's alpha of .71 would not (at least not between raters).

At what point would Cronbach's alpha level stop being "significant"? I mean to say, if we were to run both Cronbach's alpha and the t-test, and got a .75 on Cronbachs but non-significant results on the t-test, is that an indicator of reliability? Is that even possible? Is there some sort of correlation between the Cronbach's alpha and the t-test results from which we (or I) can draw conclusions?

Thanks for the help. I'm not a big statistics guy, and this seemed like an odd way to establish inter-rater reliability.

I was wondering if anyone could help me out with trying to determine inter-rater reliability for an experiment I am working on. In the experiment there are 5 people doing measurements on 6 different images. We need to test the reliability of these measurements between each group member. We have been using the one-way ANOVA test, but are not certain this is the test we need. If anyone could help me that'd be great! (: An example of a spreadsheet in excel looks like the first table in the link.

Hi there guys, i seem to have some trouble computing the inter rater reliability score for my dissertation. I will try to be as clear in explaining as possible.

Two raters observed 3 chimpanzees (one minute sampling intervals for one hour) and gave a point each time they showed a certain behaviour on the one minute dot.

The problem i encountered is that computing Cohen's Kappa will give me a score of around .55 which is not very accurate since it passes all different numbers as disagreements (even when the difference is 1).

Is there any test i can use to calculate this in a more accurate way that would account for percentage of agreement?

I'll try to explain my predicament as clearly as I can.

I have a set of 20 coded items, each of which has been coded by two independent raters across 14 variables with three possible responses for each variable: -1, 0, +1. Thus, each item has an overall score between -28 and +28. I know the formula for calculating Cohen's Kappa (although I'll probably let SPSS do it for me), but I'm unsure about which numbers to use. More so than the overall scores for the coded items, the agreement on ratings of the individual items are important to my study.

What I'm wondering is whether or not I should calculate Cohen's Kappa for each of the 20 items and then report the average, or whether I can/should calculate it for each individual coded variable across all 20 items (n = 280). Or better yet, is there a way to calculate for each individual variable while taking into account that every set of 14 variables constitutes a single rated item?

I hope I've made some sense with this post, I'm happy to clarify if needed. Thanks.

Hi r/statistics, I've got a problem that's twofold:

I'm trying to calculate the inter rater reliability of a set of 8 raters with a set of 29 questions, answered after watching a set of 100 videos. Thing is, not all the raters answered all the questions. The first issue is that of the eight raters, only two watched all 100 videos, one watched 96, and the other seven watched the same 20 each (those 20 were randomly selected from the set of 100). The second issue is that a good deal of the questions had "N/A" as an answer, but some raters scored an N/A for certain questions on certain videos while other raters gave numerical scores for the same question/video pairs.

I've looked into Fleiss' Kappa for finding inter rater reliability for more than two raters, but I have no idea how to account for the missing and N/A answers. No idea how I've made it this far without so much as opening a stats book, please help me r/statistics!

I'm in the process of developing a scale that measures aggression in individual online posts on a popular social media site. The scale has 14 items, and each post is rated on all 14 items. Individual ratings are -1, 0, +1, and each post has an overall score ranging from -28 (complete absence of aggression) to +28 (maximum aggression detectable by scale).

I'm familiar with the basic idea of Cohen's Kappa from the one behavioral statistics class I've taken, but I'm not sure how to best calculate it in this situation. After training two independent coders who have no interaction with one another, I had them both rate 20 posts. Now, I know I can use the overall scores for each post to easily calculate inter-rater reliability. However, I also want to capture the level of agreement on each of the 14 variables. As far as I can tell, I have two options (and I'm not sure if either of them will actually work). The first that comes to mind is to calculate Cohen's Kappa for each of the 20 sets of 14 ratings, and then calculate the average (not sure if this is good practice or not). My second idea would be to pool each individual rating (280 in total) and calculate Cohen's Kappa for that. Are either of these options viable? If not, what would you suggest?

As previously alluded to, my experience with statistics is somewhat rudimentary (I'm familiar with descriptives and hypothesis testing, but not much else), so I would greatly appreciate it if in your responses you would please pardon my ignorance about these matters and try to suggest "basic" solutions. Thank you.

I am working on a reliability analysis trying to compare the reliability of two faculty scoring a paper using the same rubric (ordinal). There were 140 papers scored by two faculty using the same rubric. I started with using cohen's kappa and then read that method is a better analysis for nominal data and I should use weighted kappa instead (which I am wrestling with figuring out how to get the SPSS extension to work). I also read that intraclass correlation (ICC) is more approriate. However, the results between using Cohen's kappa and ICC is vastly different. Can someone shed some light on the relationship between Cohen's kappa and ICC?