Seminar paper · the review
What the literature actually measures
Over-refusal is widely named. Far less often is it measured — with a refusal rate, on a stated instrument, against a stated baseline. The review separates the two, and the separation is the finding.
PRISMA 2020
The flow closes arithmetically
Excluded, carried forward and included sum exactly to the number screened. Click any stage to see what it holds.
Select a stage.
Why records were excluded
Most of the field does not measure refusal at all
Of 370 screening exclusions, 264 failed one criterion: the paper never reports a refusal rate on a stated instrument. A further 95 studied attack success only. That is not a judgement about their quality — it is the criterion doing its job, and the reason the included set is small.
| Reason | OpenAlex | Scopus | Total |
|---|
The two databases agree closely on the shape of the exclusion profile — 64.6 % against 67.9 % for "no refusal measurement" — which is what you want from an index-independent rule.
The included corpus
What kind of study gets included, and when
Reliability, honestly
One screener, so no κ
With a single screener there is no between-rater agreement to compute. What can be computed is whether the same rule, applied twice by the same person, lands in the same place. That is a weaker and different quantity, and it is labelled as such everywhere it appears.
The re-screen was taken minutes after the first pass, so short-term memory inflates the figure: it is an upper bound on how stable the rule would be under a genuinely delayed re-screen, and it is never presented as agreement between two people.