0Quick Answer

1What Is Judgment Sampling?

Judgment sampling selects units because an experienced person believes those units are the right ones to examine. An auditor picks the ledger entries that look irregular. An inspector picks the machines most likely to be failing. A supervisor picks the three sites that best represent the region. In each case a human expert, not a random mechanism, decides who is in the sample.

That makes the researcher the measuring instrument, and instruments have to be calibrated. In quantitative sampling you validate the procedure; in judgment sampling there is no procedure to validate, only a person. So the two questions that matter are whether the judgement is reliable, meaning another competent expert would reach a similar selection, and whether it is systematically distorted, meaning known cognitive biases pushed it in a predictable direction. Neither question is answered by the sample size.

Both are answerable. Reliability is measured by having two or more judges select independently from the same candidate pool and computing chance corrected agreement. Bias exposure is assessed by working through the specific heuristics that are known to distort expert selection: anchoring on the first cases seen, availability of memorable cases, halo effects from reputation, confirmation of an existing hypothesis, and simple accessibility. This tool does both.

Figure 1.1 High percent agreement can hide poor reliability, because most of it comes from agreeing on exclusions.

2Setup: The Pool, the Judges and the Bias Check

Cognitive bias exposure scorecard

Eight heuristics known to distort expert selection. Each answer scores 0 for low exposure, 1 for moderate and 2 for high. A higher total means the judgement was more likely to be pushed in a predictable direction.

Judges: one card per judge

Enter each judge and the case IDs they selected, comma separated. IDs can be anything consistent: numbers, file references, plot codes. The tool compares selections pairwise across the pool. Judge names are editable.

3Results

4Interpretation of Results in Detail

How to read each number

Percent agreement. The share of the candidate pool on which two judges made the same decision, counting both joint selections and joint rejections. It is intuitive and it is almost always misleadingly high, because when you select 8 cases from 200 the two judges agree on 192 exclusions before they consider a single inclusion. Never report percent agreement on its own for a selective task.

Expected agreement. How much of that agreement two judges would have reached by chance alone, given how often each of them selects. This is the quantity percent agreement ignores and kappa subtracts. When both judges select rarely, expected agreement is very high, which is precisely why raw agreement looks so good.

Cohen kappa. Agreement above chance, scaled so that 0 means no better than chance and 1 means perfect. The Landis and Koch bands are conventional: below 0 poor, 0 to 0.20 slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, above 0.80 almost perfect. For a selection task, moderate is a reasonable target and substantial is strong. A kappa near zero with high percent agreement means the two judges agreed on almost nothing that mattered.

The kappa paradox and PABAK. Kappa has a well documented failure mode: when the thing being judged is rare, kappa can be low or even negative while agreement is genuinely high. This is not a quirk to be explained away, it is a reason to report a second statistic. PABAK, the prevalence adjusted bias adjusted kappa, equals 2 times percent agreement minus 1, and it removes the dependence on how often selection occurs. When kappa and PABAK diverge sharply, the divergence itself is your finding: it says agreement is real but selection is rare, and you should report both figures with that explanation rather than quoting whichever number flatters you.

Fleiss kappa. When three or more judges each rate every case, Fleiss kappa gives one overall chance corrected agreement figure across the whole panel rather than a set of pairwise numbers. It answers a subtly different question: not whether two specific people agree, but whether the panel as a whole is applying a shared standard. Report Fleiss for the panel and the pairwise table underneath it, because a good overall figure can hide one judge who disagrees with everyone.

Jaccard overlap. The share of cases that either judge selected on which both agreed. Unlike percent agreement it ignores the exclusions entirely, so it answers the question a practitioner actually asks: of everything anyone flagged, how much did we both flag? It is often the most honest single number in this table, and it is always lower than percent agreement.

The selection rate per judge. How many of the pool each judge picked. Very different rates between judges are informative in themselves: one person applying a much looser threshold is not the same problem as two people disagreeing about which cases matter, and kappa alone cannot distinguish them. Compare the rates before you interpret the kappa.

Times each case was selected. The case by case map shows which units every judge flagged, which only one flagged, and which none did. Unanimous cases are your defensible core. Cases picked by exactly one judge are where the judgement was doing the work, and those are the ones a reviewer will ask about, so have a reason ready for each.

The bias exposure index. Eight heuristics, scored 0 to 2, rescaled to 0 to 100. This is not a statistic and it corrects nothing. It is a structured way of writing down what you already suspect about your own selection process, so that the limitations section names specific mechanisms instead of gesturing at subjectivity. Below 30 means the process had reasonable safeguards; above 60 means the selection was probably driven as much by the heuristics as by the criteria.

Independence of judges. This modifies everything else. If the judges discussed the cases first, or one saw the other choices, then high agreement measures conformity rather than reliability. The tool flags this because a kappa of 0.9 from two people in the same room is worth less than a kappa of 0.5 from two people working blind.

What none of this can tell you. Two judges can agree perfectly and both be wrong in the same direction. Agreement measures reliability, not validity. If both experts share the same training, the same institutional assumptions and the same blind spot, your kappa will be excellent and your sample will still miss what matters. High agreement plus a high bias score is the combination to worry about most, because it looks reassuring and is not.

5How to Write Your Results in Research

Use the templates below. Each one names the judge, the basis, the agreement statistics and the bias exposure, which is what reviewers and audit reviewers look for in a judgmental selection.

Rules that make a judgement defensible in writing

1. Say who judged and what qualified them. A judgment sample rests on the judge, so anonymising the expertise removes the only warrant the design has.
2. State the judgement basis as criteria, not as instinct. "Entries irregular on amount, timing or vendor" is defensible; "entries that looked wrong" is not.
3. Report the candidate pool size. Selecting 8 of 200 and 8 of 12 are entirely different acts, and without N the reader cannot tell which you did.
4. Give the selection rate. It sets the context for every agreement figure that follows.
5. Report kappa and PABAK together when selection is rare, and explain the divergence rather than choosing the flattering number.
6. Confirm the judges were independent, or say plainly that they were not. Agreement between people who conferred is conformity.
7. Name the biases you were exposed to. Specific mechanisms read as competence; a blanket claim of objectivity reads as naivety.
8. List the cases only one judge selected and give the reason each was kept or dropped. That list is where the judgement is visible.
9. Do not report a margin of error. There is no sampling distribution behind a judgement.
10. Distinguish reliability from validity explicitly. Say that agreement shows consistency, not correctness.

Common wording mistakes and the fix

6Formulas Used

7How to Use This Tool

1. Type the engagement name and, in one line, the basis on which cases were judged.
2. Enter the candidate pool size N, meaning every unit the judges could have chosen from.
3. Record the expertise level of the judges and whether they worked independently, because both modify how the agreement figures should be read.
4. Work through the eight cognitive bias questions honestly; scoring yourself low here only misleads you.
5. Add one card per judge and paste the case IDs that judge selected, comma separated.
6. Use consistent IDs across judges. The tool matches on the ID text, ignoring case and spacing.
7. With two judges you get Cohen kappa and PABAK; with three or more you also get Fleiss kappa for the whole panel.
8. Or switch to the upload tab and click the columns that should each become a cluster, one judge per column.
9. Click Score Bias and Measure Agreement, then read the kappa card and the singleton cases first.
10. Copy the methods paragraph into your report and download the judgement report for your file or working papers.

8Detailed Reference Tables

Table 8.1 The eight cognitive biases in expert selection

Table 8.2 Interpreting kappa, the Landis and Koch bands

Table 8.3 Why kappa can look wrong: the prevalence paradox

Table 8.4 Judgment sampling against the alternatives

9Example Results (8 Worked Cards)

10How to Collect Raw Data in the Field

10a. The 18 point protocol for a defensible judgement

Plan

1. Write the selection criteria before you look at a single case. This is the only defence against anchoring, and it costs ten minutes.
2. Define the candidate pool explicitly and count it. Without N there is no denominator, no selection rate and no agreement statistic.
3. Recruit a second judge before you start, not afterwards. A retrospective second opinion on cases already chosen measures nothing.
4. Decide how disagreements will be resolved in advance: unanimous only, either judge, or reconciliation. Deciding afterwards invites the choice that flatters the result.
5. Mask identifying details where you can. Removing names, units and known reputations is the cheapest available defence against halo bias.

Figure 10.1 The two judges must not see each other's selections until both are complete.

Kit

6. Give each judge the identical pool list in the same order, or in a deliberately randomised order if order effects are a concern.
7. Give each judge the written criteria and nothing else. Verbal briefing drifts between judges.
8. Use a simple in or out recording sheet with a reason column, so the basis of each decision is captured while it is fresh.
9. Keep the two sheets physically separate until both are complete.

Judge

10. Have each judge decide on every case in the pool, not just mark the ones they like. Kappa needs a decision on all N.
11. Record a one line reason for every selection. These reasons are what turn a judgement into a criterion applied.
12. Do not let either judge revise after seeing the other sheet. Revisions belong in a separate reconciliation stage, recorded separately.

Figure 10.2 A decision on every case, with a reason, is what makes the agreement statistic computable at all.

Compare

13. Compute agreement before reconciling. Once the judges discuss, the independent figure is gone and cannot be recovered.
14. List the singleton cases, those exactly one judge picked, and record the reason each was ultimately kept or dropped.
15. Compare the two selection rates before interpreting kappa, because a threshold difference and a case level disagreement look identical in kappa but need different fixes.

Figure 10.3 Singleton cases are where the judgement is visible, so each needs a recorded reason.

Figure 10.4 Three things that are easily confused; only the first is measured on this page.

Check

16. If kappa comes out below 0.40, rewrite the criteria rather than the sample. Low agreement means the criteria are not doing the work; a bigger sample will not help.
17. If kappa comes out above 0.85, verify the judges really were independent. Very high agreement on a subjective task usually means contamination.
18. File both original sheets, the agreement calculation and the reconciliation record. In judgment sampling the audit trail of who decided what, and when, is the whole evidence base.

Figure 10.5 The kappa value points to a specific next action, not just a label.

10b. Recording sheet column specification

Completeness rule: every case in the pool needs a row from every judge. A judge who records only their selections cannot be included in any agreement calculation. Independence rule: sheets stay separate until both are finished; the agreement figure is computed from the independent stage only, even if you reconcile afterwards.

10c. Filled worked recording sheet

What this sheet shows:

• Every case has a decision from both judges, which is what makes the agreement calculation possible.
• Rows 1, 5 and 9 are unanimous and form the defensible core of the sample.
• Rows 3, 4, 7 and 10 are singletons, and the paired reasons show the judges applied different criteria rather than simply disagreeing.
• Row 3 versus row 4 is instructive: A weights timing, B weights vendor approval. That is a criteria problem, not a competence problem.
• Recording both reasons side by side is what lets you diagnose the disagreement rather than merely counting it.

10d. Blank print ready recording sheet

Download the blank sheet as a .csv file, open it in Excel, Google Sheets or LibreOffice, then print one copy per judge. The Decision and Reason columns for every case in the pool are what make the agreement statistics computable, so they are pre-labelled here even though most templates only capture the selections.

11Which Sampling Method Should You Use?

Decision tree. Answer these in order and stop at the first yes.

1. Do you need to project a total or a rate to the whole population with a stated precision? Use a probability design; judgment sampling cannot support projection.
2. Are you auditing a monetary population and need both targeting and projection? Use monetary unit sampling, which weights by value while remaining a probability method.
3. Do you want to examine the cases most likely to contain the problem, and can you accept that findings do not project? Judgment sampling is the right tool; measure the judgement with this page.
4. Do you have two or more experts available? Use judgment sampling with independent parallel judging so reliability is measurable.
5. Are you doing qualitative research and need variation, saturation or theory building? Use purposive sampling and our purposive tool instead.
6. Do you need fixed counts within subgroups? Use quota sampling.
7. Are you selecting whatever is easiest to reach with no analytical rationale? That is convenience sampling; label it honestly.
8. Is the population hidden with no frame? Use snowball or respondent driven sampling.

12Troubleshooting and Common Errors

13Assumptions, Bias and Limitations

• No probability basis. Selection probabilities are unknown, so there is no margin of error, no confidence interval and no projection to the population.
• The judge is the instrument. The design inherits every limitation of the person applying it, including their training, assumptions and blind spots.
• Reliability is not validity. Two judges can agree perfectly and both be wrong in the same direction. Agreement measures consistency only.
• Kappa depends on prevalence and marginals. Rare selection depresses kappa and unequal strictness distorts it, which is why PABAK and the two indices are reported alongside.
• Independence assumption. Every agreement figure assumes the judges did not see each other's decisions. If they conferred, the statistic measures conformity.
• Complete pool assumption. Agreement is computed over the candidate pool. A judge who considered a different pool cannot be validly compared.
• The bias index is a self assessment. It carries no sampling theory and corrects nothing; it structures disclosure.
• Two judges is a minimum, not a comfort. Two experts from the same institution are far more likely to share a blind spot than two from different ones.

14Conclusion

What judgment sampling gives you

Judgment sampling concentrates limited effort on the cases most likely to matter. An auditor who examines the twenty most irregular entries learns more about control failure than one who examines twenty entries at random, and an inspector who visits the machines most likely to be failing finds more faults than one following a random schedule. When the aim is detection rather than estimation, deliberately targeting risk is not a weaker method, it is the correct one, and pretending otherwise wastes the expertise you are paying for.

What it costs you

You give up projection entirely. Nothing found in a judgment sample can be extrapolated to the population, because the cases were chosen precisely for being unusual. You also inherit the judge: their training, their recent experience, their institutional assumptions and their cognitive shortcuts all enter the sample invisibly. That is why the two questions worth asking are not about sample size at all, but whether another competent expert would have chosen similarly, and which known heuristics were in play.

What to check before you report

Confirm five things: the judge and their qualification are named, the criteria were written before any case was seen, the candidate pool size and selection rate are stated, agreement was computed from independent judging with kappa and PABAK reported together, and the cases only one judge selected are listed with a reason each. Then state plainly that agreement demonstrates reliability and not validity, because that distinction is the one most often blurred in practice.

What to do next

If your kappa fell below 0.40 the criteria are the problem, and no amount of extra sampling will fix them; rewrite them with worked examples and repeat. If it landed between 0.40 and 0.60, that is a normal and reportable result for expert selection, so publish it openly rather than quietly. If it came in above 0.85, verify that the judges really were independent before treating it as good news. And if you had only one judge, the highest value half day available to you is a colleague re-selecting from the same pool, because a kappa figure is the single most persuasive thing a judgment sample can offer.

15Test Yourself

16Frequently Asked Questions

17Cite This Tool

18Related Tools

• Purposive Sampling Tool for strategy choice, variation coverage and saturation, the qualitative side of the same family.
• Convenience Sampling Calculator and Bias Checker for scoring bias when there was no analytical rationale.
• Simple Random Sampling Calculator for a seeded probability draw when projection is required.
• Stratified Random Sampling Calculator for targeting subgroups while keeping projectability.
• Systematic Sampling Calculator for interval selection with a random start.

19Glossary of Terms

20References

1. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. https://doi.org/10.1177/001316446002000104
2. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. https://doi.org/10.1037/h0031619
3. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. https://doi.org/10.2307/2529310
4. Byrt, T., Bishop, J., & Carlin, J. B. (1993). Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46(5), 423-429. https://doi.org/10.1016/0895-4356(93)90018-V
5. Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543-549. https://doi.org/10.1016/0895-4356(90)90158-L
6. Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6), 551-558. https://doi.org/10.1016/0895-4356(90)90159-M
7. Sim, J., & Wright, C. C. (2005). The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Physical Therapy, 85(3), 257-268. https://doi.org/10.1093/ptj/85.3.257
8. Gwet, K. L. (2014). Handbook of Inter-Rater Reliability (4th ed.). Advanced Analytics. WorldCat record
9. Krippendorff, K. (2004). Reliability in content analysis: some common misconceptions and recommendations. Human Communication Research, 30(3), 411-433. https://doi.org/10.1111/j.1468-2958.2004.tb00738.x
10. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: heuristics and biases. Science, 185(4157), 1124-1131. https://doi.org/10.1126/science.185.4157.1124
11. Kahneman, D., Slovic, P., & Tversky, A. (1982). Judgment Under Uncertainty: Heuristics and Biases. Cambridge University Press. https://doi.org/10.1017/CBO9780511809477
12. Nickerson, R. S. (1998). Confirmation bias: a ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175-220. https://doi.org/10.1037/1089-2680.2.2.175
13. Kahneman, D., Sibony, O., & Sunstein, C. R. (2021). Noise: A Flaw in Human Judgment. Little, Brown Spark. WorldCat record
14. Meehl, P. E. (1954). Clinical versus Statistical Prediction. University of Minnesota Press. https://doi.org/10.1037/11281-000
15. Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 243(4899), 1668-1674. https://doi.org/10.1126/science.2648573
16. Elder, R. J., Akresh, A. D., Glover, S. M., Higgs, J. L., & Liljegren, J. (2013). Audit sampling research: a synthesis and implications for future research. Auditing: A Journal of Practice and Theory, 32(sp1), 99-129. https://doi.org/10.2308/ajpt-50394
17. Hall, T. W., Hunton, J. E., & Pierce, B. J. (2002). Sampling practices of auditors in public accounting, industry, and government. Accounting Horizons, 16(2), 125-136. https://doi.org/10.2308/acch.2002.16.2.125
18. Etikan, I., Musa, S. A., & Alkassim, R. S. (2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1-4. https://doi.org/10.11648/j.ajtas.20160501.11
19. Baker, R., et al. (2013). Summary report of the AAPOR task force on non probability sampling. Journal of Survey Statistics and Methodology, 1(2), 90-143. https://doi.org/10.1093/jssam/smt008
20. Lohr, S. L. (2021). Sampling: Design and Analysis (3rd ed.). CRC Press. https://doi.org/10.1201/9780429298899