Picture a bathroom scale that reads 150, 163 and 157 pounds across three weigh-ins in a row. Your weight has not changed. The scale simply measures badly.
The same question applies to every questionnaire, and researchers call it reliability. Reliability is the accuracy of a measurement: a questionnaire, a test or a scale counts as reliable when repeating it under the same conditions produces the same or very similar values, because the differences you observe come from real differences and not from measurement error. By the end of this article you will know which method fits your own study and which number you can report.
📌 The key points at a glance
- Reliability describes how accurate a measurement is, not whether it is meaningful.
- The reliability coefficient sits between 0 and 1.
- From 0.70 upward, reliability usually counts as adequate for group comparisons.
- Four methods estimate reliability, and student research almost always uses internal consistency.
- APA Style requires reliability coefficients for the scores you analyzed.
Create a survey for free
With empirio.ai you can create a modern online survey in minutes — free to start.
- AI-built survey
- Adjust by drag & drop
- Real-time analysis
Reliability means measurement accuracy: the definition in one sentence
Reliability is the dependability or accuracy of a measurement. It is high when the differences measured between people come from genuine differences and only to a small extent from chance and measurement error.
In quantitative research, reliability is one of the three main quality criteria alongside objectivity and validity. Only one question is being asked: would a repeat under the same conditions produce the same result? Whether the result is meaningful in substance is something reliability deliberately does not answer.
Reliable is the adjective that goes with the term. A digital scale is reliable because it shows the same body weight even on repeated measurement. If a research team estimates the weight of the people it observes by eye, the result shifts from one round to the next. A measurement like that is not reliable, however experienced the person doing the estimating.
The degree of reliability is expressed as a correlation between two measurements and is called the reliability coefficient. It sits between 0 and 1: at 1 the measurement is free of measurement error, at 0 the value comes entirely from measurement error. The coefficient is not a fixed property of the questionnaire, it holds only for that instrument in that group.
How reliability is tested: four methods and their limits
Reliability is tested by correlating two measurements of the same characteristic. The higher that correlation, the smaller the measurement error. Classical test theory uses four methods and they differ mainly in where the second measurement comes from. The Standards for Educational and Psychological Testing, published jointly by AERA, APA and NCME in 2014, devote a full chapter to reliability, precision and errors of measurement, and the 2014 edition is free to download.
Which method is open to you is rarely a statistical decision. What decides it is whether you can reach your respondents a second time and whether you asked enough questions about the same characteristic.
Test-retest reliability: the same measurement at two points in time
Test-retest reliability compares two measurements taken with the same instrument and the same people at two points in time. If both rounds produce similar values, the instrument counts as dependable. The catch lies in the interval: schedule the two rounds too close together and respondents remember their answers, leave too long a gap and the characteristic itself has changed.
Parallel-forms reliability: two equivalent versions of the same questionnaire
Parallel-forms reliability compares two versions of the same instrument that capture the same characteristic equally well. Test theory treats this as the ideal route because memory effects drop out. For a thesis it is almost never feasible, because the pool of items is rarely large enough.
Split-half reliability: dividing the questionnaire into two halves
Split-half reliability divides a single questionnaire into two halves and correlates the results of each half. No second round of data collection is needed, which is why the method works with a one-time survey. Because each half contains only half the items, the number comes out too low and is corrected with the Spearman-Brown formula.
Internal consistency: every item as its own part of the test
Internal consistency treats each individual question as a part of the test in its own right and examines how closely the items of a scale hang together. It is usually reported as Cronbach's alpha, and in student research it is the normal case because a single round of data collection is enough. It does require a scale made up of several questions about the same characteristic, for example several statements on a Likert scale.
Note
Cronbach's alpha needs several questions about the same characteristic. For a single question, internal consistency cannot be calculated at all, not even with a very large sample. Plan the scale with several statements while you are still designing the questionnaire.
Interrater reliability: when people rather than questionnaires do the measuring
Interrater reliability shows how far two or more people judging the same observation arrive at the same result. It is needed wherever a human being classifies rather than a questionnaire measuring: in observations, in open-ended answers and in qualitative content analysis. APA Style spells the term without a hyphen, so write interrater reliability in a paper that follows APA.
The clearest example sits in every classroom. Two graders score the same essay and award two different grades, which is exactly why rubrics and calibration sessions exist. As a statistic, agreement between two raters is reported as Cohen's kappa and agreement among more than two as Fleiss' kappa. Both strip out the agreement you would expect by chance, which is why they say more than a plain percentage of matching decisions.
The line to objectivity of scoring is blurred, because both criteria ask about the influence of the person doing the assessing. A simple division helps in practice: a clear coding manual secures objectivity, while measured agreement between two people evidences reliability.
What reliability value is good enough for a thesis
A reliability coefficient of 0.70 or above usually counts as adequate for studies at group level, 0.80 as good and above 0.90 as high. That grading comes from the GESIS Survey Guideline on reliability by Daniel Danner (2015), published by the Leibniz Institute for the Social Sciences.
| Coefficient | How to read it | What it supports |
|---|---|---|
| below 0.70 | too low | scale needs justifying or dropping |
| 0.70 to 0.79 | adequate | comparisons between groups |
| 0.80 to 0.89 | good | the normal case in student research |
| 0.90 and above | high | statements about individual people |
One distinction gets skipped in most study guides: 0.70 is enough as long as you are comparing groups, students against working adults, say. As soon as a value carries a statement about one individual, as in diagnostics or a selection test, the requirement is considerably higher. That is precisely why textbooks quote different cutoffs.
American readers have one obligation their European counterparts do not. The APA Journal Article Reporting Standards for quantitative research, JARS-Quant (2020), ask authors to estimate and report values of reliability coefficients for the scores analyzed, including interrater reliability for subjectively scored measures. Quoting the alpha printed in a scale's manual does not satisfy that, because the number has to come from your own sample.
Careful
A pilot run to check reliability is data collection, so it needs IRB approval before it starts, not afterward. And test-retest reliability cannot be reported at all if you only collected data once. Where no number emerges because the design did not allow one, that belongs in your limitations section together with the reason.
Create a survey for free
With empirio.ai you can create a modern online survey in minutes — free to start.
- AI-built survey
- Adjust by drag & drop
- Real-time analysis
Common mistakes with reliability
Three mistakes turn up especially often in student research and all three surface at the defense at the latest.
The cause is the same every time: reliability gets confused with something that looks like greater rigor but answers a different question. Once you recognize the three mix-ups, they are easy to spot in your own methods chapter.
Expecting more participants to raise reliability
A larger sample does not raise reliability. Reliability is a property of the measurement, meaning of the questionnaire and of the situation in which it is completed. More participants only make the estimate of the coefficient more stable. A loosely worded item stays loose, even with 500 people in the sample rather than 50.
Calculating Cronbach's alpha for a single question
Cronbach's alpha assumes several questions about the same characteristic, because it measures how they relate to one another. For a single question that relationship does not exist, so the value is undefined. Where a number appears anyway, several substantively different questions have usually been lumped into one scale by accident.
Confusing parallel forms with two different methods of data collection
The parallel-forms method calls for two equivalent versions of the same instrument, not two different methods of data collection. Comparing a survey with an observation tests something else and belongs to validity. This mix-up sits in a great many online glossaries and travels from there into student papers.
Conclusion
Reliability is the least glamorous of the three quality criteria and the one most student projects come unstuck on, because it is settled before data collection rather than afterward. Ask several questions about the same characteristic and you can report a number later. Ask only one, and no amount of analysis will retrieve it.
Reliability tells you how accurately you measured. It says nothing about whether you measured the right thing.
Where to go next
- Want all three quality criteria side by side? Objectivity, reliability and validity compared
- Need the definition of the next term? What does validity mean?
- Putting the questionnaire together right now? Creating a questionnaire: guide and structure
- Want the whole process in view? Empirical research: definition, methods, guide
Our reading tip: The Standards for Educational and Psychological Testing (AERA, APA and NCME, 2014) devote chapter 2 to reliability, precision and errors of measurement, and the edition is available at no cost. For the statistical mechanics of each estimation method, Danner, Daniel (2015): Reliabilität, die Genauigkeit einer Messung, GESIS Survey Guidelines, Mannheim, works through every formula.
Need several questions about the same characteristic so you can report a reliability coefficient later?
With empirio.ai, an online survey tool from Germany, you can set up scales with several statements in a few steps and export the raw data for analysis in SPSS.