Picture a bathroom scale that shows 68, 74 and 71 kilograms in three weigh-ins one after another. Your weight has not changed. The scale simply measures badly.
The same question applies to every questionnaire, and researchers call it reliability. Reliability is the accuracy of a measurement: a questionnaire, a test or a scale counts as reliable when repeating it under the same conditions produces the same or very similar values, because the differences you observe come from real differences and not from measurement error. By the end of this article you will know which method suits your own study and which figure you can report.
📌 The key points at a glance
- Reliability describes how accurate a measurement is, not whether it is meaningful.
- The reliability coefficient sits between 0 and 1.
- From 0.70 upwards, reliability usually counts as adequate for group comparisons.
- Four methods estimate reliability, and dissertations almost always use internal consistency.
- Ofqual runs a whole programme on the reliability of exam marking.
Create a survey for free
With empirio.ai you can create a modern online survey in minutes — with hosting in the EU.
- AI-built survey
- Adjust by drag & drop
- Real-time analysis
Reliability means measurement accuracy: the definition in one sentence
Reliability is the dependability or accuracy of a measurement. It is high when the differences measured between people come from genuine differences and only to a small extent from chance and measurement error.
In quantitative research, reliability is one of the three main quality criteria alongside objectivity and validity. Only one question is being asked: would a repeat under the same conditions produce the same result? Whether the result is meaningful in substance is something reliability deliberately does not answer.
Reliable is the adjective that goes with the term. A digital scale is reliable because it shows the same body weight even on repeated measurement. If a research team estimates the weight of the people it observes by eye, the result shifts from one round to the next. A measurement like that is not reliable, however experienced the person doing the estimating.
One warning about the word itself, and it is a British peculiarity worth knowing. In UK official statistics, reliability has a different technical meaning: the Government Analysis Function guidance on quality statistics defines it as how close early estimates are to later revised values. Psychometrics means consistency of measurement instead. If your reading list mixes both fields, check which sense an author has in mind.
How reliability is tested: four methods and their limits
Reliability is tested by correlating two measurements of the same characteristic. The higher that correlation, the smaller the measurement error. Classical test theory uses four methods and they differ mainly in where the second measurement comes from.
Which method is open to you is rarely a statistical decision. What decides it is whether you can reach your respondents a second time and whether you asked enough questions about the same characteristic.
Test-retest reliability: the same measurement at two points in time
Test-retest reliability compares two measurements taken with the same instrument and the same people at two points in time. If both rounds produce similar values, the instrument counts as dependable. The catch lies in the interval: schedule the two rounds too close together and respondents remember their answers, leave too long a gap and the characteristic itself has changed.
Parallel-forms reliability: two equivalent versions of the same questionnaire
Parallel-forms reliability compares two versions of the same instrument that capture the same characteristic equally well. Test theory treats this as the ideal route because memory effects drop out. In a dissertation it is almost never feasible, because the pool of items is rarely large enough.
Split-half reliability: dividing the questionnaire into two halves
Split-half reliability divides a single questionnaire into two halves and correlates the results of each half. No second round of data collection is needed, which is why the method works with a one-off survey. Because each half contains only half the items, the figure comes out too low and is corrected with the Spearman-Brown formula.
Internal consistency: every item as its own part of the test
Internal consistency treats each individual question as a part of the test in its own right and examines how closely the items of a scale hang together. It is usually reported as Cronbach's alpha, and in dissertations it is the normal case because a single round of data collection is enough. It does require a scale made up of several questions about the same characteristic, for example several statements on a Likert scale.
Note
Cronbach's alpha needs several questions about the same characteristic. For a single question, internal consistency cannot be calculated at all, not even with a very large sample. Plan the scale with several statements while you are still designing the questionnaire.
Inter-rater reliability: when people rather than questionnaires do the measuring
Inter-rater reliability shows how far two or more people judging the same observation arrive at the same result. It is needed wherever a human being classifies rather than a questionnaire measuring: in observations, in open-ended answers and in qualitative content analysis.
British universities offer the clearest example, and most readers have been graded by it. Two markers assess the same dissertation and award two different marks, which is exactly why double marking, moderation and external examiners exist. Ofqual has published work on estimating inter-rater reliability, on the grounds that inconsistency between markers can be a major source of unreliability. As a statistic, agreement between two raters is reported as Cohen's kappa and agreement between more than two as Fleiss' kappa. Both strip out the agreement you would expect by chance, which is why they say more than a plain percentage.
The line to objectivity of scoring is blurred, because both criteria ask about the influence of the person doing the assessing. A simple division helps in practice: a clear coding manual secures objectivity, while measured agreement between two people evidences reliability.
What reliability value is good enough for a dissertation
A reliability coefficient of 0.70 or above usually counts as adequate for studies at group level, 0.80 as good and above 0.90 as high. That grading comes from the GESIS Survey Guideline on reliability by Daniel Danner (2015), published in German by the Leibniz Institute for the Social Sciences.
| Coefficient | How to read it | What it supports |
|---|---|---|
| below 0.70 | too low | scale needs justifying or dropping |
| 0.70 to 0.79 | adequate | comparisons between groups |
| 0.80 to 0.89 | good | the normal case in dissertations |
| 0.90 and above | high | statements about individual people |
One distinction gets skipped in most study guides: 0.70 is enough as long as you are comparing groups, students against people in work, say. As soon as a value carries a statement about one individual, as in diagnostics or a selection test, the requirement is considerably higher. That is precisely why textbooks quote different cut-offs.
These figures are conventions rather than hard boundaries, and British assessment research is candid about it. A review of marking reliability written by NFER for Ofqual notes that there is variability in how measurement uncertainty gets reported in the first place. With a one-off online survey you have two routes open: internal consistency via Cronbach's alpha and the split-half method.
Careful
Test-retest reliability cannot be reported if you only collected data once. Where no figure emerges because the study design did not allow one, that belongs in your methods discussion together with the reason. A limitation you name is worth more in a viva than a coefficient that evidences nothing!
Create a survey for free
With empirio.ai you can create a modern online survey in minutes — with hosting in the EU.
- AI-built survey
- Adjust by drag & drop
- Real-time analysis
Common mistakes with reliability
Three mistakes turn up especially often in dissertations and all three surface in the viva at the latest.
The cause is the same every time: reliability gets confused with something that looks like greater rigour but answers a different question. Once you recognise the three mix-ups, they are easy to spot in your own methods chapter.
Expecting more participants to raise reliability
A larger sample does not raise reliability. Reliability is a property of the measurement, meaning of the questionnaire and of the situation in which it is completed. More participants only make the estimate of the coefficient more stable. A loosely worded item stays loose, even with 500 people in the sample rather than 50.
Calculating Cronbach's alpha for a single question
Cronbach's alpha assumes several questions about the same characteristic, because it measures how they relate to one another. For a single question that relationship does not exist, so the value is undefined. Where a figure appears anyway, several substantively different questions have usually been lumped into one scale by accident.
Confusing parallel forms with two different methods of data collection
The parallel-forms method calls for two equivalent versions of the same instrument, not two different methods of data collection. Comparing a survey with an observation tests something else and belongs to validity. This mix-up sits in a great many online glossaries and travels from there into dissertations.
Conclusion
Reliability is the least glamorous of the three quality criteria and the one most dissertations come unstuck on, because it is settled before data collection rather than afterwards. Ask several questions about the same characteristic and you can report a figure later. Ask only one, and no amount of analysis will retrieve it.
Reliability tells you how accurately you measured. It says nothing about whether you measured the right thing.
Where to go next
- Want all three quality criteria side by side? Objectivity, reliability and validity compared
- Need the definition of the next term? What does validity mean?
- Putting the questionnaire together right now? Creating a questionnaire: guide and structure
- Want the whole process in view? Empirical research: definition, methods, guide
Our reading tip: For the UK context, Ofqual's Reliability Programme overview (2013) is unusually readable: it explains the consistency of examination results and the repeatability of candidates' results from one test to the next. For the statistical mechanics, Danner, Daniel (2015): Reliabilität, die Genauigkeit einer Messung, GESIS Survey Guidelines, Mannheim, sets out every estimation method with its formula.
Need several questions about the same characteristic so you can report a reliability figure later?
With empirio.ai, an online survey tool from Germany, you can set up scales with several statements in a few steps and export the raw data for analysis. Your data stays in the EU.