Data Quality Control: Validity, Reliability and Pretesting
Data quality control comprises planned actions that prevent, detect and correct errors across instrument design, recruitment, collection, entry, analysis and storage. High-quality data are fit for the research purpose and conclusions.
Learning objectives
- Differentiate validity, reliability, accuracy and precision.
- Explain content, construct and criterion validity.
- Apply test–retest, inter-rater and internal-consistency reliability.
- Plan pretesting, training, calibration, supervision and data checks.
Core concepts
| Concept | Meaning | Example |
|---|---|---|
| Validity | Measures the intended concept | A depression scale accurately identifies depressive symptoms |
| Reliability | Produces consistent results under comparable conditions | Two trained observers classify the same finding similarly |
| Accuracy | Closeness to the true value | Calibrated scale reports true weight |
| Precision | Closeness of repeated values to each other | Repeated weights vary minimally |
Types of validity
- Face validity: instrument appears appropriate to informed reviewers or participants.
- Content validity: items adequately cover the full concept domain.
- Construct validity: scores behave as theory predicts.
- Criterion validity: results agree with a credible reference standard, concurrently or predictively.
- Internal validity: study conclusions are credible for the studied participants.
- External validity: findings may apply to relevant other populations or settings.
Types of reliability
Test–retest
Stability when the same instrument is repeated after an appropriate interval.
Inter-rater
Agreement between different observers applying the same criteria.
Internal consistency
Extent to which items intended to measure one construct relate coherently.
Parallel forms
Agreement between equivalent versions of an instrument.
Pretest versus pilot study
A pretest checks individual instrument wording, flow, comprehension, response options and administration time in a population similar to—but usually separate from—the final sample. A pilot tests the feasibility of the broader study process, including recruitment, logistics, collection, entry and preliminary analysis. Institutions sometimes use the terms differently, so define the intended procedure.
Pretesting workflow
- Select participants similar to the intended population.
- Obtain appropriate consent and administer under realistic conditions.
- Record misunderstandings, missing options, discomfort and duration.
- Debrief data collectors and participants.
- Review missingness, distributions and skip failures.
- Revise wording, order, translation, coding and procedures.
- Version-control the final approved instrument.
Quality assurance across the study
| Stage | Control measure |
|---|---|
| Before collection | SOPs, training, translation, calibration, pretest and role-play |
| During collection | Supervision, observation, daily review, repeat measurement and audit trails |
| Data entry | Validation rules, range checks, double entry or verification |
| Cleaning | Duplicates, missingness, logic checks, outlier review and documented corrections |
| Analysis | Reproducible syntax, assumption checks and independent review |
| Storage | Encryption, backups, access control and version management |
Scenario: reliable but invalid
A scale gives 62.0 kg at every repeat measurement but a calibration weight shows it reads 3 kg too high. It is precise and reliable but inaccurate and invalid for true weight. Calibration addresses systematic error; repeated measurements alone do not.
Error prevention
- Use operational definitions and validated instruments.
- Train and certify data collectors before fieldwork.
- Standardise interview wording and measurement technique.
- Blind assessors to exposure or group when feasible.
- Maintain equipment calibration and maintenance logs.
- Separate identifiers and use automatic electronic checks.
- Document rather than silently delete or change unusual values.
Common mistakes
- Claiming an instrument is valid merely because it was pretested.
- Using Cronbach’s alpha as proof of all forms of validity.
- Including pretest participants in the final sample without justification.
- Changing tools mid-study without version tracking and retraining.
- Correcting implausible data by guessing the intended value.
Review questions
- Differentiate validity, reliability, accuracy and precision.
- Compare a pretest with a pilot study.
- Design quality controls for a blood-pressure survey.
- Explain three reliability assessment methods.
References
- WHO. Health Research Methodology.
- UAHEB. Research Proposal and Report Writing Guidelines.
Educational content only; validated instruments and approved protocols should guide actual research.
