Guidelines for Quantitative Research Critique
Design
1. Are any of the following threats to internal validity (alternative explanations for the findings in the study) present?
• History: unrelated events occurring at the same time as the study
• Subject maturation: changes in the subjects during the study period
• Testing: effects of repeated measurements or the test itself on the dependent variable
• Instrumentation: changes in the measurement instruments or evaluators between data collection points
• Statistical regression: extreme measurement scores in one group that become less extreme
• Selection bias: baseline differences in the treatment/control groups
• Mortality: selective loss of subjects from treatment/control groups or the study
• Diffusion or imitation of the treatment: evidence of contract between groups
• Compensatory equalization with treatment of control subjects: evidence that control subjects received some of the treatment
• Resentful demoralization of control subjects: evidence that control subjects performed less well that they normally would
• Characteristics of the person collecting the data: evidence to suggest that characteristics of data collectors could have influenced the outcomes
Sample
1. What sampling method was used?
2. What is the sample size? Are response and attrition rates discussed?
3. Are you concerned about the adequacy of the sample size? Why or why not?
4. Is there evidence that the investigator(s) did a power analysis in order to avoid a Type II error?
5. How were subjects assigned to treatment groups (randomization)?
6. Is the sample culturally diverse?
7. Are there threats to external validity (generalizability)? What are they? If there are potential threats, are they acknowledged?
8. Is there mention of the protection of the rights of subjects? (IRB approval?)
9. Does any of the above affect the credibility of the study’s findings?
