Evaluation Tool for ‘Mixed Methods’ Study Designs.
The ‘mixed method’ evaluation tool was developed from the evaluation tools for ‘quantitative’ and ‘qualitative’ studies,i themselves created within the context of a project exploring the feasibility of undertaking systematic reviews of research literature on effectiveness and outcomes in social care. The ‘mixed method’ tool draws on appropriate questions from the quantitative and qualitative evaluation tools. It provides a template of key questions to assist in the critical appraisal of studies using more than one method.ii
Review Area Key Questions
(1) STUDY EVALUATIVE OVERVIEW
Bibliographic Details Author, title, source (publisher and place of publication), year
Purpose What are the aims of this paper? If the paper is part of a wider study, what are its aims?
Key Findings What are the key findings?
Evaluative Summary What are the strengths and weaknesses of the study and theory, policy and practice implications?
(2) STUDY AND CONTEXT (SETTING, SAMPLE AND OUTCOME MEASUREMENT)
The Study What type of study is this? What was the intervention? What was the comparison intervention? Is there sufficient detail given of the nature of the intervention and the comparison intervention? What is the relationship of the study to the area of the topic review?
Context: (1) Setting Within what geographical and care setting is the study carried out? What is the rationale for choosing this setting? Is the setting appropriate and/or sufficiently specific for examination of the research question? Is sufficient detail given about the setting? Over what time period is the study conducted?
Context II: Sample What was the source population? What were the inclusion criteria? What were the exclusion criteria? How was the sample (events, persons, times and settings) selected? (For example, theoretically informed, purposive, convenience, chosen to explore contrasts) Is the sample (informants, settings and events) appropriate to the aims of the study? If there was more than one group of subjects, how many groups were there, and how many people were in each group? Is the achieved sample size sufficient for the study aims and to warrant the conclusions drawn? What are the key characteristics of the sample (events, persons, times and settings)?
Context III: Outcome Measurement What outcome criteria were used in the study? Whose perspectives are addressed (professional, service, user, carer)? Is there sufficient breadth (e.g. contrast of two or more perspective) and depth (e.g. insight into a single perspective)?
Evaluative Tool for Mixed Method Studies
Evaluative Tool for Mixed Method Studies
Prof Andrew Long (2005), School of Healthcare, University of Leeds
Review Area Key Questions
(3) ETHICS
Ethics Was Ethical Committee approval obtained? Was informed consent obtained from participants of the study? How have ethical issues been adequately addressed?
(4) GROUP COMPARABILITY
Comparable Groups If there was more than one group was analysed, were the groups comparable before the intervention? In what respects were they comparable and in what were they not? How were important confounding variables controlled (e.g. matching, randomisation, or in the analysis stage)? Was this control adequate to justify the author’s conclusions? Were there other important confounding variables controlled for in the study design or analyses and what were they? Did the authors take these into account in their interpretation of the findings?
(5) QUALITATIVE DATA COLLECTION AND ANALYSIS
Data Collection Methods What data collection methods were used in the study? (Provide insight into: data collected, appropriateness and availability for independent analysis) Is the process of fieldwork adequately described? (For example, account of how the data were elicited; type and range of questions; interview guide; length and timing of observation work; note taking)
Data Analysis How were the data analysed? How adequate is the description of the data analysis? (For example, to allow reproduction; steps taken to guard against selectivity) Is adequate evidence provided to support the analysis? (For example, includes original / raw data extracts; evidence of iterative analysis; representative evidence presented; efforts to establish validity – searching for negative evidence, use of multiple sources, data triangulation); reliability / consistency (over researchers, time and settings; checking back with informants over interpretation) Are the findings interpreted within the context of other studies and theory?
Researcher’s Potential Bias What was the researcher’s role? (For example, interviewer, participant observer) Are the researcher’s own position, assumptions and possible biases outlined? (Indicate how these could affect the study, in particular, the analysis and interpretation of the data)
Evaluative Tool for Mixed Method Studies
Evaluative Tool for Mixed Method Studies
Prof Andrew Long (2005), School of Healthcare, University of Leeds
Review Area Key Questions
(6) POLICY AND PRACTICE IMPLICATIONS
Implications To what setting are the study findings generalisable? (For example, is the setting typical or representative of care settings and in what respects? If the setting is atypical, will this present a stronger or weaker test of the hypothesis?) To what population are the study’s findings generalisable? Is the conclusion justified given the conduct of the study (For example, sampling procedure; measures of outcome used and results achieved?) What are the implications for policy? What are the implications for service practice?
(7) OTHER COMMENTS
Other comments What was the total number of references used in the study? Are there any other noteworthy features of the study? List other study references
Reviewer Name of reviewer Review date
i Long AF, Godfrey M, Randall T, Brettle AJ and Grant MJ (2002) Developing Evidence Based Social Care Policy and Practice. Part 3: Feasibility of Undertaking Systematic Reviews in Social Care. Leeds: Nuffield Institute for Health.
ii This tool was developed while the lead author was at the Health Care Practice R&D Unit (HCPRDU) at the University of Salford. It has since been slightly modified.
