Defining Synthesis 11
A systematic literature review
This chapter discusses:
· the meaning of a systematic literature review and
Defining Synthesis 11
A systematic literature review
This chapter discusses:
· the meaning of a systematic literature review and how it differs from a dissertation or thesis literature review;
· the process of doing a systematic literature review;
· different approaches to the analysis and synthesis of data: meta-analysis and narrative synthesis
· a checklist of questions for the critical appraisal of a systematic literature review.
What is a systematic literature review?
A systematic review is a literature review usually carried out by a group of experienced professionals which addresses clearly stated aims and/or research questions. Traditionally, systematic reviews examined the effectiveness of a healthcare intervention but more recently they have looked at wider issues such as how and why an intervention works, its feasibility, appropriacy and cost effectiveness. The aim is to synthesise research findings from a large number of different studies on a particular intervention or issue which can then potentially be used to inform policy and practice in the field investigated. The methods are made explicit so as to be replicable and these include a clear search strategy, inclusion and exclusion criteria for publications and studies selected from the search, a synthesis of data from the sources included and a detailed coding and analysis of the findings from all the studies. There is also likely to be input in the process from stakeholders, for example policy makers, and users of a service (the Campbell Collaboration; EPPI-Centre, 2007; Rutter et al., 2010; Jesson et al., 2011).
Typically a systematic review is commissioned by a government body or public service to inform decision making. Initially, reviews were introduced in the field of health in the 1970s and 1980s; these later came under the auspices of the Cochrane Collaboration (http://www.cochrane.org/), whose first centre was officially opened in Oxford in 1993 to oversee and offer guidance to those conducting reviews investigating the effectiveness of healthcare interventions. The Cochrane Collaboration has since developed into an international network of researchers and healthcare professionals involved in evidence-based medical and health related decisions. Subsequently, different groups such as the Campbell Collaboration (http://www.campbellcollaboration.org/), EPPI (Evidence-informed Policy and Practice Centre) (http://www.eppi.ioe.ac.uk), and SCIE (Social Care Institute for Excellence) (http://www.scie.org.uk) have all extended the use of systematic reviews to other fields including criminal justice, social care and welfare, education, and employment. All of these organisations have libraries of systematic reviews which can be accessed online. Their web sites also offer guidance on the process of conducting a systematic review and the range of methods and approaches to analysis and synthesis that can be used.
The need for systematic reviews in an ever wider range of fields has arisen from recognition of the importance of evidence based practice to inform policy decisions and professional practice. Before choices can be made, an overview has to be gained of all the evidence available and a rigorously conducted systematic review can provide this, preventing a situation where decisions are made on the basis of a single, possibly poor quality, study. Furthermore, the huge increase in the number of publications available, as the Internet has become the predominant means of information sharing, makes it impossible for an individual to read everything that is being published. Thus, systematic reviews can provide a synthesis of evidence on a specific issue which can be a great time saver for busy practitioners.
This type of review is clearly different from a dissertation or thesis literature review, which has been the main focus of this book so far. Nevertheless, an understanding of the role and nature of systematic literature reviews is important even if not undertaking one yourself. It is often relevant to consult systematic reviews on particular issues as part of a dissertation or thesis literature review. To recap on the differences between the two types, a literature review which is part of a research project involving the collection of primary data, e.g. a masters research dissertation, provides a context and theoretical underpinning to the research, evaluates related empirical studies, and identifies how the current project extends or fills a gap in previously reported research in the field. In contrast, a systematic literature review is in itself a research study, addressing research questions and using the literature as data to be coded, analysed and synthesised to reach overall conclusions. Many of the techniques and tools described in earlier chapters (e.g. literature searching, reading and note taking, reference management, citation practices) are applicable for both sorts of review but the underlying purposes of each type are very different.
The process of conducting a systematic literature review
The web sites of the organisations cited above all include detailed advice on the process of conducting a systematic literature review. A concise summary is given below although it is recommended that the above web sites are visited if more specific detail is required. If conducting a stand-alone literature review for a degree programme assignment (as mentioned in Chapter 1), some of these steps may also offer a useful framework.
1. When a general area of enquiry has been identified, a scoping activity of the literature available is conducted by doing key word searches on all relevant catalogues and databases and systematically recording results.
2. A protocol or plan for the systematic review can then be developed to ensure that the study is transparent and rigorously carried out. It will need to be approved by a committee if the review is being conducted for a particular organisation (e.g. the Campbell Collaboration). The protocol and should cover the following:
· the proposed title, aim and research questions;
· the review team members;
· an overview of the background context including relevant legislation and current debates;
· literature searching and screening methods including details on how records will be kept;
· inclusion and exclusion criteria for the articles to be used as part of the review;
· data extraction strategies, i.e. what categories will be used for extracting data from the articles (aim, location, sample size, methods, findings etc);
· data synthesis approaches, e.g. meta-analysis (synthesis of numerical results) and/or coding and narrative synthesis of qualitative data.
· stakeholder involvement and input in the review; when and in what ways interested parties will have a role (e.g. when deciding on the aims of the review).
· timetable for the whole process.
An updated and revised version of the protocol can become the methods part of the systematic review itself when this is written.
1. After the protocol has been agreed, a comprehensive search of the literature is conducted and detailed records of all key word searches, databases used, dates, and numbers of hits are kept. As well as electronic sources, print sources and grey literature are explored (see Chapter 3 for more on conducting searches and keeping records).
2. Inclusion and exclusion criteria to all the retrieved articles and publications are then applied. It is common practice to apply these criteria at two levels, first, after reading the title and abstract, and second, after reading the article in its entirety. All decisions made along with the reasons should be documented on an Excel spreadsheet or table. Specified criteria for inclusion and exclusion in the review may relate to a number of different aspects of an article such as its date of publication, location of the research, detail and type of methodology, identity and size of sample, the type of intervention or the data analysis method. For example in a systematic review on the effect of vitamin C intake on the occurrence and duration of common colds, studies were excluded if the dose of vitamin C was less than 0.2 grams per day and if no placebo group was used for comparison. The studies included were not restricted to those based on randomised controlled trials (RCTs) (Hemilä et al. 2007). RCTs are experiments where participants are randomly allocated to a control group or treatment group as part of an investigation into the effects of an intervention; these trials are very influential in decision making about the most effective treatments in healthcare.
3. Studies may also be excluded for poor quality or insufficient detail or relevance to the review questions, so critical and evaluative reading is important. A checklist of questions can be devised to help with these decisions (see questions to guide critical reading in Chapters 4 and 8).
4. After the articles selected for inclusion have been identified, the data extraction takes place (see section on a tabular comparison of source texts in Chapter 4). All the chosen literature sources are read and consistent data is recorded about each one. This could be done as Word tables, Excel spreadsheets or in NVivo (see Chapters 4 and 5). Examples of the criteria that might be recorded about each article are as follows: full reference of the article, country of origin, setting of the study, objective of the research, study design, details about participants, type of intervention, details of control group (Torgerson et al. 2006).
5. Following the data extraction from each source, the synthesis of evidence takes place. This is when connections are made between all the articles read to address the research questions of the review. This may include a narrative synthesis of qualitative data or a meta-analysis of quantitative data. The latter would include a tabular presentation of data and/or charts incorporating statistics as a meta-analysis involves a synthesis of numerical results from comparable studies to produce an overall statistic. (See sections below and the further reading list at the end of the chapter for suggestions and links to more information about meta-analysis and narrative synthesis in systematic literature reviews.)
6. A mixed method approach to reviewing is also possible. An example is outlined by Harden and Thomas (2005), who describe the methods they adopted in a systematic review on the barriers and facilitators to healthy eating amongst young children. Their review included a quantitative meta-analysis of healthy eating intervention studies, qualitative coding using NVivo and a synthesis of studies exploring personal perspectives on the issue, and a final matrix where the two sets of findings were compared. This combination of techniques allows for the findings from different types of research study to be combined so as to address the research questions posed for a systematic review in a fully comprehensive manner.
7. Based on the synthesis of data, recommendations for practice are made and conclusions are drawn.
8. The quality of systematic reviews is supported through the peer review process, which takes place on the final draft of the review. This means that professional colleagues read and comment on the work and make a recommendation regarding its suitability for publication.
9. If the article is accepted for publication after peer review, the final step involves dissemination of the results and recommendations to stakeholders and professionals in the field through publication in journals and on web pages, and presentations at seminars and conferences.
10. Regular updates will be implemented of systematic reviews which are made available through the online libraries of organisations such as the Cochrane Collaboration. As part of this process, key word searches are conducted in appropriate databases for recent and relevant studies and any necessary updates to the data synthesis and recommendations for practice will be included in the review.
There are now a range of different approaches to data synthesis in systematic reviews and it is important that reviewers take into account the topic, purpose and question of the review, the type of primary research evidence available, and the methodological backgrounds and preferences of the reviewers when making decisions about the most appropriate choice for a particular review (Boaz et al. 2006). In the following two sections, the two most widely used alternatives are discussed, but as Boaz et al. (2006) demonstrate, drawing on five examples of reviews from a variety of fields, there are many ways of combining and/or varying these.
What is a meta-analysis?
The various organisations mentioned above focus on different types of review. For example, as already explained, the Cochrane Collaboration investigates healthcare and health policy interventions and assesses their benefits and drawbacks. They also conduct reviews of diagnostic tests for diseases. The reviews are usually quantitative and often based on randomised controlled trials (RCTs). The synthesis of data from such studies involves meta-analysis, which is a technique that combines the statistical results from a number of similarly conducted individual investigations to reach an overall result. The analysis assesses the ‘outcome measure’ or ‘effect’ of an intervention by combining statistics from comparable studies. It determines the direction and size of the effect and whether it is consistent or different across studies. This synthesis gives more strength to the assessment of the effectiveness of an intervention because it is a combined result from a number of different investigations and thus gives a more precise measure.
It is beyond the scope of this book, to consider the detailed statistical techniques that can take place in various types of meta-analysis. However, more details are available in the Cochrane Handbook for Systematic Reviews of Interventions (available from www.cochrane-handbook.org) and further reading is listed at the end of this chapter. Nevertheless, some general principles involved in the meta-analysis process are outlined briefly below.
Meta-analysis is most appropriate when the studies combined in the review have addressed the same or comparable questions, have had a similar design and intervention, have measured the same outcomes and have used comparable statistical techniques, i.e. the studies are homogenous. If any of these aspects of the studies are too dissimilar and diverse, meta-analysis may not be suitable, or it should be analysed using a ‘random-effects model’ in an appropriate statistical package (see below). Homogenous studies can be analysed using a ‘fixed effect model’ to calculate a ‘mean outcome measure’ from all the data using a statistical package such as RevMan (2008), the Cochrane Collaboration Review Manager. This measure can be assumed to be attributable to the intervention and be used to produce a ‘pooled effect estimate’ of the intervention (Higgins and Green 2011).
Different studies may have different weightings in the meta-analysis based on the quality of the individual trials carried out. Decisions on the weightings of individual studies involve evaluative judgments based on factors such as the number of participants or the precise nature of outcome measures. In some fields, there may be established scales for measuring the weighting that individual studies should have in a meta-analysis (e.g. Jadad et al. 1996); for reviews in other areas, however, these scales may need to be established on a review by review basis to take into account the particular range of studies included in a particular instance.
The internal validity of a meta-analysis is established through a ‘risk of bias’ assessment. This is required to show that the conclusions drawn from a study are trustworthy with regard to the intervention being studied and that the research question posed in the review protocol has indeed been answered. The Cochrane Handbook (Higgins and Green 2011) includes a table for assessing bias which includes the following categories: selection bias, performance bias, detection bias, attrition bias and reporting bias. Under these headings, the following questions should be addressed: To what extent is the allocation to a treatment group random and blind? To what extent are participants and researchers blind to the intervention that has been applied? To what extent are those measuring outcomes blind to which intervention has been applied? To what extent are attritions reported and accounted for? Have outcomes been reported selectively and if so, to what extent? Studies can be rated according to the responses to each of these questions. For example, in RCTs, there is deemed to be less likelihood of bias if allocation to treatment and control groups is random and blind.