Computer Search Report Checklist (CSRC): Reproducibility in SR - Taxonomy

As a way to help focus on what is meant by the reproducibility of computer searches in systematic reviews (SR), this section briefly lists and describes what it means for different SR components to be reproduced. A goal is to suggest a way of understanding the reproducibility of computer searches by contrasting the reproducing of computer searches with the reproducing of other SR components.

The list below briefly describes 14 SR components that might be reproduced. Each numbered item first names a component of the SR process. The component names are followed by brief descriptions. The first component points to the whole literature search, recognizing that guidelines speak of reproducing the literature search or the search strategy.  Next, more targeted components are listed that focus on the computer search. Components 3-6 focus specifically on computer search activities as a part of the literature search. Additional components focus on what is done with the computer search results.

1. Reproduced literature searches. Of course the literature search is a broad component of SR. Literature searches use a range of resources and strategies to identify sources that contain substantive information to be synthesized in SR.  To date common search strategies include using commercial computer databases, Google Scholar or other Internet search engines.  Also, there can be the evaluating of references found in key articles, hand searching of journals, contacting experts, and searching targeted web sites and repositories. Reproducing the literature search involves the same researcher, or a second, using actions and resources that mirror the steps and use of resources that were used for the original literature search. 
2. Reproduced literature search results.  This can be viewed as seeing the same information sources to consider for inclusion after doing 1. For example, having the same books or articles to consider; or having the same bibliographic information to consider that points to resources that might be used.
3. Reproduced computer searches. Here we are referring to the use of resources like PubMed, PsycINFO, EMBASE, Google Scholar, etc. Reproducing the computer search involves using resources and steps that mirror those used for the original computer search.
4. Reproduced computer search results. This involves ending up with and seeing the same hits to consider for inclusion after completing 3.  In computer searches these “hits” typically are groupings of information (e.g., in Google Scholar) or records with bibliographic information (e.g., in PubMed).  This information points to or indicates how to access documents that have the fuller narratives and/or data to be used for the SR.
5. Reproduced de-duplication process.  This involves using the same resources and/or steps to eliminate duplicate hits.  This is typically done after combining search results from computer searches completed on different computer resources.
6. Reproduced de-duplication results.  This is having the same hits to consider after step 5.
7. Reproduced first level inclusion/exclusion (IE) decision process (in computer searching). This involves what is done with the hits noted in 6. Compared to the original, a reproduced IE process involves using an identical set of evaluation criteria and selection steps to identify hits to retrieve in full text for further consideration. This process commonly uses “hit” information such as titles and abstracts (or similar information with Google Scholar search results, etc.).
8. Reproduced first level IE decision results.  Compared to the original, this is having the same list of hits chosen (after the IE actions taken in 7) that point to full text documents to retrieve and consider for use.
9. Reproduced full text source selection process. After completing 7., this is a second level of IE decision to determine what is done with the hits (those retrieved in full text). It is a selecting of which full text sources to retain for further evaluation.  Compared to the original, this second level IE process uses an identical set of evaluation criteria and selection steps to identify which sources to retain in full text.
10. Reproduced full text source selection results. Compared to the original, this is having the same sources to use after 9.
11. Reproduced data extraction process. This pertains to the steps (and tools) used to select specific information for use from the full text sources selected in 10. It involves using an identical extraction processes to identify and record the information to use for analysis, synthesis, etc.
12. Reproduced data extraction results. This is having the same content for use after completing 11.
13. Reproduced data analysis/synthesis process. This is what is done with the content identified in 12. It involves using identical analytical steps (or steps for synthesis) to arrive at results.
14. Reproduced data synthesis results.  The end analysis or synthesis product mirrors what was presented in the original SR.

List created in Spring of 2014