Clinical Decision Making

Clinical Decision Making

CIN: Computers, Informatics, Nursing & Vol. 31, No. 10, 477–495 & Copyright B 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins C O N T I N U I N G E D U C A T I O N 2.4 ANCC Contact Hours Features of Computerized Clinical Decision Support Systems Supportive of Nursing Practice A Literature Review SEONAH LEE, PhD, MSN, RN According to the Institute of Medicine,1 the development and implementation of more sophisticated information systems are essential not only to enhance quality and efficiency of patient care but also to support clinical decision making. Clinical decision support becomes more and more a core function of health information systems to eliminate preventable medical errors,2 and the investments in decision support technologies targeted at nursing practice have increased.3 A computerized clinical decision support system (CDSS) refers to any electronic system designed to aid directly in clinical decision making. To generate patientspecific recommendations, CDSSs use the characteristics of individual patients; these recommendations are then presented to nurses for consideration.4,5 The knowledge base embedded in CDSSs contains the rules and logic statements that encapsulate knowledge required for clinical decisions so that it generates tailored recommendations for individual patients.6 With this, CDSSs assist nurses in completing the knowledge base rule–driven decision making or standardized rule-driven decision making,7 instead of using their own biases and intuition.8–10 On the one hand, CDSSs applied to nursing care are an expansion of the CDSS prototype defined above. For example, CDSSs for nursing care provide prebuilt forms for data entry of patient assessment, care plans, or outcome evaluation on given nursing interventions.8 Although it is not the case of recommendations

ORDER A PALGIARISM FREE PAPER NOW

automatically generated by the algorithm, the predesigned forms help decision making for nurses because these present the full scope of components that should be included for This study aimed to organize the system features of decision support technologies targeted at nursing practice into assessment, problem identification, care plans, implementation, and outcome evaluation. It also aimed to identify the range of the five stage-related sequential decision supports that computerized clinical decision support systems provided. MEDLINE, CINAHL, and EMBASE were searched. A total of 27 studies were reviewed. The system features collected represented the characteristics of each category from patient assessment to outcome evaluation. Several features were common across the reviewed systems. For the sequential decision support, all of the reviewed systems provided decision support in sequence for patient assessment and care plans. Fewer than half of the systems included problem identification. There were only three systems operating in an implementation stage and four systems in outcome evaluation. Consequently, the key steps for sequential decision support functions were initial patient assessment, problem identification, care plan, and outcome evaluation. Providing decision support in such a full scope will effectively help nurses’ clinical decision making. By organizing the system features, a comprehensive picture of nursing practice– oriented computerized decision support systems was obtained; however, the development of a guideline for better systems should go beyond the scope of a literature review. KEY WORDS Computerized clinical decision support systems & Features & Nursing care & Sequential decision support related nursing care activities. Thus, CDSSs for nursing care in this study include all the CDSS prototypes and the expanded versions. Author Affiliation: College of Nursing, University of Missouri-St. Louis, Missouri. The author has disclosed that she has no significant relationships with, or financial interest in, any commercial companies pertaining to this article. Corresponding author: Seonah Lee, PhD, MSN, RN, College of Nursing, University of Missouri-St. Louis, One University Boulevard, St. Louis, MO 63121 (ah7909@hotmail.com). DOI: 10.1097/01.NCN.0000432127.99644.25 CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. 477 Because using CDSSs to support nurses’ decision making is widespread, it is worth capturing which features of CDSSs were empirically effective for optimum decision support for frontline nurses. Currently, there are studies on CDSSs used to improve the clinical practice of nurses; however, system features addressing particular nursing care activities have been dispersed in individual reports. Nursing does not have the well-organized knowledge base on the features of nursing practice–oriented CDSSs in real practice settings. The purpose of this study was to organize the features of CDSSs useful for nursing practice through a literature review, especially using the categories of assessment, problem identification (ie, diagnosis), care plans, implementation, and outcome evaluation. The current decision support technologies typically operate in these five stages. A certain CDSS helps decision making in a single stage, while other CDSSs help decision making in two or more stages. However, because of a lack of empirical investigations, it has not been clear whether a CDSS providing decision support in all the stages from assessment to outcome evaluation was more clinically useful than a CDSS operating, for example, in only a single stage of assessment. If there are evidential data to answer this question, the evidence should be included as an important feature for better decision support. As a preliminary to conducting an empirical study to address the question above, the first priority was in conducting a literature review to identify to the extent of sequential decision support provided by CDSSs in the stages from assessment to outcome evaluation. In this study, the sequential decision support, which is another important concept, is one of the CDSS features. METHODS Studies Eligible for Review To obtain the most relevant studies, studies eligible for inclusion were primary studies on CDSSs used for nursing practice and designed to contain at least two aspects of assessment, problem identification, care plans, implementation, and outcome evaluation. Studies published in peerreviewed journals and in English were included. On the other hand, studies were excluded if they were studies on a nonelectronic decision support system such as a paperbased system, studies not providing a description on a CDSS, and studies providing only a technical description of a CDSS application (ie, testing algorithms of an application). Review studies on CDSSs were also excluded. Data Sources Databases of MEDLINE, CINAHL, and EMBASE were searched up to 2012 by using the search terms computer478 assisted decision support system, automated decision support, computerized evidence-based decision making, computerized evidence-based practice, and evidence, decision support system, having nursing in common. Conference proceedings and the reference lists of all included articles were reviewed to identify additional primary studies. Study Selection The author reviewed titles and abstracts of identified references and rated each article as ‘‘potentially relevant’’ or ‘‘not relevant’’ by using the inclusion and exclusion criteria. The author reviewed the full texts of potentially relevant primary studies and again rated each article as ‘‘potentially relevant’’ or ‘‘not relevant’’ using a screening checklist. Thus, the final selection of studies for review was made. A screening checklist was to check the presence or absence of and appropriateness of data that should be extracted from studies. Its content is identical to a data extraction form for double-checking (see ‘‘Data Extraction’’ section). Use of the checklist prevented important data from inadvertently being omitted. Before actual use of the checklist, the author piloted it on a sample of three articles to address the issues of arranging the checklist items in user-friendly sequence and completing the checklist.11 Data Extraction The author extracted necessary information from each of the finally selected articles by using a data extraction form. The form was to record study purpose, study design, data collection methods, study settings and participants, nursing care areas addressed by the use of a CDSS, functions of a CDSS, study results, and features of a CDSS. The functions of a CDSS were categorized into assessment, problem identification, care plans, implementation, and outcome evaluation. A CDSS was considered having the functions of the stages from assessment to outcome evaluation: when a CDSS had preformulated forms for data entry that are embedding evidence to support clinical decision making relating from assessment to outcome evaluation, when the rule engine of a CDSS automatically generated recommendations or instructions for a next action based on data entered in a prior step, or when the sections from assessment to outcome evaluation were automatically linked to each other for a logical continuity of clinical decision making and then relevant data have to be entered in a prebuilt form or selected from a prebuilt list. For example, if an assessment entry form existed, the CDSS had the function for patient assessment. If care plans were automatically generated based on assessment data entered, the CDSS had the functions of assessment and care plans. When a set of care plans was linked to patient outcome evaluation and then an outcome measurement CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. form should be filled out, the CDSS had the functions of care plans and outcome evaluation. Study results are any changes by the use of a CDSS. These would include improvement or nonimprovement in terms of, but not limited to, nurses’ decision making, nurse performance, and patient outcomes. As the features of CDSSs, components of CDSSs that improved nurses’ decision making, nurse performance, or patient outcomes were extracted. If some components deteriorated them (eg, ‘‘the need to devise care plans made nurses spend much time’’), the author treated the logically opposite component as a potential improvement component (eg, ‘‘removing the need to devise care plans made nurses save time’’).12,13 In addition, if authors of studies mentioned important features of their CDSS, the features were also included here. The functions of CDSSs mentioned above were integrated as part of the features of CDSSs. The author recorded extracted information on the data extraction form and also double-checked extracted information with original articles for accuracy. Data Analysis The extracted data, including study purpose, design, data collection methods, settings and participants, nursing care areas addressed by the use of a CDSS, functions of a CDSS, and study results, were organized in tables. To synthesize CDSS features across the reviewed studies, the author carefully read and compared the features extracted from each study and divided them into meaning units. The meaning units were assessment, problem identification, care plans, implementation, and outcome evaluation. The author integrated or separately organized the features into key words and phrases capturing core content of each unit. The synthesized results were organized in a separate table. RESULTS Of 681 potentially relevant studies published from 1990 to 2012, 27 studies met the eligibility criteria and the items on the screening checklist. The study description in Table 1 combines study purpose, design, data collection methods, settings, and participants. Table 2 presents a summary of Table 1, which includes study purpose, design, data collection methods, CDSS-applied nursing care areas, and sequential decision support functions of CDSSs. Of the 27 studies reviewed, 17 were system development, and eight of the 17 studies piloted their system immediately after system development (Table 2). In the study purpose of Table 2, others included two studies examining barriers to use of computerized advice6,26 and a study evaluating completeness of nursing documentation.19 The designs of 20 studies that conducted system evaluation or pilot test, except for seven studies of system development only, varied (Table 2). When considering the presence of a CDSS as the given intervention, 15 studies, which were mostly pilot tests, were posttest studies without a control group. Two pretest-posttest studies used different groups for comparison before and after system use. Four studies used a one-group pretest-posttest format. Also included were a quasi-experimental study with two nonrandomized control groups and a randomized controlled trial. Three studies used two different designs for their system evaluation or pilot test7,25,34; thus, they were counted twice in the design. Data collection methods used in the 20 studies for system evaluation or pilot test were individual interviews, focus group interviews, observations, chart review, analysis of screen usage, questionnaires for nurses and other healthcare providers, and questionnaires for patients. Eight studies collected data by mixed methods; three studies, by quantitative methods; and nine studies, by qualitative methods. Nursing care areas addressed by the use of a CDSS varied; however, fall, pressure ulcer, pain, blood glucose control, and patient referral overlapped, as shown in Tables 1 and 2. Eighteen studies targeted a single area of nursing care, while nine studies covered multiple areas of nursing care. Two mobile-based decision support systems targeted multiple areas of nursing care (Table 2). Table 1 presents the functions of CDSSs that provided decision support in the stages available from assessment to outcome evaluation. The reviewed CDSSs showed the diverse ranges of sequential decision support functions. Sequential decision support for patient assessment and care plans existed in all of the reviewed CDSSs (Table 2). With reference to the sequence, movement to a next stage such as from assessment to problem identification or to care plans occurred as a next screen automatically showed up or was clicked after completion of a prior stage; a nurse was forced to implement the movement. Two studies’ assessment entry forms were to assess patients’ responses to treatments (ie, patient outcomes),34,35 instead of initial assessment for patients (Table 1). Most CDSSs started their function for patient assessment with a nurse’s entry in an electronic assessment form (Table 2). Five CDSSs started their function as they automatically retrieved necessary data from hospital databases or other connected information systems and a nurse inputs additional information. Three CDSSs were a real-time system for patient assessment,23,29,37 and two of them were tele-advice systems.29,37 Two CDSSs automatically assessed patients without input of a nurse (Table 2).23,29 For the details of CDSS functions from problem identification to outcome evaluation, see Table 1. Table 1 presents the study results on patient outcomes, nurse performance, and nurses’ decision making by the use of CDSSs. The CDSSs were of benefit to patients and nurses as they improved patient status in the CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. 479 480 CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. Bakken et al (2007)16 Fall-injury management Browne et al (2004)15 Single nursing care area Delirium care Fick et al (2011)14 Study System development at the Columbia University Medical Center campus of Presbyterian Hospital in New York System development System evaluation by chart review after system use in all units at the Methodist Healthcare System of San Antonio in Texas Pilot study for feasibility (1) using questionnaires for 15 patients and their caregivers in a medical-surgical unit and (2) by analysis of screen uses and 34 nurses’ feedback at an acute care hospital in the central Pennsylvania region Study Description Characteristics of the 27 Studies Reviewed—Part 1 T a b l e 1 A fall-injury risk management system within the hospital-wide information system A computerized documentation system for fall risk stand-alone) A decision support system for delirium superimposed on dementia within the electronic medical record (EMR) CDSS C: Problem-specific care plans are generated by the system. Fall risk information is integrated into an interdisciplinary communication network including report sheets, care conferences, and audits until solved. A: Fall-injury risk is assessed by the system with a nurse’s input. The system rates a risk score. C: Institution-specific standard care plans are preselected and a nurse selects care plans from a drop-down box, based on a risk score. A: Fall risk is assessed by the system with a nurse’s input. The system rates a risk score. P: Fall risk category–specific problems are generated by the system. P: Presence of delirium is triggered by the system.a C: Individualized nonpharmacological care plans for the management and prevention of delirium are generated by the system. A: Delirium is assessed by the system with a nurse’s input and delirium-associated data automatically pulled from other electronic records. CDSS Functions of Assessment (A), Problem Identification (P), Care Plans (C), Implementation (I), and Outcome Evaluation (E) (continues) Fall and injury rates decreased but were not statistically significant at 6 mo after system use. 93% of patients improved or stayed the same on their mental health scores from admission to discharge. Overall, nurses did not have problems in using the system. Study Results on Patient Outcomes and Nurses’ Performance and Decision Making CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. 481 Gunningberg et al (2009)19 Clarke et al (2005)18 Pressure ulcer management Quaglini et al (2000)17 Study System development Pilot study for feasibility using questionnaires and qualitative data for nurses, mentors, experts in seven healthcare organizations (acute, home, intermediate, and extended care) in a Canadian urban health region A study examining the quality and comprehensiveness of nursing documentation by chart review before and after system use in a surgical, medical, and geriatric unit at the Swedish University Hospital System development Pilot study of feasibility by chart review for 40 patients in a general medicine ward Study Description CDSS A nursing documentation system for pressure ulcer within the electronic health record A decision support system for pressure ulcer prevention and treatment (stand-alone) A system for pressure ulcer prevention and treatment within the electronic patient record (EPR) Characteristics of the 27 Studies Reviewed—Part 1, Continued T a b l e 1 A: Pressure ulcer is assessed by the system with a nurse’s input. The system rates a risk score. C: Standard care plans are generated by the system. In addition, nurses were required to record nursing diagnosis, implementation of care plans, and evaluation of care. It was not part of the system. A: Pressure ulcer risk is assessed by the system with a nurse’s input and data automatically retrieved from the EPR. New inputs are required by pre-set time intervals. C: Care plans are generated by the system. Care plans can be overruled by entering a justification. I: Completion and noncompletion of care activities are entered at the end of a shift. Tasks not completed automatically go over to the next shift. E: Ulcer development is evaluated during every shift. New care plans are generated by the system. Every shift starts with new care plans. A: Pressure ulcer is assessed by the system with a nurse’s input. C: Care plans are generated by the system. Nurses can revise the care plans. CDSS Functions of Assessment (A), Problem Identification (P), Care Plans (C), Implementation (I), and Outcome Evaluation (E) (continues) The system increased knowledge about pressure ulcer prevention, treatment strategies, and resources required. Barriers were lack of administrative leadership, competencies on learning computer skills, implementing new guidelines, and technological deficiencies. There were significant improvements in quality and comprehensiveness of recording pressure ulcer after system use, although more improvement about recording was required. Improved care plans, detailed documentation, and facilitating handing on noncompletion to a next shift were useful. More flexibility on risk assessment and setting action timings and minimizing data entrys time were required. Study Results on Patient Outcomes and Nurses’ Performance and Decision Making 482 CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. 20 Body temperature monitoring Kroth et al (2006)23 Huang et al (2003)22 Pain management Im and Chee (2003)21 Fossum et al (2011) Study System evaluation by a randomized controlled trial examining the effect of system use of bedside nursing staff in a medical-surgical unit at the Wishard Memorial Hospital in Indiana System development Pilot study for feasibility by two test-retest studies using questionnaires for 24 patients with bone metastasis-related pain and using a focus group of four physicians System development (19 nursing faculty members in oncology from 10 countries participated in e-mail discussions and an online survey to identify culturally sensitive pain descriptions) System evaluation by a pretest-posttest study with nonequivalent control groups using questionnaires for 491 patients in 46 units at 15 nursing homes in four counties from rural areas in Norway Study Description CDSS A bedside system for temperature monitoring A decision support system for pain management (stand-alone) A decision support system for cancer pain management (stand-alone) A decision support system for pressure ulcer and malnutrition prevention within the electronic health record Characteristics of the 27 Studies Reviewed—Part 1, Continued Ta b l e 1 P: When a low temperature is identified, a warning pop-up window is generated by the system. C: Instruction to remeasure body temperature is generated by the system. The instruction can be overruled by selecting an ignoring reason from a menu or by typing a free text answer. A: Vital data are continuously measured and displayed by the bedside system. C: Specific pain treatment strategies following the WHO recommendation are generated by the system. A: Pain is assessed by the system with a patient’s input. The system generates a single-page summary on pain assessment. C: Pain management notes are generated by the system. A: Pain is assessed by the system with a nurse’s input. The system computes the assessment result. A: Pressure ulcer and nutrition are assessed by the system with a nurse’s input. C: Patient-specific care plans are generated by the system. CDSS Functions of Assessment (A), Problem Identification (P), Care Plans (C), Implementation (I), and Outcome Evaluation (E) (continues) The system was effective for nurses in improving the accuracy of temperature collection at the bedside. The system was feasible and acceptable for patients and healthcare providers. The proportion of malnourished patients decreased in the intervention group using the system. Risk and prevalence of both pressure ulcer and malnutrition showed no difference between groups. Study Results on Patient Outcomes and Nurses’ Performance and Decision Making CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. 483 Vogelzang et al (2005)25 Blood glucose control Henry et al (1998)24 Study System development System evaluation (1) by chart review after system use and (2) a pretest-posttest study using questionnaires for nurses in a 12-bed surgical intensive care unit at a tertiary teaching hospital System development Study Description CDSS A decision support system for insulin therapy linked to the central databases A nursing documentation system for an initial visit of diabetes mellitus within the electronic health record. Characteristics of the 27 Studies Reviewed—Part 1, Continued T a b l e 1 A: The assessment screens for initial visit of diabetes mellitus are completed by a nurse’s input. A nurse can make some changes in the assessment template. C: Care plans containing check boxes, blanks, and free-text entries within a structured text template are generated by the system. The system provides hyperlinks to resources for care plans. A: Blood glucose is assessed by the system, with relevant data automatically retrieved from the central laboratory database and a nurse’s input. C: A new insulin infusion rate and a next blood sampling time are generated by the system and stored in the hospital database. The recommendations can be overruled at any time. The main screen of the system shows the overview of glucose controls by applying different colors to the beds in the intensive care unit. Each bed on the screen is clickable to yield a more detailed information panel. CDSS Functions of Assessment (A), Problem Identification (P), Care Plans (C), Implementation (I), and Outcome Evaluation (E) (continues) The system provided safe and efficient blood glucose control. Nurses’ acceptance was high. Study Results on Patient Outcomes and Nurses’ Performance and Decision Making 484 CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. 6 Blood potassium control Hoekstra et al (2010)7 Campion et al (2011)26 Sward et al (2008) Study System evaluation (1) by chart review before and after system use in a 12-bed surgical and a 14-bed cardiothoracic intensive care unit and (2) using questionnaires for 76 nurses in intensive care units after system use at a tertiary academic center A study examining barriers and facilitators to using computerized advice by observations and unstructured interviews for nurses in a 21-bed surgical and a 31-bed trauma intensive care unit at Vanderbilt University A study examining reasons of declining computerized advice (1) by analysis of nursing records and (2) using questionnaires for 14 nurses in an adult intensive care unit at a tertiary care hospital Study Description CDSS A decision support system for potassium regulation linked to the central databases A decision support system for insulin therapy linked to other hospital information systems A decision support system for insulin therapy (stand-alone) Characteristics of the 27 Studies Reviewed—Part 1, Continued T a b l e 1 P: The value is categorized to hypokalemia, normokalemia, or hyperkalemia, and if abnormal, it is triggered by the system. A: Blood potassium is assessed by the system with a nurse’s input and relevant data automatically retrieved from the central laboratory database. C: An insulin order including dose, rate, and duration, and next glucose test time is generated by the system. Rationale for insulin recommendations is viewed on the same screen. The recommendation can be overruled. P: Hypoglycemia or hyperglycemia is triggered by the system. A: Blood glucose is assessed by the system with a nurse’s input. C: A new insulin infusion rate is generated by the system. A nurse can refuse the recommendation by typing his/her reason for declining the recommendation or choosing reasons for decline from a drop-down list. A: Blood glucose is assessed by the system with a nurse’s input. CDSS Functions of Assessment (A), Problem Identification (P), Care Plans (C), Implementation (I), and Outcome Evaluation (E) (continues) The system reduced the prevalence of hypokalemia and hyperkalemia. Nurses indicated improvement in potassium control by use of the system and a full compliance rate beyond 5 wk. Facilitators were trust in the system, nurse resilience, and paper serving as an intermediary between patient bedside and the system. Barriers were workload tradeoff between system use and direct patient care, inadequate user interfaces, and potential errors in operating medical devices. The recommendations were refused by related patient data, physician orders, nurses’ disagreement, nurse workload, medication errors, patient or family requests, and software problems. Study Results on Patient Outcomes and Nurses’ Performance and Decision Making CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. 485 Guite et al (2006)28 Referral automation Heermann and Thompson (1997)27 Study System development Pilot study for feasibility by compliance audit, training scenario, and staff meetings and using a questionnaire for nurses in a surgical and a medical-surgical unit at the Christiana Care System in Delaware Pilot study for feasibility by chart review for 19 transported neonates System development Study Description CDSS A decision support system for automation of admission referral process within the electronic health record A decision support system for stabilization of neonates before transport (stand-alone) Characteristics of the 27 Studies Reviewed—Part 1, Continued T a b l e 1 A: Neonate stability is assessed by the system with a nurse’s input. C: Instructions to stabilize and prepare a neonate’s condition before transport are generated by the system. A: Admission information is assessed by the system with a nurse’s input using a wireless device at the patient’s bedside and previous data automatically pulled from the hospital database. C: A list of all referrals with detailed information for each referral is generated by the system. The system sends electronic referrals to appropriate departments. I: Completion of the task is recorded by the referral departments. The original nurse identifies the completed task on screen. C: A potassium administration rate and a next blood sampling time are generated by the system. In case of extremely abnormal potassium values, the system prompts notification of the attending clinician. This can be overruled by nurses and physicians, and such instances are automatically recorded. CDSS Functions of Assessment (A), Problem Identification (P), Care Plans (C), Implementation (I), and Outcome Evaluation (E) (continues) The system simplified nursing work and led to more appropriate referrals to ancillary departments. Automatic retrieval of patient information by the system eliminated duplicate documentation. The system was safe and effective for neonate transport. Study Results on Patient Outcomes and Nurses’ Performance and Decision Making 486 CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. Multiple nursing care areas Jirapaet (2001)30 Tele-advise system Adams et al (2003)29 Study Pilot study for feasibility by a one-group pretest-posttest study using questionnaires on case simulations given to 16 nurses in a neonatal intensive care unit at a tertiary care hospital System development System development Study Description CDSS An expert system for mechanically ventilated neonate (stand-alone) A tele-decision support system for children with persistent asthma linked to the EMR Characteristics of the 27 Studies Reviewed—Part 1, Continued Ta b l e 1 A: A neonate is assessed by the system with a nurse’s input. The data entry form for assessment provides links to videos, pictures, tables, and graphs illustrating normal and abnormal neonatal data were for nurses’ accurate data entry. P: Diagnoses are generated by the system by a click of a diagnosis button. C: Prebuilt care plans specific to the suggested diagnosis are generated by the system. A: A child is assessed by the system with automated telephone conversation responding to a child’s/parent’s call and asking a child/parent additional questions. P: When problems are identified during conversations, the system generates alerts and sends to tele-asthma nurses. C: Customized education and behavioral intervention are provided by the system and a tele-asthma nurse. Tele-conversation logs and tele-asthma nurses’ case management are transferred to the EMR as a summary report and then physicians provide new orders based on it. CDSS Functions of Assessment (A), Problem Identification (P), Care Plans (C), Implementation (I), and Outcome Evaluation (E) (continues) The system increased nurses’ performance of diagnosis and managed care and nurses’ information access and clinical judgment ability. Study Results on Patient Outcomes and Nurses’ Performance and Decision Making CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. 487 System evaluation by interviews with 12 nurses after system use in three respiratory intensive care units in Taiwan System development Pilot study for feasibility by chart review, and dialogue, focus group, and observations of nurses before and after NOC use in an ambulatory and two home care units in Michigan. Keenan et al (2002)31 Study Description Lee et al (2002)13 Study CDSS An automated nursing data system (stand-alone) A computerized nursing care plan system (stand-alone) Characteristics of the 27 Studies Reviewed—Part 1, Continued T a b l e 1 The system eliminated a need to write care plans by hand and provided standardized care guidelines. There was no consensus among nurses about diagnoses selected by them. Study Results on Patient Outcomes and Nurses’ Performance and Decision Making (continues) P: A nurse selects nursing diagnoses from NANDA on the system. C: The system lists standardized guidelines of care plans. Based on these, patient-specific care plans are selected or devised by a nurse’s input. Each nursing diagnosis is evaluated every shift and care evaluation is documented on paper. It was not part of the system. Charting time on the system A, P, C, and E: The system is a decreased. Web-based application used The process of documenting to create care plans using the and information accessible standardized terminologies, were useful in planning and NANDA, NOC, and NIC. These evaluating care. terminologies are linked to each other and sequentially embedded in the system. A nurse selects appropriate things for care plans through sequential access to NANDA, NOC, and NIC. All the entries are stored and updated from admission to discharge. The system scores patient outcomes on both current and expected status. A nurse queries nursing activities under the care plans on the system. The system provides references on 21 topics of newborn critical care to guide neonatal intensive care unit nurses. A: A nurse selects ones applicable from prebuilt patient assessment data. CDSS Functions of Assessment (A), Problem Identification (P), Care Plans (C), Implementation (I), and Outcome Evaluation (E) 488 CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. Kim et al (2007)33 Kim et al (2007)32 Study System development Pilot study for feasibility by chart review of 1141 patients about activity tolerance from a medical, surgical, and intensive care unit at the Aurora Health Care in Wisconsin System development at the Severance Hospital in Korea Study Description CDSS A clinical documentation system for 22 nursing phenomena within the electronic health record A nursing diagnosis automation system within the EMR Characteristics of the 27 Studies Reviewed—Part 1, Continued Ta b l e 1 A: A nurse selects ones applicable from prebuilt patient assessment data. The prebuilt assessment data were developed from nursing plans and activities that are done in real hospital settings. P: The system automatically presents nursing diagnoses from NANDA based on the selected assessment data. C: NANDA is linked to the NIC items. The nursing plans and activities done in real settings were located as a substructure of the NIC items. E: NANDA is linked to NOC. NOC is tied to nursing plans and activities that are done in real hospital settings. A: The structured assessment form is completed by a nurse’s input. P: The problem is triggered and placed on the problem list by the system. C: Preformulated care plans for the triggered problem are generated by the system. Nursing activities are included in care plans. The system provides hyperlinks to references of care plans. I: Care activities are implemented and documented by a nurse. The triggered problem is automatically removed from the problem list. CDSS Functions of Assessment (A), Problem Identification (P), Care Plans (C), Implementation (I), and Outcome Evaluation (E) (continues) The results indicated a need on the system redesign to adapt nurses’ decisional workflow and increasing staff education. Study Results on Patient Outcomes and Nurses’ Performance and Decision Making CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. 489 Pilot study for feasibility (1) using questionnaires, interviews, and observations for 30 nurses and seven other healthcare providers after system use and (2) by chart review for 38 patients before and after system use in two inpatient units at a large mental health facility in Canada System development (35 nurses from medical and surgical units of two hospitals and 16 nurses from two home care settings in Canada participated in focus group interviews and work sampling observations to identify nurses’ information needs before development) Doran et al (2007)35 Study Description Doran et al (2010)34 Study CDSS A PDA-based decision support system linked to the electronic health record A computerized care planning system for mental health disorders and substance addictions (stand-alone) Characteristics of the 27 Studies Reviewed—Part 1, Continued Ta b l e 1 C: Care plans from best practice guidelines are generated by the PDA. Benchmarking outcome E: An outcome measurement form included in preformulated care plans is completed by a nurse. The system covers 22 nursing phenomena associated with activity tolerance, medication adherence, delirium, fall, sedation, fluid overload, venous thromboembolism, depression, discharge readiness, knowledge deficit on heart failure, intravenous infection, urinary tract infection, dyspnea, and health promotion with hypertension. A or E: Patient outcomes on treatments are assessed by the system with a nurse’s input. Real-time feedback on the assessment is generated by the system. C: Best practice guidelines in a drop-down box are triggered by the system. A nurse selects or customizes care plans from guidelines. The system provides a hyperlink to sources of the practice guidelines. A or E: Patient outcomes on treatments are assessed by the system with a nurse’s input. Real-time feedback on the assessment is generated through the PDA. CDSS Functions of Assessment (A), Problem Identification (P), Care Plans (C), Implementation (I), and Outcome Evaluation (E) (continues) Overall, users were satisfied with the system. There was a significant improvement in some patient outcomes, specifically, aggressive behavior, depression, withdrawal, and psychosis. Study Results on Patient Outcomes and Nurses’ Performance and Decision Making 490 CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. System evaluation by semistructured interviews for eight tele-nurses after system use at three telephone call centers in Sweden Ernesäter et al (2009)37 A CDSS of each study. a System development at the Columbia University in New York Study Description Lee and Bakken (2007)36 Study CDSS A tele-decision support system for triage and advice A PDA-based decision support system for depression, obesity, and smoking cessation (stand-alone) Characteristics of the 27 Studies Reviewed—Part 1, Continued T a b l e 1 achievement of similar patients is possible through the PDA. The best practice guidelines for assessment, prevention, and treatment on pressure ulcers, pain, dyspnea, and falls were developed as formats suitable for PDA use. A: The PDA’s encounter screen and screening screen are completed by a nurse’s input. One screening screen from the options of obesity, smoking, and depression is selected by a nurse. C: Five parts on the care plan screen are completed by a nurse’s input. The five parts are diagnostics, procedures, medications, teaching and counseling, and referrals. They are displayed in drop-down boxes, and if not necessary based on entered patient data, they are automatically dim. A: A patient is assessed by the system with automated telephone conversation with a caller and additional input by a nurse asking relevant questions while hearing the conversation. C: Either visit to a health center or self-care advice is generated by the system. Tele-nurses can override the system by entering a justification. CDSS Functions of Assessment (A), Problem Identification (P), Care Plans (C), Implementation (I), and Outcome Evaluation (E) The system was both supporting and inhibiting for tele-nurses. The system simplifies nurse work, complemented nurse knowledge, and enhanced nurse credibility. However, there were disagreements between nurses and advice by the system. Study Results on Patient Outcomes and Nurses’ Performance and Decision Making T a b l e 2 Characteristics of the 27 Studies Reviewed—Part 2 Characteristics: Number of Studies (Reference/s) Study purpose System development: seven (16, 21, 24, 29, 32, 35, 36) System development and pilot: eight (17, 18, 22, 27, 28, 30, 31, 33) System development and evaluation: two (15, 25) System evaluation: five (7, 13, 20, 23, 37) Pilot: two (14, 34) Others: three (6, 19, 26) Stand-alone CDSSs: 11 (6, 13, 15, 18, 21, 22, 27, 30, 31, 34, 36) Design of pilot and evaluation studies (except seven studies of system development only) Posttest without a control group: 15 (6, 7, 13–15, 17, 18, 22, 25–28, 33, 34, 37) Pretest-posttest using different groups: two (19, 31) One-group pretest-posttest: four (7, 25, 34, 30) Pretest-posttest with nonequivalent control groups: one (20) Randomized controlled trial: one (23) Data collection methods of pilot and evaluation studies Mixed methods: eight (6, 7, 14, 18, 22, 25, 28, 34) Quantitative methods: three (20, 23, 30) Qualitative methods: nine (13, 15, 17, 19, 26, 27, 31, 33, 37) Nursing care areas addressed by CDSSs A single area of nursing care: 18 Delirium care: one (14) Fall-injury management: two (15, 16) Pressure ulcer management: four (17–20) Pain management: two (22, 21) Body temperature monitoring: one (23) Blood glucose control: four (6, 24–26) Blood potassium control: one (7) Referral automation: two (27, 28) Tele-advice for asthma: one (29) Multiple areas of nursing care: nine Depression, obesity, and smoking (mobile based): one ( 36) Pressure ulcer, pain, dyspnea, and fall (mobile based): one (35) For mechanically ventilated neonates: one (30) Mental health disorders and substance addition: one (34) 22 nursing phenomena (see Table 1): one (33) All nursing care areas: three (13, 31, 32) All nursing care areas (tele-advice): one ( 37) Sequential decision support functions of CDSSs Assessment, problem identification, and care plans: eight (7, 13, 14, 15, 23, 26, 29, 30) Assessment, problem identification, care plans, and outcome evaluation: two (31, 32) Assessment and care plans: 27 (all studies) Assessment, care plans, and implementation: one (28) Assessment, care plans, implementation, and outcome evaluation: one (17) Assessment, problem identification, and care plans, implementation, and outcome evaluation: one (33) Starting patient assessment By a nurse’s input: 18 (6, 13, 15, 16, 18–21, 24, 26, 27, 30–36) By a nurse’s input and automatic retrieval of data saved in other electronic systems or databases: five (7, 14, 17, 25, 28) By real-time automatic collection of data: two (23, 29) By real-time automatic collection of data and a nurse’s input: one (37) By a patient’s input: one (22) CDSS-applied nursing care areas,7,14,15,20,34 improved nurses’ work,7,13,17,19,22,23,25,27,28,30,31 simplified nurses’ work, 13,28,37 and complemented nurses’ knowl- edge.18,30,31,37 However, there were still problems in integration with nurses’ workflow,6,17,33 system flexibility,17 user interface,26 learning computer skills, and implementing CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. 491 new guidelines.18 Other problems were malfunctioning computer system issues, lack of administrative leadership,18 and disagreement on system advice.6,13,37 In the studies of system development, because it was common that an interdisciplinary team participated in their system development, it was not described as study subjects in Table 1. As sources of knowledge embedded in decision support systems, all of the studies reviewed ba- sically used scientific evidence such as nationally recognized clinical practice guidelines, randomized controlled trials, systematic review studies, literature review of other study designs, and topic-specific, valid assessment tools. The patterns and types of evidence used were similar among the studies. The features of CDSSs across the studies are synthesized and organized in Table 3. The system features collected Ta b l e 3 Features of Computerized Decision Support Systems Used for Nursing Practice Assessment (reference/s) Providing a prepackaged entry form for accurate and comprehensive patient assessment (all studies) Allowing selection of assessment data applicable to a patient from a prebuilt set (13, 32) Automatically assessing a patient after input of a nurse and/or automatic retrieval of necessary data from other electronic systems/records or databases (all studies, except 13, 24, 31, 36) Automatically transferring assessed data to the electronic medical record for an update (29) Not having an assessment form that is too long to fill it out or to update it (34) Generating some default values of assessment to eliminate the need of entry (17, 31) Problem identification/diagnosis (reference/s) Automatically identifying and triggering a problem of a patient based on assessment data entered (7, 14, 15, 23, 26, 29, 30, 33) Providing NANDA nursing diagnoses translated for cultural differences (13) Care plans (reference/s) Providing evidence-based, standardized, and preprocessed recommendations/guidelines/protocols (all studies) Generating problem-specific care plans based on assessment data (all studies, except 13, 16, 24, 31, 34, 36) Allowing selection of tailored care plans from a drop-down box, a list or check boxes without the need to come up with them (13, 16, 24, 34, 36) Providing recommendations with simple text explanation of the logic, instead of providing only instructions (6, 24, 26) Providing entry space to customize care plans for a specific patient (13, 18, 24, 34, 37) Not providing care plans that are too wordy and have too much text (22) Allowing declination of suggested care plans by selecting reasons from a drop-down list or by typing free text answers (6, 7, 17, 23, 25, 26, 37) Providing hyperlinks to sources of evidence-based guidelines/recommendations (24, 30, 33, 34) Providing nursing activities under care plans (30, 32, 33) Implementation (reference/s) Automatically putting tasks not completed into a next shift (17) Showing completion of the planned referral for a patient (28) Removing a solved problem from a problem list (33) Outcome evaluation (reference/s) Providing a prebuilt form for outcome measurement on implemented care (17, 31, 33) Generating new care plans based on evaluation (17, 34, 35) Others (reference/s) Providing automatic links between CDSS functions (all studies) Using structured (prebuilt) and standardized electronic formats (all studies) Available at the point of care from any location (all studies) Being used in a clinical routine (all studies) Being integrated into nurses’ workflow by allowing of access to CDSS functions at the point of care (all studies) Being integrated into the nursing charting system as the necessary part of documentation, such as generating automatic documentation on care plans, instead of the extra part (15, 17, 24, 26, 28, 33) Having simplicity of the entire routine to use CDSS, such as having fewer screens to access (7, 14, 28) Using standardized terminologies in the forms for sharing data and care continuity among departments (16, 24, 28, 31, 33, 36) Providing adequate user interfaces of the CDSS itself and between other systems and the CDSS to avoid medical errors and for easy use (25, 26, 36) Being easy to personalize templates without requiring specialized skills (19, 24) Limiting the number of reminders to avoid alert fatigue (17, 26) Providing a link to an interdisciplinary communication network such as care conferences and audits for care continuity across settings (15, 24, 34) 492 CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. represented the characteristics of each category of the five stages from patient assessment to outcome evaluation. However, there were differences in the numbers of the system features extracted for each category. The features separately grouped as ‘‘others’’ in Table 3 were associated with the five stages. Certain features, such as being available at the point of care and being used in a clinical routine, were common among all of the CDSSs reviewed. The first feature in the others of Table 3, ‘‘providing automatic links between CDSS functions,’’ means the sequential decision support of CDSSs provided in the stages available from assessment to outcome evaluation. DISCUSSION This study aimed to organize the features of CDSSs useful for nursing practice into assessment, problem identification, care plans, implementation, and outcome evaluation. As a part of the CDSS features, the study identified the diverse ranges of sequential decision support of CDSSs that operated in the stages from assessment to outcome evaluation. The CDSS features related to patient assessment and care plans comparatively varied, whereas the features related to implementation and outcome evaluation did not (Table 3). This indicates that a small number of related studies limited the number of features to be extracted. In fact, there were only three CDSSs providing decision support in an implementation stage and four CDSSs operating in an outcome evaluation stage (Table 2). Eleven of the reviewed CDSSs operated in the stage of problem identification and two features for it were identified. In a single area of nursing care addressed by CDSSs, the step of problem identification by CDSSs would be skipped because the CDSSs were developed and implemented to address the targeted nursing care area. For example, in the study by Gunningberg et al,19 problem identification by a CDSS was not needed because the target area of nursing care was pressure ulcer and the CDSS was used to address the identified problem. However, CDSSs, which operated in multiple areas of nursing care, needed to have useful features for problem identification. In the study by Lee et al,13 nurses had to select nursing diagnoses from a list from the North American Nursing Diagnosis Association (NANDA) that are consistent with patient assessment data. However, there was no consensus among nurses about the diagnoses selected by them. In the implementation step of care plans (Table 3), the CDSSs provided three features about checking the completion of care activities. Unlike other categories with prebuilt formats embedding evidence from literature, decision support in the implementation step was grounded on the performance of nurses. The CDSSs in four studies provided decision support in an outcome evaluation stage (Table 2). Outcome evaluation is a very important stage that should not be omitted for quality patient care. Outcome evaluation allows nurses to determine relationships between patients’ outcome achievement and nursing interventions. After the effectiveness of care plans and intervention is evaluated, the results are fed back into nursing practice.35 Outcome evaluation is an ongoing activity to conduct reassessment of patient status, reordering of priorities, new goal-setting, and revision of care plans. However, most CDSSs reviewed in the study, except the four studies, did not include the function of outcome evaluation on the given nursing care. In two studies, outcome evaluations were implemented outside their CDSS function.13,19 In the case that patient outcome evaluation is not a routine, nurses need to search for appropriate measurements or evidence for patient outcome evaluation; however, such a search may not be carried out for many reasons including a lack of time based on workload, difficulty accessing computers, and/or difficulty finding proper materials. A CDSS needs to provide a prepackaged measurement form or evidence-based recommendations for outcome evaluation. On the other hand, Table 3 shows the common features provided by all of the CDSSs reviewed. Through the organized system features, a comprehensive picture of nursing practice–oriented CDSSs that were attempted up to now was identified. All of the CDSSs reviewed provided sequential decision support in at least two steps; nine CDSSs, in three stages; three CDSSs, in four stages; and a CDSS, in five steps (Table 2). The important thing to which we have to pay attention is the decision support provided in the full scope from initial assessment to outcome evaluation. As grounded in this review, the key steps of a CDSS for sequential decision support were initial patient assessment, problem identification, care plan, and outcome evaluation. It is to provide decision support at the most effective level of nursing care. If such a CDSS is used in a clinical routine, it allows for safe and continuous decision support from the initial stage of patient assessment to the outcome evaluation. Such decision support must be an indispensable part of the CDSS features for quality patient care. There were limitations, although various studies were included in this review to extract the features of CDSSs useful for nursing practice. As most of the studies reviewed were in the stage of system development immediately followed by pilot test or evaluation, one limitation would be that the CDSS features were extracted from such studies, instead of rigorous study designs such as randomized controlled trials. Regardless, the types of the reviewed studies became an advantage in discerning the features of each CDSS because they focused on CDSS functionality. Of the 27 studies reviewed, three studies developed a CDSS as a tool to implement evidence-based practice in nursing, as carefully reviewed and selected evidence was embedded CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. 493 in a CDSS.16,18,36 One study developed a CDSS as a tool to increase the completeness and quality of nursing documentation.19 Therefore, there was a limitation to extracting the features of CDSSs because these studies focused on compliance with evidence-based recommendations and nursing documentation. Lastly, as one study lacked information on system function31 and one study lacked information on outcome evaluation,32 there was difficulty describing the system functions from those studies. For nursing practice and research, the development of a guideline toward an optimum CDSS that best supports nursing practice will have to go beyond the scope of system features identified from a literature review. The steps of sequential decision support by a CDSS were identified, and its importance was emphasized. On the other hand, for empirical support, there is the need to conduct a study to examine clinical effectiveness of CDSSs providing decision support in sequence from initial assessment to outcome feedback. Two suggestions for further research to mitigate the weakness of the reviewed studies are the following: that more nursing care areas become targets of CDSSs and that the effectiveness of CDSSs on decision support for nurses, nurse performance, and patient outcomes be evaluated by rigorous study designs, to have stronger nursing practice-oriented CDSSs. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. CONCLUSION 12. This study organized the features of CDSSs useful for nursing practice into the categories of assessment, problem identification, care plans, implementation, and outcome evaluation, and identified the diverse ranges of the five category-related sequential decision supports that CDSSs provided. This review added the evidence-based knowledge regarding the features of nursing practice-oriented CDSSs. To design the optimum CDSS for nursing practice, a wider range of evidence-based knowledge is needed. Furthermore, providing continuous decision support from the initial stage of patient assessment to outcome evaluation cannot be overemphasized. 13. 14. 15. 16. 17. 18. Acknowledgment 19. I specially thank Dr Jane White at the College of Nursing and Public Health for her assistance with editing. 20. 21. REFERENCES 22. 1. Institute of Medicine Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for 494 the 21st Century. Washington, DC: National Academy press; 2001. Cho I, Kim JA, Kim JH, Kim H, Kim Y. Design and implementation of a standards-based interoperable clinical decision support architecture in the context of the Korean EHR. Int J Med Inform. 2010;79:611–622. doi:10.1016/j.ijmedinf.2010.06.002. Dowding D, Mitchell N, Randell R, et al. Nurses’ use of computerized clinical decision support systems: a case site analysis. J Clin Nurs. 2009;18:1159–1167. Grag AX, Adhikari NKJ, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223–1238. Randell R, Mitchell N, Dowding D, Cullum N, Thompson C. Effects of computerized decision support systems on nursing performance and patient outcomes: a systematic review. J Health Serv Res Policy. 2007:12(4):242–249. Sward K, Orme J Jr, Sorenson D, et al. Reasons for declining computerized insulin protocol recommendations: application of a framework. J Biomed Inform. 2008;41(3):488–497. Hoekstra M, Vogelzang M, Drost JT, et al. Implementation and evaluation of a nurse-centered computerized potassium regulation protocol in the intensive care unit—a before and after analysis. BMC Med Inform Decis Mak. 2010;10:5. http://www.biomedcentral.com/ 1472-6947/10/5. Accessed June 1, 2012. Brokel JM. Infusing clinical decision support interventions into electronic health records. Urol Nurs. 2009;29(5):345–352. Coopmans VC, Biddle C. CRNA performance using a handheld, computerized, decision-making aid during critical events in a simulated environment: a methodologic inquiry. AANA J. 2008;76(1): 29–35. Cho I, Staggers N, Park I. Nurses’ responses to differing amounts and information content in a diagnostic computer-based decision support application. Comput Inform Nurs. 2010;28(2):95–102. Kitchenham B. Procedures for Performing Systematic Reviews. Keele, United Kingdom: Keele University, Software Engineering Group, Department of Computer Science; 2004. NICTA Technical Report 0400011T.1. http://tests-zingarelli.googlecode.com/svn-history/ r336/trunk/2-Disciplinas/MetodPesquisa/kitchenham_2004.pdf. Accessed July 15, 2011. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features crucial to success. BMJ. 2005; 330(7494):765. doi:10.1136/bmj.38398.500764.8F Lee TT, Yeh CH, Ho LH. Application of a computerized nursing care plan system in one hospital: experiences of ICU nurses in Taiwan. J Adv Nurs. 2002;39(1):61–67. Fick DM, Steis MR, Mion LC, Walls JL. Computerized decision support for delirium superimposed on dementia in older adults. J Gerontol Nurs. 2011;37(4):39–47. doi:10.3928/00989134-20100930-01. Browne JA, Covington BG, Davila Y. Using information technology to assist in redesign of a fall prevention program. J Nurs Care Qual. 2004;19(3):218–225. Bakken S, Currie LM, Lee NJ, et al. Integrating evidence into clinical information systems for nursing decision support. Int J Med Inform. 2007;77(6):413–420. Quaglini S, Grandi M, Baiardi P, et al. A computerized guideline for pressure ulcer prevention. Int J Med Inform. 2000;58–59:207–217. Clarke HF, Bradley C, Whytock S, et al. Pressure ulcers: implementation of evidence-based nursing practice. J Adv Nurs. 2005; 49(6):578–590. Gunningberg L, Fogelberg-Dahm M, Ehrenberg A. Improved quality and comprehensiveness in nursing documentation of pressure ulcers after implementing an electronic health record in hospital care. J Clin Nurs. 2009;18(11):1557–1564. Fossum M, Alexander GL, Ehnfors M, Ehrenberg A. Effects of a computerized decision support system on pressure ulcers and malnutrition in nursing homes for the elderly. Int J Med Inform. 2011; 80(9):607–617. Im EO, Chee WS. Decision support computer program for cancer pain management. Comput Inform Nurs. 2003;21(1):12–21. Huang HY, Wilkie DJ, Zong SP, et al. Developing computerized data collection and decision support system for cancer pain management. Comput Inform Nurs. 2003;21(4):206–217. CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. 23. Kroth PJ, Dexter PR, Overhage JM, et al. A computerized decision support system improves the accuracy of temperature capture from nursing personnel at the bedside. AMIA Annu Symp Proc. 2006:444–448. 24. Henry SB, Douglas K, Galzagorry G, Lahey A, Holzemer WL. A template-based approach to support utilization of clinical practice guidelines within an electronic health record. J Am Med Inform Assoc. 1998;5(3):237–244. 25. Vogelzang M, Zijlstra F, Nijsten MWN. Design and implementation of GRIP: a computerized glucose control system at a surgical intensive care unit. BMC Med Inform Decis Mak. 2005;5:38. doi:10.1186/1472-6947-5-38. 26. Campion TR Jr, Waitman LR, Lorenzi NM, May AK, Gadd CS. Barriers and facilitators to the use of computer-based intensive insulin therapy. Int J Med Inform. 2011;80(12):863–871. 27. Heermann LK, Thompson CB. Prototype expert system to assist with the stabilization of neonates prior to transport. Proc AMIA Annu Fall Symp. 1997:213–217. 28. Guite J, Lang M, McCartan P, Miller J. Nursing admissions process redesigned to leverage EHR. J Healthc Inf Manag. 2006;20(2):55–64. 29. Adams WG, Fuhlbrigge AL, Miller CW, et al. TLC-Asthma: an integrated information system for patient-centered monitoring, case management, and point-of-care decision support. AMIA Annu Symp Proc. 2003:1–5. 30. Jirapaet V. A computer expert system prototype for mechanically ventilated neonates. Comput Nurs. 2001;19(5):194–203. 31. Keenan GM, Stocker JR, Geo-Thomas AT, et al. The HANDS project: studying and refining the automated collection of a crosssetting clinical data set. Comput Inform Nurs. 2002;20(3):89–100. 32. Kim Y, An M, Park J, et al. New method of realization of nursing diagnosis based on 3N in an electronic medical record system. Stud Health Technol Inform. 2007:19(pt 1):364–366. 33. Kim T, Lang NM, Berg K, et al. Clinician adoption patterns and patient outcomes results in use of evidence-based nursing plans of care. AMIA Annu Symp Proc. 2007;11:423–427. 34. Doran D, Paterson J, Clark C, et al. A pilot study of an electronic interprofessional evidence-based care planning tool for clients with mental health problems and addictions. Worldviews Evid Based Nurs. 2010;7(3):174–184. 35. Doran DM, Mylopoulos J, Kushniruk A, et al. Evidence in the palm of your hand: development of an outcomes-focused knowledge translation intervention. Worldviews Evid Based Nurs. 2007;4(2):69–77. 36. Lee N, Bakken S. Development of a prototype personal digital assistant-decision support system for the management of adult obesity. Int J Med Inform. 2007;76(Suppl 2):S281–S292. 37. Ernesäter A, Holmström I, Engström M. Telenurses’ experiences of working with computerized decision support: supporting, inhibiting and quality improving. J Adv Nurs. 2009;65(5):1074–1083. For more than 27 additional continuing education articles related to electronic information in nursing, go to NursingCenter.com\CE. CIN: Computers, Informatics, Nursing & October 2013 Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. 495 International Journal of Medical Informatics 64 (2001) 15 – 37 www.elsevier.com/locate/ijmedinf Evaluating informatics applications —clinical decision support systems literature review Bonnie Kaplan * Center for Medical Informatics, Yale Uni6ersity School of Medicine, New Ha6en, CT, USA Received 24 September 2000; accepted 5 July 2001 Abstract This paper reviews clinical decision support systems (CDSS) literature, with a focus on evaluation. The literature indicates a general consensus that clinical decision support systems are thought to have the potential to improve care. Evidence is more equivocal for guidelines and for systems to aid physicians with diagnosis. There also is general consensus that a variety of systems are little used despite demonstrated or potential benefits. In the evaluation literature, the main emphasis is on how clinical performance changes. Most studies use an experimental or randomized controlled clinical trials design (RCT) to assess system performance or to focus on changes in clinical performance that could affect patient care. Few studies involve field tests of a CDSS and almost none use a naturalistic design in routine clinical settings with real patients. In addition, there is little theoretical discussion, although papers are permeated by a rationalist perspective that excludes contextual issues related to how and why systems are used. The studies mostly concern physicians rather than other clinicians. Further, CDSS evaluation studies appear to be insulated from evaluations of other informatics applications. Consequently, there is a lack of information useful for understanding why CDSSs may or may not be effective, resulting in making less informed decisions about these technologies and, by extension, other medical informatics applications. © 2001 Published by Elsevier Science Ireland Ltd. Keywords: Evaluation; Decision support; CDSS; Clinical decision support systems; Clinical practice guidelines; Randomized controlled clinical trials 1. Introduction Systems to aid in medical decision making were introduced over 25 years ago. Relatively few are in general use in clinical settings. * Kaplan Associates, 59 Morris Street, Hamden, CT 06517, USA. Tel.: +1-203-777-9089; fax: + 1-203-777-9089. E-mail address: bonnie.kaplan@yale.edu (B. Kaplan). Despite their potential usefulness, the lack of widespread clinical acceptance long has been of concern among researchers and medical informaticians [1–3]. This paper reviews literature that focuses on evaluation of clinical decision support systems (CDSS). The paper discusses the following key findings: The main emphasis is on changes in clinical performance that could 1386-5056/01/$ – see front matter © 2001 Published by Elsevier Science Ireland Ltd. PII: S 1 3 8 6 – 5 0 5 6 ( 0 1 ) 0 0 1 8 3 – 6 16 B. Kaplan / International Journal of Medical Informatics 64 (2001) 15–37 affect patient care. Many evaluations of CDSSs use designs based on laboratory experiment or randomized controlled clinical trials (RCTs) to establish how well the systems or physicians perform under controlled conditions. Other approaches to evaluation, such as ethnographic field studies, simulation, usability testing, cognitive studies, record and playback techniques, and sociotechnical analyses rarely appear in this literature. As was the case over ten years ago, few systems have been evaluated using naturalistic designs to study actual routine CDSS use in clinical settings. Consequently, the CDSS evaluation literature focuses on performance or specific changes in clinical practice patterns under pre-defined conditions, but seems lacking in studies employing methodologies that could indicate reasons for why clinicians may or may not use CDSSs or change their practice behaviors. Further, there is little reference in the CDSS literature to a theoretical basis for understanding the many issues that arise in developing and implementing CDSSs. In addition, the studies concern physicians to the near exclusion of other clinicians or potential users. Lastly, the literature seems not to be informed by studies of other medical computer applications, such as hospital information systems (HISs), computer based patient records (CPRs), physician order entry (POE), or ancillary care systems. These studies could provide useful insights into issues that likely would be relevant to acceptance and use of CDSSs. 2. Literature review methods An automated literature search was done using Medline with the assistance of a librarian. This search identified papers classified as about a: (1) decision support system; (2) clinical decision support system; (3) expert sys- tem; and (4) decision aid. ‘CDSS’ has a variety of definitions. Any system that was considered a CDSS by the authors and catalogers of the papers reviewed was considered so for purposes of this review. This decision was made, instead of using an a priori definition of CDSS, so as to provide a view of the literature as it is presented and categorized by those involved. Using the authors’ and catalogers’ keywords is indicative of how those authors wish to have their work categorized and how this work is viewed within the discipline. It indicates how ‘CDSS’ is construed by those who are working within or commenting upon this area. Moreover, an a priori definition could result both in excluding papers authors consider as reporting on CDSSs, and in biasing results towards some particular type of system or definition. Further, the focus here is on evaluation, not on any particular type of CDSS. Hence, as in the guide published by Journal of American Medical Association to using articles evaluating the clinical impact of a CDSS [4], the search did not focus on any particular types of CSSS, such as alerting systems or diagnostic systems, but included them all. The automated search spanned the years 1997 and 1998. To supplement the automated search, a manual search also was done. This included papers that had been referenced frequently by other papers, papers and authors known by reputation, review papers, papers in recent journals and proceedings, and books. The manual search was not limited in time period, but included years both before and after the automated search. This was especially the case for well-known review papers. Including recent review papers provided a more comprehensive scope to this undertaking. By examining review papers and commentaries that were published in past years, current work could be compared with prior trends in the CDSS literature. Doing so also B. Kaplan / International Journal of Medical Informatics 64 (2001) 15–37 helped insure that significant works over the history of CDSSs were considered. Inclusion criteria for the manual review were any work concerning evaluation or success factors for expert systems or clinical decision support systems, and works describing evaluations of other systems or of evaluation approaches. Papers identified in the search, but that clearly were irrelevant, were omitted from further consideration, leaving over 140 items that were reviewed thoroughly. Of these, only ten were found to be totally irrelevant. Those that were reviewed included research reports, editorials, reviews, descriptive and normative writings— in short anything that Medline returns from a search — and books. What follows is an analysis of themes and trends in the literature that was reviewed. 3. Usefulness of CDSSS The literature indicates a general consensus that clinical decision support systems are thought to have the potential to improve care, or at least to change physicians’ behavior [5]. Reminders [6–10]. alerts [11– 17], treatment plans [6], and patient education [6], have been reported as effective in changing practice behaviors. Evidence of positive effect is more equivocal for guidelines [18–21]. Some studies suggest that guidelines are effective [19,22–28], and others that they are not [19,29]. There have been substantial rates of physician noncompliance with standards [29,30]. There is little evidence that physicians comply with guidelines, whether or not incorporated into a CDSS [20,27,31–34]. Whether systems aid physicians with diagnosis also is unclear [8,35 – 38]. Some see these results as exciting valida- 17 tions of the value of CDSSs. Others point out that, at best, the results are a ‘disappointment’ [36]. In addition, although physicians’ behavior may be shown to change, there has been little study of whether the thinking behind the behavior has changed [39]. Studies of patient outcomes showed little significant improvement [5]. It also has been difficult to establish that patient outcomes have been affected [8,20,29,40,41]. Lastly, there is general consensus that a variety of systems are little used despite their demonstrated or potential benefits [18,42–47]. 4. Evaluations of CDSS Appendix A profiles all the evaluation studies of CDSSs found in the literature search. There are 27 studies reported in 35 papers. Papers reporting related studies are counted as one study each, though they are listed separately. Two of the 35 papers [48] are substantially the same, and, therefore, listed as one entry in the table. A review of the studies in Appendix A suggests several notable tendencies: 1. As Appendix A shows, most studies are of specific changes in clinical performance that could affect patient care. 2. As is evident from Appendix A, most studies use an experimental or RCT design. With only six multi-methods studies, plus three more using qualitative methods, methodological diversity is limited. Other approaches to evaluation [49,50], such as ethnography, simulation, usability testing, cognitive studies, record and playback techniques, network analysis, or sociotechnical analyses rarely appear in this literature. Few studies involve field tests of a CDSS and almost none (two studies of CDSSs per se [51,52]) use naturalistic 18 B. Kaplan / International Journal of Medical Informatics 64 (2001) 15–37 designs in actual clinical settings with real patients (although one study used simulated patient encounters with actors playing the part of patients [37], and a number of studies of effects of alerts or reminders are based on actual treatment records). 3. There is little theoretical discussion in these papers. One study presented a theoretical analytical model [53]. Although a few mention theory, explicit theory is absent from most papers. Tacitly, papers are permeated by a rationalist or rational choice perspective. 4. As indicated in Appendix A, studies concern physicians to the near exclusion of other clinicians or potential users such as patients, administrators, project team members, insurers etc. Three studies include nurses [48,51,54]; one included providers, assistants, and patients [55]; and one concerns the project team and project history [54,56,57]. 5. Judging from citations as well as the text, there is little mention of evaluations of other informatics applications. These trends are reflected in recent review papers as well, as shown in Appendix B, which summarizes these review papers. Discussion of these tendencies follows, with focus on the first three. This paper describes and analyzes the literature. Fuller implications of these observations, together with an analytical critique of the literature, are discussed elsewhere in this volume [58]. 4.1. Focus on system and clinical performance It has been reported that evaluations of CDSSs tend to concern system accuracy rather than either how well clinicians perform when actually using these systems, or the impact of system use on clinical care [35,43, 59,60].1 Elsewhere, it was found that evaluations of diagnostic systems tend toward process measures concerning performance of the system’s user [61]. Evaluations identified in this review tend towards two kinds. The first are studies assessing CDSS accuracy and performance. A recent review emphasizes system functionality [62,63], and studies of decisionsupport systems usually rate the objective validity of the knowledge base, for example, by measuring performance against some gold standard [60,64,65]. However, only one study [43,66] listed in Appendix A concerns system performance. 2 Although few applications are evaluated in practice [67], the second kind of evaluation, which dominates in Appendix A, concerns patient care more directly. Appendix C lists studies according to the kind of CDSS involved. As shown in Appendix C, most of the evaluation studies (21 of 27 studies) concern systems for alerts or reminders (nine papers), guidelines (six studies), and diagnosis (six studies). These studies are of specific changes in clinical performance that could affect patient care. This preponderance also is evident in Appendix B. Some of the studies investigate 1 It is possible this depends on when the observation was made (or, as suggested in the next footnote, on search criteria). There is some evidence to suggest, both in others’ work as well as in this review, that there has been a shift from system performance to user behavior. However, these citations are from 1998 to 1999, suggesting that the question of shift in emphasis bears further investigation. 2 This may be due to the search criteria. For example, searching on ‘diagnosis, computer assisted’ identified 14 papers from the years 1997 – 2000. Of these, 11 assessed system performance. For reasons explained above, because no particular kind of CDSS was to be favored in this review, neither ‘diagnosis, computer assisted’ nor any other was included as a term in the automated search for this paper. Possibly this orientation predominates more in some publication outlets than in others. As noted in Appendix D, a large proportion of the papers were published in American Medical Informatics Association outlets and J. Am. Med. Assoc., while almost all papers were published by journals based in the US, even though authors may not be US based. B. Kaplan / International Journal of Medical Informatics 64 (2001) 15–37 changes in physicians’ practices, such as whether alerts affect prescribing behavior. Others are studies of changes in workflow and order processing, for example, time from order to delivery. This focus is well suited to study through experiment, and RCT is the dominant influence on study design. These kinds of measures are proxies for the very difficult issue of determining system effect on patient outcomes. 4.2. Study design and setting RCTs and other experimental approaches have a long tradition as the standards for research design in clinical medicine [60,68,69]. It is not surprising that this also is the case in medical informatics. Van der Loo’s study of evaluations of health care information systems from 1974 to early 1994 examined 108 studies against a set of quality standards. Study designs were ranked so that randomized trials were considered the ‘highest’, while qualitative designs are not discussed. Although 50% of the studies concerning what he classified as diagnostic systems or as treatment systems used an RCT design, only six of all the 108 studies met the stringent standard of economic analysis combined with an RCT. Disappointing quality scores for many of the studies he reviewed led him to call for a more a rigorous approach [61]. A substantial body of opinion in medical informatics supports this view. In pleading for controlled trials in medical informatics, for example, Tierney et al. state in the American Informatics Association editorial [70]: Only by performing rigorous clinical studies can we define whether a new information system will help, result in no change, or make the problem worse. The CDSS literature clearly reflects this opinion. Normative papers call for randomized 19 controlled trials [5]. Physicians are advised to apply the same criteria to assessing evaluations of CDSSs as of drugs or any other intervention. [4]. As indicated in Appendix 1, most papers reporting evaluations were experiments done under controlled conditions, even when in natural settings, so there was little methodological diversity. Of the papers listed in Appendix 1, two use these designs involving surveys [26,34], while only one uses surveys without experimental or RCT design [17]. Only four are multi-method (e.g. combinations of surveys, interviews, or observations) [71–74], plus two more studies are not for CDSSs per se but primarily involve computer-based clinical records [54,55,57]. Only the six multi-method studies plus three others [51,52,64] use qualitative methods, for a total of nine in all. Some of these authors explicitly stated how valuable they found using multiple methods, perhaps feeling a need to address the dominance of experimental approaches in this way. Five reported getting useful information through interviews and observations that would guide systems development [52,64,71– 73]. As shown in Appendix 2, the RCT emphasis dominates for CDSS review papers. The same appears true for papers reviewing clinical guidelines’ effectiveness, educational strategies, or barriers (though a comprehensive search was not done for these papers). Despite reviewers’ claims that ‘the simple randomised trial cannot be regarded as the gold standard in behavioural research’ [25], their reviews are limited to randomized trials and other experimental and statistical methods considered rigorous. Authors make a distinction between showing that a CDSS works under laboratory conditions and showing that it works under clinical conditions. Some recommend a multi-stage evaluation process, with evaluating functionality in real-life situations and evaluating system impact as the last stages [65]. Some 95% of 20 B. Kaplan / International Journal of Medical Informatics 64 (2001) 15–37 systems never reach the stage of field evaluation [37]. Apparently, there has been little change in this number over the years. It comes from a review published 10 years (1987) prior that noted that ‘[a]pproximately 90% of all computerized medical expert systems have not been evaluated in clinical environments’ [75]. A few years later, according to a 1990 report [76]: [O]nly about 10% of the many medical knowledge-based systems that have been described over the years have been tested in laboratory conditions, while even fewer have been exposed to clinical trials. Appendix A lists few studies involving field tests. Thus, it seems that very few CDSSs have been independently evaluated in clinical environments (or that no one has recounted them). This remains the case even though calls repeatedly have been made to field test CDSSs so as to demonstrate that they work in patient care settings, and even though some of those calls are from researchers who have not conducted their studies in this way, e.g. [36,66,75,76]. Some authors further recommend that these field settings be remote from and relatively independent of system developers because ‘study design needs to rest upon making sure that the reasons for success or failure are clear’ and ‘be broad enough to detect both intended and unintended effects’ [77]. Some call for assessing systems in actual use, under routine conditions, and for understanding why the results of such assessments are as they turn out to be. Nevertheless, they say little about how to achieve this understanding, and further, they either propose, or actually carry out, evaluations based on a clinical trials or experimental models, e.g. [35,36,76]. Clinical trials, even in practice settings, are considered the ‘ob- vious’ approach [59]. As substantiated in the appendices, the evaluation focus is on how CDSSs affect clinical processes or outcomes [5]. In what is perhaps the closest simulation to a real patient visit, Ridderikhoff and van Herk use cases constructed from real patient data and an actor playing the patient. They also include observational data in their report [37]. Berg [51] and Kaplan et al. [52] each are unusual in reporting detailed naturalistic observational field studies of CDSSs in actual use with real patients under routine clinical conditions. A review of evaluations of all medical informatics applications reported in the 1997. AMIA Proceedings found patterns similar to those reported here. Almost all of those evaluations were of CDSSs and the primary evaluation design was modelled on controlled trials. Generally, individual systems were evaluated against expert human performance, or subjects were given simulated patient cases so that their performance with and without an automated system was compared [78]. 4.3. Theoretical orientation Although few authors discuss theory, this review indicates a strong theoretical preference underlying most studies. As indicated above, most employ an experimental or RCT design and use solely quantitative data collection and data analysis methods. Thus, studies reflect an objectivist epistemological stance and quantitative methodological approach [49,50]. They evidence a rationalist or rational choice perspective and focus on measurable variances by comparing effects of system use with other circumstances [78– 81]. This perspective often is tacit. As one example, it was reported in 1990 that decision B. Kaplan / International Journal of Medical Informatics 64 (2001) 15–37 aids in medicine usually are evaluated by measuring the structure of the aid, the function of the aid, or the impact of the aid on users and patients. While ‘impact … on users and patients’ might seem to imply a different approach, instead, here it refers to effects of the system on process measures, such as accuracy, timing, and confidence of decisions; or effects of the system on outcome measures, such as patient morbidity and mortality, or cost per procedures [76]. A similar orientation is evident in three categories of reasons that are given for evaluation: ethical, legal, and intellectual. Despite the apparent breadth of these three categories, the focus is on measurable economic and technical factors. For example, legal evaluation, in this instance, includes how effective and how safe a system is, and how it might change resource use. Thus, even where authors recognize that ‘the unusual properties of medical expert systems’ make it necessary to modify randomized doubleblinded controlled trial for field trials, their suggestions remain within a rationalist framework [76]. This tacit perspective also is apparent among other evaluators. Advocates of a systems approach that includes taking full account of ‘medical, economic, technical, organisational and behavioural dimensions’ when doing an economic evaluation [82], thereby subordinate these concerns to economic analysis. Some discuss user acceptance without mentioning cultural and sociologic factors [4], while others state that these factors need to be considered. Nevertheless, these authors, like those who do not mention such contextual factors [46], discuss acceptability in terms of user- and machine-machine interfaces, response time, and similar technical issues. Some limit discussion of user acceptance to the interface while emphasizing that the purpose of eval- 21 uation is safety through accuracy and adequacy of the domain knowledge [83,84]. When considering the usability and acceptance of the interface, subjective measures such as user questionnaires and expert review are not valued highly [84], even though physicians consider having tools that add value to the practice setting more valuable than usability [71]. A broader emphasis on user satisfaction, if discussed at all, is on developing generic satisfaction instruments and appropriate controls [5]. As these examples illustrate, the underlying approach fits an objectivist, rationalist philosophical orientation and design employing quantitative methods to measure variance, even if not explicitly acknowledged. 5. Conclusions Despite calls for alternatives, or recommendations to select designs congruent with system development stage and different evaluation questions [49,50,65,67], RCTs remain the standard for evaluation approaches for CDSSs [85,86], making evaluation traditions for CDSSs similar to those for other computer information systems, whether or not they may be intended for use in health care. Most commonly, systems, whether medical or not, have been evaluated according to selected outcomes pertaining to features such as technical or economic factors at the expense of social, cultural, political, or work life issues [79,80,87]. RCT and other experimental designs are excellent for studying system performance or specific changes in clinical practice behaviors, but not well suited for investigating what influences whether systems are used. Consequently, some other evaluation approaches have been developed, including simulation, usability testing, cog- 22 B. Kaplan / International Journal of Medical Informatics 64 (2001) 15–37 nitive studies, record and playback techniques, ethnography, sociotechnical analyses, and social interactionism among them. Commentary concerning implementation issues and barriers to system use are little different today from what has been reported over the past 50 years [2]. This may be partly because system evaluations often ignore issues concerning user acceptance or changes in work, an omission also evident in the literature that was reviewed for this paper. By focusing on pre-specified outcome measures, evaluations do not examine processes of actual system use during daily activities [88]. As a result, we have excellent studies that indicate decreases in medication errors with physician order entry [11] or when notified by pharmacists or radiology technicians about drug alerts [16], changes in physician prescribing behavior for at least 2 years after a study [22], and greater compliance with guidelines [20]. Yet we have not sufficiently studied why these were the results. Nor have we investigated reasons behind other, less encouraging, findings. We have little understanding of why, for example, physicians agreed with 96% of one system’s recommendations but only followed 65% of them [31], why, in another study, there was an overall increase in compliance with guidelines but the compliance rate still was low [27]; or, in another, why there was an increase in compliance, except for three items [28]; or why only the same four of six groups of preventive practice were improved with either reminders that were computer generated or those that were manual, but all six groups improved with computer plus manual reminders [10]. Despite these improvements, another study indicates that there were no significant differences in complying with guidelines between physicians who received computerized reminders and those who did not [19]. What accounts for these differences? Elsewhere, individuals found their post-implementation experiences fell short of their expectations [72]. Why did this happen, and how much does it matter? Study designs did not address questions that allow deeper understanding of these findings, understanding that could indicate why different results obtain in different studies. Consequently, we cannot learn what to do that might improve the practices that these studies measure. Other research approaches are little reflected in the CDSS evaluation literature. These omissions are impoverishing our understanding of CDSS as they might actually be used [58]. RCT-type studies are excellent for demonstrating whether a particular intervention has a pre-specified effect. Such studies of CDSSs are valuable. Nevertheless, they tell us little about whether clinicians will incorporate a particular CDSS into their practice routine and what might occur if they attempt to do so. Such studies cannot inform us as to why some systems are (or will be) used and others are not (or will not be), or why the same system may be useful in one setting but not in another. They do not indicate why a CDSS may or may be not effective. Different study designs answer different questions. A plurality of methodological approaches and research questions in evaluation is needed so as to broaden our understanding of clinical acceptance and use of informatics applications [58]. Acknowledgements I am grateful to Dr Richard Spivack of the US National Institute of Standards and Technology for his invaluable assistance in the automated literature search. B. Kaplan / International Journal of Medical Informatics 64 (2001) 15–37 23 Appendix A CDSS Evaluation Authors System Study design Findings Bates et al. 1998 [11]* Drug Alerts Comparison of medi- POE decreased rate of medication errors. POE cation errors before Team intervention conferred no additional and after implemen- benefit over POE. tation, and also with and without team intervention Bates et al., 1999 [12]* Drug Alerts Comparison of medi- POE decreased rate of medication errors POE cation errors at different time periods Berner et al. [43]+ Dx DSS Comparison of physi- Physicians’ performed better on the easier cians’ performance cases and on the cases for which QMR on constructed cases could provide higher-quality information. Berner et al., Dx DSS 1994 [66]+ Comparison of programs’ performance Berg, 1997 [51] Case studies in clini- Actor-network theory is used to describe cal settings how system implementation changed both the system and work practices. Dx DSS No single computer program scored better than the others. The proportion of correct diagnoses ranged from 0.52 to 0.71, and the mean proportion of relevant diagnoses ranged from 0.19 to 0.37. Bouaud et al., Guidelines Measured physicians’ Clinicians agreed with 96% of the recom1998 [31] agreement and com- mendations and followed one of the recompliance with guidemendations in 65% of cases. lines Buchan et al., Guidelines Comparisons of pre- Participation was followed by a favorable 1996 [22] scribing behavior change in clinical behavior which persisted for at least two years. Friedman et Dx DSS al., 1999 [35]† Comparison of physi- DSS consultation modestly enhanced cians’ Dx using dif- subjects’ diagnostic reasoning. ferent systems in laboratory setting B. Kaplan / International Journal of Medical Informatics 64 (2001) 15–37 24 Gadd et al., 1998 [71] DSS inter- Comparison of per- Features that improve perceived usability face ceptions of different were identified. prototype versions of the system through video observation, surveys, and interviews Gamm et al., Computer 1998 [72] based patient record Comparison of preand post-installation survey data. Also did interviews and observations. Pre-installation, most respondents were moderately positive about the helpfulness and utility of computerization in their practice. Post-installation experience fell short of those expectations. Jha et al., 1998 [13]* Compare computerbased adverse drug event (ADE) monitor against chart review and voluntary report by nurses and pharmacists The computer-based monitor identified fewer ADEs than did chart review but many more ADEs than did stimulated voluntary report. Kaplan et Guidelines Case study using al., 1997 [52] observation and interviews concerning diagnostic and treatment guidelines in psychiatry Karlsson et DSS Study how clinicians al., 1997 [64] viewed using this way of accessing information through interviews using ‘stimulated recall’. Design suggestions and user acceptance issues were identified. Kuperman et Lab Alerts Compare time to al., 1999 [14] treatment with and without automatically paging the physician. The automatic alerting system reduced the time until treatment was ordered. Drug Alerts The major uses of the system were for patient-specific support and continuing medical education. Three parametersrelevance, validity, and work were important. B. Kaplan / International Journal of Medical Informatics 64 (2001) 15–37 Lauer et al., 2000 [53] Patient Case study assessing The model helps provide a theory-based scheduling system against a understanding for collecting and reviewing priori model. users’ reactions to, and acceptance or rejection of, a new technology or system. Litzelman et al., 1993 [26]‡ Reminder Prospective, randomized, controlled trial. Compared compliance with computergenerated reminders between 2 groups of physicians. 25 Compliance with computer-generated reminders was higher in the group that received printed reminders and also was required to indicate response to reminders than in the group not required to indicate response. Litzelman Reminders Survey of physicians. 55% of computer-generated reminders were and Tierney, not complied with. Of those, 23% were 1996 [34]‡ not applicable and 23% would be done at the next visit. Of those to be done at the next visit, the stated reason was 84% because of lack of time this visit. Lobach and Hammond 1997 [27] Guidelines Controlled trial comp…
Purchase answer to see full attachment