Domain

Analytic Methods

Type

Empirical Study

Theme

effectiveness; quality

Start Date

7-6-2014 2:55 PM

End Date

7-6-2014 4:15 PM

Structured Abstract

Introduction

The availability of high fidelity electronic clinical data for quality improvement (QI) and comparative effectiveness research (CER) is a hallmark of the learning healthcare system. Yet, these data are not directly ‘fit for use’. Challenges of data extraction, aggregation, and standardization abound. Washington State’s Surgical Care Outcomes and Assessment Program (SCOAP) is a network of hospitals that participate in clinician-led QI registries, for which data are manually abstracted from electronic health records (EHRs). The goal of the Comparative Effectiveness Research and Translation Network (CERTAIN) was to semi-automate SCOAP data abstraction. Using a federated data model that captures HL7 standard messages in real time, CERTAIN created a centralized data repository (CDR), and assessed whether clinical data collected as a by-product of healthcare delivery can be secondarily used to conduct QI and CER. Herein we describe automation and validation, and illustrate the complexities involved.

Methods

Investigators installed Caradigm’s Amalga. Amalga retrieves data from all types of EHRs, storing message queues that permit data manipulation independently from clinical systems. Virtual private networks allowed data to flow from select feeds to a common data model. Within the CDR, data were transformed to SCOAP standards. Throughout the project (10/2012-7/2013), investigators operated manual and automated abstraction systems in parallel, and conducted three phases of validation. In phase 1, ingested data from 50 patients per site were validated. In phase 2, 20 SCOAP cases per registry, per site, were randomly selected from within the CDR. In phase 3, matched cases of transformed, automated data elements were compared to those manually abstracted, creating pairs. Pair concordance and discordance were calculated. All discordant pairs and 10% of matched records were validated. In each phase, data elements (#1/#2) or pairs (#3) were compared against the EHR (gold standard), and iterative improvements made until each matched the EHR with 95% accuracy.

Findings

Four of five sites participated in validating data from 2-4 registries (general/vascular/oncology/spine), through 20 feeds (registration/laboratory/dictation/medication/radiology). Depending on type of SCOAP registry, 6%-15% of data elements were prioritized and abstracted; 51%-86% from structured data, the remaining using natural language processing (NLP) algorithms. In phase 1, 12/20 feeds reached 95% accuracy. In phase 2, 55% of structured data elements performed with 96-100% accuracy, 35% between 85-95%, 10% with ≤ 85%. NLP algorithms performed with 89-91% accuracy. In phase 3, concordance ranged from 69%-89%, depending on type of SCOAP registry. For discordant pairs, 58% of manually abstracted data elements matched the EHR, 38% of automated. Improvements were continuing at project end.

Lessons Learned

Semi-automated data abstraction is useful to collect certain types of EHR data. The CERTAIN validation project illustrates the complexities involved in creating a common data model from multiple EHRs, across diverse health systems. Inherent limitations in EHRs and resources limited the proportion of data elements abstracted. The residual variation in validation metrics reveals that raw data collected as a by-product of healthcare delivery is not immediately ready for secondary use.

Call to Action

Secondary use of EHR data will require new approaches to data analysis that can accommodate extant EHR data, and more systematic approaches to gathering EHR data.

Acknowledgements

Acknowledgements

This project was supported by grant number R01HS020025 from the Agency for Healthcare Research and Quality (all authors), and grant number UL1TR000423 from the UW Institute of Translational Health Sciences (ITHS), National Center for Advancing Translational Sciences (Tarczy-Hornoch). The content is solely the responsibility of the authors and does not necessarily represent the official vies of the Agency for Healthcare Research and Quality. The Surgical Care and Outcomes Assessment Program (SCOAP) is a Coordinated Quality Improvement Program of the Foundation for Health Care Quality. CERTAIN is a program of the University of Washington, the academic research and development partner of SCOAP. Personnel contributing to this study: Centers for Comparative and Health Systems Effectiveness (CHASE Alliance), University of Washington, Seattle, WA: Peter Tarczy-Hornoch, MD; Erik Van Eaton, MD; Daniel Capurro, MD; Allison Devlin, MS; Beth Devine, PhD, PharmD, MBA; Rafael Alfonso, MD, PhD, Michael Tepper, PhD; Meliha Yetisgen-Yildiz, PhD, MSc; Marisha Hativa, MSHS; Kevin Middleton; Megan Zadworny, MHA. Institutions contributing to this study: Harborview Medical Center (Seattle, WA); EvergreenHealth (Kirkland, WA); Northwest Hospital and Medical Center (Seattle, WA); University of Washington Medical Center (Seattle, WA); and Virginia Mason Medical Center (Seattle, WA).

Creative Commons License

Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License.

Share

COinS
 
Jun 7th, 2:55 PM Jun 7th, 4:15 PM

Preparing electronic clinical data for quality improvement and research: The CERTAIN validation project

Introduction

The availability of high fidelity electronic clinical data for quality improvement (QI) and comparative effectiveness research (CER) is a hallmark of the learning healthcare system. Yet, these data are not directly ‘fit for use’. Challenges of data extraction, aggregation, and standardization abound. Washington State’s Surgical Care Outcomes and Assessment Program (SCOAP) is a network of hospitals that participate in clinician-led QI registries, for which data are manually abstracted from electronic health records (EHRs). The goal of the Comparative Effectiveness Research and Translation Network (CERTAIN) was to semi-automate SCOAP data abstraction. Using a federated data model that captures HL7 standard messages in real time, CERTAIN created a centralized data repository (CDR), and assessed whether clinical data collected as a by-product of healthcare delivery can be secondarily used to conduct QI and CER. Herein we describe automation and validation, and illustrate the complexities involved.

Methods

Investigators installed Caradigm’s Amalga. Amalga retrieves data from all types of EHRs, storing message queues that permit data manipulation independently from clinical systems. Virtual private networks allowed data to flow from select feeds to a common data model. Within the CDR, data were transformed to SCOAP standards. Throughout the project (10/2012-7/2013), investigators operated manual and automated abstraction systems in parallel, and conducted three phases of validation. In phase 1, ingested data from 50 patients per site were validated. In phase 2, 20 SCOAP cases per registry, per site, were randomly selected from within the CDR. In phase 3, matched cases of transformed, automated data elements were compared to those manually abstracted, creating pairs. Pair concordance and discordance were calculated. All discordant pairs and 10% of matched records were validated. In each phase, data elements (#1/#2) or pairs (#3) were compared against the EHR (gold standard), and iterative improvements made until each matched the EHR with 95% accuracy.

Findings

Four of five sites participated in validating data from 2-4 registries (general/vascular/oncology/spine), through 20 feeds (registration/laboratory/dictation/medication/radiology). Depending on type of SCOAP registry, 6%-15% of data elements were prioritized and abstracted; 51%-86% from structured data, the remaining using natural language processing (NLP) algorithms. In phase 1, 12/20 feeds reached 95% accuracy. In phase 2, 55% of structured data elements performed with 96-100% accuracy, 35% between 85-95%, 10% with ≤ 85%. NLP algorithms performed with 89-91% accuracy. In phase 3, concordance ranged from 69%-89%, depending on type of SCOAP registry. For discordant pairs, 58% of manually abstracted data elements matched the EHR, 38% of automated. Improvements were continuing at project end.

Lessons Learned

Semi-automated data abstraction is useful to collect certain types of EHR data. The CERTAIN validation project illustrates the complexities involved in creating a common data model from multiple EHRs, across diverse health systems. Inherent limitations in EHRs and resources limited the proportion of data elements abstracted. The residual variation in validation metrics reveals that raw data collected as a by-product of healthcare delivery is not immediately ready for secondary use.

Call to Action

Secondary use of EHR data will require new approaches to data analysis that can accommodate extant EHR data, and more systematic approaches to gathering EHR data.