Submission Type

Empirical Research


machine learning, data use and quality, health information technology, natural language processing, classification, electronic health records, data collection, quality improvement


Introduction: As the number of clinical decision support systems incorporated into electronic medical records increases, so does the need to evaluate their effectiveness. The use of medical record review and similar manual methods for evaluating decision rules is laborious and inefficient. Here we use machine learning and natural language processing (NLP) algorithms to accurately evaluate a clinical decision support rule through an electronic medical record system and compare it against manual evaluation.

Methods: Modeled after the electronic medical record system EPIC at Maine Medical Center, we developed a dummy dataset containing physician notes in free text for 3621 artificial patients records undergoing a head computed tomography scan for mild traumatic brain injury after the incorporation of an electronic best practice approach. We validated the accuracy of the BPA using three machine learning algorithms (SVC, DecisionTreeClassifier; KNeighborsClassifier) by comparing their accuracy for adjudicating the occurrence of a mild traumatic brain injury against manual review. We then used the best of the three algorithms to evaluate the effectiveness of the BPA and compared the algorithm’s evaluation of the BPA to that of manual review.

Results: The electronic best practice approach was found to have a sensitivity of 98.8% (96.83-100.0), specificity of 10.3%, PPV = 7.3%, and NPV = 99.2% when reviewed manually by abstractors. Though all the machine learning algorithms were observed to have a high level of prediction, the SVC displayed the highest with a sensitivity 93.33% (92.49-98.84) , specificity of 97.62% (96.53-98.38), PPV = 50.00, NPV = 99.83. The SVC algorithm was observed to have a sensitivity of 97.9% (94.7-99.86), specificity 10.30%, PPV 7.25%, and NPV 99.2% for evaluating the best practice approach, after accounting for 17 cases (0.66%) where the patient records had to be reviewed manually due to the NPL systems inability to capture the proper diagnosis.

Discussion: Evaluation of clinical decision support systems incorporated into electronic medical records can be achieved in an automatic fashion by using natural language processing and machine learning techniques.

Creative Commons License

Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License.