Domain

Analytic Methods

Type

Case Study or Comparative Case Study

Theme

effectiveness

Start Date

7-6-2014 1:15 PM

End Date

7-6-2014 2:45 PM

Structured Abstract

Introduction

The American Recovery and Reinvestment Act of 2009 (ARRA) directed nearly $18.2 million to comparative effectiveness research (CER) methods development to foster the production of rigorous and unbiased research and to disseminate the evidence to patients, clinicians, and other decision-makers. To help inform future investments in CER methods, we describe the ARRA-funded methods projects, identify project facilitators and barriers, and discuss the alignment of topics studied with published methods development priorities.

Methods

Twenty-three projects had a primary focus on methods development. To identify project facilitators, and barriers we (1) interviewed 15 investigators and held a webinar discussion with three other investigators and (2) analyzed results from an investigator survey. We identified methods development priorities from a variety of sources released before and during the funding period. Using project proposals and interview notes, we assessed which methods priorities were addressed by the funded projects.

Findings

Projects studied one or more aspects of methods research including comparability of interventions, generalizability of findings, outcomes measurement, data analysis, and dissemination of methods knowledge. Topics included methods that address confounding and bias; Bayesian and adaptive designs; standardization and validation of patient-reported outcomes; methods to identify heterogeneous treatment effects; methods for combining data from randomized and observational studies; and simulation modeling.

Projects directly addressed 12 of 18 identified priority topics. Several projects addressed topics that can help inform methods priority topics. For example, two projects studied Bayesian methods, highly relevant to the priority topic that calls for development of guidance on use of Bayesian methods in CER.

Common project challenges cited by investigators included lack of access to data with sufficient detail, data limitations (missing data or lack of standardization of data elements, for example, in electronic medical records), and delays with data access.


Key Lessons

Although funded projects explored many identified priority topics, investigators noted that much work remains. For example, observational data analysis methods (such as controlling for confounding, risk-adjustment, indirect treatment comparisons, and standardization of patient-reported outcomes) will become increasingly important for CER with the increasing availability of large and rich observational datasets.

Although several projects developed guidance and translation products and organized learning networks, investigators reported that opportunities remain to improve the availability, usability, and dissemination of new methods. For example, software to support new analytic approaches can be difficult to find or apply to new datasets and research questions. Some investigators suggested that a central repository of information would make programming code easier to find and to apply to various datasets and research questions.

Conclusions

CER will benefit from ongoing methods work to improve applications to emerging CER data resources. Given the considerable investment in data infrastructure, the methods development field can benefit from additional efforts to educate researchers about the availability of new data sources and about how best to select and properly apply methods to fit their research questions and data sources.

Acknowledgements

We are grateful for the funding provided by the Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation.

Creative Commons License

Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License.

Share

COinS
 
Jun 7th, 1:15 PM Jun 7th, 2:45 PM

Lessons from CER Methods Development Projects Funded Under the Recovery Act

Introduction

The American Recovery and Reinvestment Act of 2009 (ARRA) directed nearly $18.2 million to comparative effectiveness research (CER) methods development to foster the production of rigorous and unbiased research and to disseminate the evidence to patients, clinicians, and other decision-makers. To help inform future investments in CER methods, we describe the ARRA-funded methods projects, identify project facilitators and barriers, and discuss the alignment of topics studied with published methods development priorities.

Methods

Twenty-three projects had a primary focus on methods development. To identify project facilitators, and barriers we (1) interviewed 15 investigators and held a webinar discussion with three other investigators and (2) analyzed results from an investigator survey. We identified methods development priorities from a variety of sources released before and during the funding period. Using project proposals and interview notes, we assessed which methods priorities were addressed by the funded projects.

Findings

Projects studied one or more aspects of methods research including comparability of interventions, generalizability of findings, outcomes measurement, data analysis, and dissemination of methods knowledge. Topics included methods that address confounding and bias; Bayesian and adaptive designs; standardization and validation of patient-reported outcomes; methods to identify heterogeneous treatment effects; methods for combining data from randomized and observational studies; and simulation modeling.

Projects directly addressed 12 of 18 identified priority topics. Several projects addressed topics that can help inform methods priority topics. For example, two projects studied Bayesian methods, highly relevant to the priority topic that calls for development of guidance on use of Bayesian methods in CER.

Common project challenges cited by investigators included lack of access to data with sufficient detail, data limitations (missing data or lack of standardization of data elements, for example, in electronic medical records), and delays with data access.


Key Lessons

Although funded projects explored many identified priority topics, investigators noted that much work remains. For example, observational data analysis methods (such as controlling for confounding, risk-adjustment, indirect treatment comparisons, and standardization of patient-reported outcomes) will become increasingly important for CER with the increasing availability of large and rich observational datasets.

Although several projects developed guidance and translation products and organized learning networks, investigators reported that opportunities remain to improve the availability, usability, and dissemination of new methods. For example, software to support new analytic approaches can be difficult to find or apply to new datasets and research questions. Some investigators suggested that a central repository of information would make programming code easier to find and to apply to various datasets and research questions.

Conclusions

CER will benefit from ongoing methods work to improve applications to emerging CER data resources. Given the considerable investment in data infrastructure, the methods development field can benefit from additional efforts to educate researchers about the availability of new data sources and about how best to select and properly apply methods to fit their research questions and data sources.