RAND researchers recently sent me a letter and an attachment , which they have circulated to others, commenting on CBOs analysis of a recent RAND health IT study. Our analysis is summarized here and the full analysis is here . As I have noted previously, I will occasionally use this blog to respond to critiques of our work, and that is the purpose of this entry.
The RAND study estimated potential savings of approximately $80 billion per year from health IT if it were widely adopted. As CBO concluded in its recent report, however, that $80 billion figure is not an appropriate guide to the effects of legislative proposals aimed at increasing the use of health IT for several reasons. For example, the RAND study attempted to measure the potential impact of the widespread adoption of health IT -- assuming the occurrence of appropriate changes in health care -- rather than the likely impact, which would take account of factors that might impede its effective use. In addition, the RAND study was based solely on empirical studies from the literature that found positive effects for the implementation of health IT systems; it excluded studies of health IT that failed to find favorable results.
Nothing in the RAND letter would cause us to modify our previous conclusions. The letter emphasizes that the RAND study was published in a peer-reviewed journal (Health Affairs ), was implemented with the advice and review of a steering group of experienced and respected professionals, and was carried out with transparency with respect to its methods and assumptions. CBO did not, though, criticize the report for failing to be peer-reviewed, having inappropriate leadership, or lacking transparency. Our concerns are instead based on the substance of the study itself -- especially the questions it was designed to answer -- and perhaps more importantly how it has been used in the policy debate. (Similarly, CBO did not criticize the RAND study for being funded "by companies interested in health information technology." The issue we addressed is instead the analysis itself and how that analysis has been presented in policy circles.)
The letter also argues that CBO did not take account of other possible benefits of adopting health IT beyond those considered in the RAND report (such as improvements in health and safety), and that those benefits imply that RANDs estimate of savings is conservative. In our paper we specifically identified the sources of savings considered in the RAND study and also described some other possible areas of savings. For example, our paper stated that One significant potential benefit of health IT that has thus far gone relatively unexamined involves its role in research on the comparative effectiveness of medical treatments and practices. However, we also recognized that obtaining those benefits would require a number of steps beyond merely a greater diffusion of health IT. For example, it would require creating databases, commissioning studies, and then most importantly using the results of those studies to alter the practice of medicine. In other words, much more would have to change to obtain these benefits than just the adoption level of IT.
Another area of confusion appears to be the question of the appropriate counterfactual: that is, what are the scenarios one is trying to compare when evaluating the impact of health IT? For our work at CBO, we compare what would happen under a proposed piece of legislation with what would happen if current laws remained in place. RANDs $80 billion savings estimate, by contrast, is generated by comparing the level of adoption in 2004 with the level of adoption attained in a future year if IT were to follow a diffusion pattern that has been observed in other industries. Even if we agreed with other assumptions and calculations that led RAND to the $80 billion estimate (which we do not), we believe that health IT will continue to diffuse under current law, so that the appropriate calculation for our purposes is to compare savings under a new law with savings under the current-law pattern of diffusion. The point matters because health IT in the future will almost certainly have attained greater adoption rates than had occurred in 2004. Indeed, in a different RAND study, the authors compared savings under a baseline of continued adoption with a subsidy program that would speed adoption by 50 percent. That studys estimate of the impact of the subsidy program follows more closely the approach that CBO would take.
Our published analysis covers, in more detail, the other technical reasons why we believe that the RAND study's estimates overstate the cost savings from policy interventions to increase the adoption of health IT.
Finally, we very much appreciate and value the input that outside reviewers provide as we prepare our reports. As I have noted in a previous post , the fact that someone is listed as a reviewer of our paper in no way implies that the reviewer agrees with the analysis in the paper.