How Informative Are Risk-Adjusted Hospital Quality Measures?

03/13/2024
This figure is a line graph titled, Mortality Rates and Hospital Closures. The y-axis is labeled change in mortality rate, percentage points. It ranges from negative 1 to positive 1, increasing in increments of 0.5. The x-axis is labeled year relative to hospital closures. It ranges from negative 3 to positive 2, increasing in increments of 1. There is a dotted vertical line at 0 labeled, year of hospital closure.  The graph displays two lines, one labeled "closure of a high-quality hospital" and the other labeled "closure of a low-quality hospital," both showing the impact of hospital closures on a particular metric over time. The line representing the closure of a high-quality hospital remains at 0 until one year before the hospital closure (indicated as -1 on the x-axis). At the time of closure (0 on the x-axis), there is a significant jump to approximately 0.85. The line then slightly declines, reaching around 0.7 two years after the closure. In contrast, the line representing the closure of a low-quality hospital starts at -0.5 and fluctuates around 0 prior to the hospital closure. One year after the closure (1 on the x-axis), the line dips and continues to decline, reaching about -0.75 two years after the closure. The source line reads, Source: researchers’ calculations using Medicare data.

Hospital quality indicators are intended to provide public information about differences in the quality of care across hospitals. The measures — such as 30-day mortality rates and 30-day hospitalization costs for admitted patients — are adjusted for the characteristics of each hospital’s patient population. In principle, these risk-adjusted indicators are not affected if a hospital treats especially unhealthy patients. If the risk adjustment is inadequate, however, the quality measures could confound patient health status with hospital quality. 

In Are Hospital Quality Indicators Causal? (NBER Working Paper 31789), Amitabh Chandra, Maurice Dalton, and Douglas O. Staiger assess risk-adjusted hospital quality measures. Specifically, they ask how well these measures predict changes in outcomes for Medicare beneficiaries when a local hospital closure induces patients to switch to other hospitals. If hospital quality indicators capture meaningful variation in quality, the effect of a closure should depend on the difference between the quality measures of the closing hospital and of the remaining hospitals in the area.

The analysis focuses on outcomes among fee-for-service patients who were admitted to a hospital due to heart attacks, hip fractures, pneumonia, congestive heart failure, or strokes between 1992 and 2015. It evaluates four hospital quality measures that are based on different outcomes: 30-day mortality, 30-day readmissions, 30-day inpatient hospital costs, and length of stay.

The figure shows that, when a hospital closes, mortality changes in the direction that risk-adjusted 30-day mortality rates predict. When quality indicators predict that a hospital closure will worsen health outcomes, mortality rises at the time of the closure. In contrast, when quality measures predict that a hospital closure will improve health outcomes, mortality falls. Similar patterns emerge for all four measures of hospital quality. 

How well do the quality indicators predict the magnitude of the changes in outcomes? Risk-adjusted quality measures based on health outcomes perform better than those that are based on resource utilization. The risk-adjusted indicators overestimate the impacts of closures on mortality and readmissions by about 7 percent and overstate effects on costs and length of stay by around 40 percent.

Risk-adjusted quality indicators generally perform better than their non-risk-adjusted counterparts. While unadjusted quality measures accurately predict whether outcomes will improve or worsen after a closure, they overestimate changes by more than the risk-adjusted indicators. They overstate effects on mortality by 40 percent, readmissions by 30 percent, and costs and length of stay by 50 percent.

The researchers highlight that closing higher-quality hospitals may harm patients who would otherwise have used them, while closing lower-quality hospitals may help such patients. 

— Robin McKnight


The researchers acknowledge financial support from the National Institutes of Health through a Pilot grant and a P01 grant to Dartmouth and the NBER (P30AG012810 and P01AG019783).