In an article titled “Association Between State Laws Facilitating Pharmacy Distribution of Naloxone and Risk of Fatal Overdose,” in the June 2019 issue of JAMA: Internal Medicine, Abouk et al. claim state laws granting direct authority to pharmacists to provide naloxone are associated with greater declines in opioid-related mortality than other laws facilitating access to naloxone. Furthermore, Abouk et al. claim laws other than these direct access laws are not associated with declines in opioid-related mortality. Their study - which aims to estimate the effect of state naloxone distribution laws on mortality - addresses an important, timely, question given the widespread interest in identifying mechanisms to reduce opioid-related mortality and the contentious debate over the regulation of naloxone access. In the months following publication, the article received significant attention from health experts and the media, including the blog of the Director of the NIH, Reuters, Vox, and Kaiser Health News, amongst others.
Curious about the evidence behind these claims, Stephanie Roses and I took a closer look at the article shortly after its publication. During our examination, we noticed some issues with the interpretation of some of the empirical results. In particular, the empirics largely did not test the predictions of the authors and the evidence at times appeared to be inconsistent with the authors’ conclusions.
In light of these concerns, we submitted a letter to JAMA: Internal Medicine shortly after the article’s publication. While the letter was rejected for publication, I am posting our letter here for those interested:
Abouk et al.’s June 2019 JAMA: Internal Medicine article makes valuable contributions to the debate regarding the current opioid crisis.1 Nevertheless, we show that incongruencies between hypotheses and methods cause the authors to prematurely - and perhaps inaccurately - conclude that laws granting direct pharmacist authority to provide naloxone (DPALs) are associated with greater mortality declines than other naloxone access laws (NALs) and that NALs other than DPALs “appear not to be associated with decreases or increases in mortality.”
Abouk et al. propose two hypotheses. First, they hypothesize “NALs granting direct authority to pharmacists to provide naloxone will have the greatest potential for reducing fatal overdoses.” Second, they hypothesize “laws that facilitate greater distribution of naloxone will result in greater decreases in opioid mortality.” These hypotheses have two important features: both make predictions about the differences between effects of various laws and both are “directional” – predicting signs rather than magnitudes of effects and their differences.
Despite this, the authors do not test hypotheses regarding differences or directions of effects. Instead, they use two-tailed tests assessing whether the magnitude of a law’s effect is non-zero. From these analyses, they conclude the significant and large effects of DPALs exceed the non-significant and small effects of other NALs. However, the fact that one effect is significant while another effect is not significant does not imply the two effects are significantly different. Furthermore, testing if an effect is non-zero is not testing if it is positive.
Addressing these incongruencies between theory and empirics may produce substantively different conclusions. For example, the use of two-tailed tests leads to the authors finding “little evidence of association [with opioid mortality] for indirect authority to dispense [laws] or other [non-DPAL] NALs.” This conclusion is based on two-tailed p-values of 0.09 and 0.17 for the effect of these NAL types. However, when using one-tailed p-values to test whether these laws reduce mortality, the former effect is significant (p=0.045) and the latter is nearly significant (p=0.085).
Abouk et al. claim states authorizing direct pharmacist naloxone dispensing experience lower opioid-related mortality than other states. They also claim non-DPAL NALs are unassociated with mortality reductions. While their data contradict the latter claim, their analyses do not provide direct evidence for the former. Luckily, testing for differences in effects is trivial with the raw data and we encourage such analyses to improve evidence-based efforts in combatting the opioid crisis.
