Win-probabilities for comparing two binary outcomes

Abstract

This article considers the problem of choosing between two treatments that have binary outcomes with unknown success probabilities p1 and p2. The choice is based upon the information provided by two observations X1 ∼ B(n1, p1) and X2 ∼ B(n2, p2) from independent binomial distributions. Standard approaches to this problem utilize basic statistical inference methodologies such as hypothesis tests and confidence intervals for the difference p1 − p2 of the success probabilities. However, in this article the analysis of win-probabilities is considered. If X*1 represents a potential future observation from Treatment 1 while X*2 represents a potential future observation from Treatment 2, win-probabilities are defined in terms of the comparisons of X*1 and X*2. These win-probabilities provide a direct assessment of the relative advantages and disadvantages of choosing either treatment for one future application, and their interpretation can be combined with other factors such as costs, side-effects, and the availabilities of the two treatments. In this article, it is shown how confidence intervals for the win-probabilities can be constructed, and examples of their use are provided. Computer code for the implementation of this new methodology is available from the authors. © 2017 Taylor & Francis Group, LLC.