Boschloo's test

Boschloo's test is a statistical hypothesis test for analysing 2x2 contingency tables. It examines the association of two Bernoulli distributed random variables and is a uniformly more powerful alternative to Fisher's exact test. It was proposed in 1970 by R. D. Boschloo.[1]

Setting

A 2x2 contingency table visualizes ${\displaystyle n}$ independent observations of two binary variables ${\displaystyle A}$ and ${\displaystyle B}$:

${\displaystyle {\begin{array}{c|cc|c}&B=1&B=0&{\mbox{Total}}\\\hline A=1&x_{11}&x_{10}&n_{1}\\A=0&x_{01}&x_{00}&n_{0}\\\hline {\mbox{Total}}&s_{1}&s_{0}&n\\\end{array}}}$

The probability distribution of such tables can be classified into three distinct cases.[2]

1. The row sums ${\displaystyle n_{1},n_{0}}$ and column sums ${\displaystyle s_{1},s_{0}}$ are fixed in advance and not random.
   Then all ${\displaystyle x_{ij}}$ are determined by ${\displaystyle x_{11}}$. If ${\displaystyle A}$ and ${\displaystyle B}$ are independent, ${\displaystyle x_{11}}$ follows a hypergeometric distribution with parameters ${\displaystyle n,n_{1},s_{1}}$:
   ${\displaystyle x_{11}\sim {\mbox{Hypergeometric}}(n,n_{1},s_{1})}$.
2. The row sums ${\displaystyle n_{1},n_{0}}$ are fixed in advance but the column sums ${\displaystyle s_{1},s_{0}}$ are not.
   Then all random parameters are determined by ${\displaystyle x_{11}}$ and ${\displaystyle x_{01}}$ and ${\displaystyle x_{11},x_{01}}$ follow a binomial distribution with probabilities ${\displaystyle p_{1},p_{0}}$:
   ${\displaystyle x_{11}\sim B(n_{1},p_{1})}$
   ${\displaystyle x_{01}\sim B(n_{0},p_{0})}$
3. Only the total number ${\displaystyle n}$ is fixed but the row sums ${\displaystyle n_{1},n_{0}}$ and the column sums ${\displaystyle s_{1},s_{0}}$ are not.
   Then the random vector ${\displaystyle (x_{11},x_{10},x_{01},x_{00})}$ follows a multinomial distribution with probability vector ${\displaystyle (p_{11},p_{10},p_{01},p_{00})}$.

Fisher's exact test is designed for the first case and therefore an exact conditional test (because it conditions on the column sums). The typical example of such a case is the Lady tasting tea: A lady tastes 8 cups of tea with milk. In 4 of those cups the milk is poured in before the tea. In the other 4 cups the tea is poured in first. The lady tries to assign the cups to the two categories. Following our notation, the random variable ${\displaystyle A}$ represents the used method (1 = milk first, 0 = milk last) and ${\displaystyle B}$ represents the lady's guesses (1 = milk first guessed, 0 = milk last guessed). Then the row sums are the fixed numbers of cups prepared with each method: ${\displaystyle n_{1}=4,n_{0}=4}$. The lady knows that there are 4 cups in each category, so will assign 4 cups to each method. Thus, the column sums are also fixed in advance: ${\displaystyle s_{1}=4,s_{0}=4}$. If she is not able to tell the difference, ${\displaystyle A}$ and ${\displaystyle B}$ are independent and the number ${\displaystyle x_{11}}$ of correctly classified cups with milk first follows the hypergeometric distribution ${\displaystyle {\mbox{Hypergeometric}}(8,4,4)}$.

Boschloo's test is designed for the second case and therefore an exact unconditional test. Examples of such a case are often found in medical research, where a binary endpoint is compared between two patient groups. Following our notation, ${\displaystyle A=1}$ represents the first group that receives some medication of interest. ${\displaystyle A=0}$ represents the second group that receives a placebo. ${\displaystyle B}$ indicates the cure of a patient (1 = cure, 0 = no cure). Then the row sums equal the group sizes and are usually fixed in advance. The column sums are the total number of cures respectively disease continuations and not fixed in advance.

An example for the third case can be constructed as follows: Simultaneously flip two distinguishable coins ${\displaystyle A}$ and ${\displaystyle B}$ and do this ${\displaystyle n}$ times. If we count the number of results in our 2x2 table (1 = head, 0 = tail), we neither know in advance how often coin ${\displaystyle A}$ shows head or tail (row sums random), nor do we know how often coin ${\displaystyle B}$ shows head or tail (column sums random).

Test hypothesis

The null hypothesis of Boschloo's one-tailed test (high values of ${\displaystyle x_{1}}$ favor the alternative hypothesis) is:

${\displaystyle H_{0}:p_{1}\leq p_{0}}$

The null hypothesis of the one-tailed test can also be formulated in the other direction (small values of ${\displaystyle x_{1}}$ favor the alternative hypothesis):

${\displaystyle H_{0}:p_{1}\geq p_{0}}$

The null hypothesis of the two-tailed test is:

${\displaystyle H_{0}:p_{1}=p_{0}}$

There is no universal definition of the two-tailed version of Fisher's exact test.[3] Since Boschloo's test is based on Fisher's exact test, a universal two-tailed version of Boschloo's test also doesn't exist. In the following we deal with the one-tailed test and ${\displaystyle H_{0}:p_{1}\leq p_{0}}$.

Boschloo's idea

We denote the desired significance level by ${\displaystyle \alpha }$. Fisher's exact test is a conditional test and appropriate for the first of the above mentioned cases. But if we treat the observed column sum ${\displaystyle s_{1}}$ as fixed in advance, Fisher's exact test can also be applied to the second case. The true size of the test then depends on the nuisance parameters ${\displaystyle p_{1}}$ and ${\displaystyle p_{0}}$. It can be shown that the size maximum ${\displaystyle \max \limits _{p_{1}\leq p_{0}}{\big (}{\mbox{size}}(p_{1},p_{0}){\big )}}$ is taken for equal proportions ${\displaystyle p=p_{1}=p_{0}}$[4] and is still controlled by ${\displaystyle \alpha }$.[1] However, Boschloo stated that for small sample sizes, the maximal size is often considerably smaller than ${\displaystyle \alpha }$. This leads to an undesirable loss of power.

Boschloo proposed to use Fisher's exact test with a greater nominal level ${\displaystyle \alpha ^{*}>\alpha }$. Here, ${\displaystyle \alpha ^{*}}$ should be chosen as large as possible such that the maximal size is still controlled by ${\displaystyle \alpha }$: ${\displaystyle \max \limits _{p\in [0,1]}{\big (}{\mbox{size}}(p){\big )}\leq \alpha }$. This method was especially advantageous at the time of Boschloo's publication because ${\displaystyle \alpha ^{*}}$ could be looked up for common values of ${\displaystyle \alpha ,n_{1}}$ and ${\displaystyle n_{0}}$. This made performing Boschloo's test computationally easy.

Test statistic

The decision rule of Boschloo's approach is based on Fisher's exact test. An equivalent way of formulating the test is to use the p-value of Fisher's exact test as test statistic. Fisher's p-value is calculated from the hypergeometric distribution (for ease of notation we write ${\displaystyle x_{1},x_{0}}$ instead of ${\displaystyle x_{11},x_{01}}$):

${\displaystyle p_{F}=1-F_{{\mbox{Hypergeometric}}(n,n_{1},x_{1}+x_{0})}(x_{1}-1)}$

The distribution of ${\displaystyle p_{F}}$ is determined by the binomial distributions of ${\displaystyle x_{1}}$ and ${\displaystyle x_{0}}$ and depends on the unknown nuisance parameter ${\displaystyle p}$. For a specified significance level ${\displaystyle \alpha ,}$ the critical value of ${\displaystyle p_{F}}$ is the maximal value ${\displaystyle \alpha ^{*}}$ that satisfies ${\displaystyle \max \limits _{p\in [0,1]}P(p_{F}\leq \alpha ^{*})\leq \alpha }$. The critical value ${\displaystyle \alpha ^{*}}$ is equal to the nominal level of Boschloo's original approach.

Modification

Boschloo's test deals with the unknown nuisance parameter ${\displaystyle p}$ by taking the maximum over the whole parameter space ${\displaystyle [0,1]}$. The Berger & Boos procedure takes a different approach by maximizing ${\displaystyle P(p_{F}\leq \alpha ^{*})}$ over a ${\displaystyle (1-\gamma )}$ confidence interval of ${\displaystyle p=p_{1}=p_{0}}$ and adding ${\displaystyle \gamma }$.[5] ${\displaystyle \gamma }$ is usually a small value such as 0.001 or 0.0001. This results in a modified Boschloo's test which is also exact.[6]

Comparison to other exact tests

All exact tests hold the specified significance level but can have varying power in different situations. Mehrotra et al. compared the power of some exact tests in different situations.[6] The results regarding Boschloo's test are summarized in the following.

Modified Boschloo's test

Boschloo's test and the modified Boschloo's test have similar power in all considered scenarios. Boschloo's test has slightly more power in some cases, and vice versa in some other cases.

Fisher's exact test

Boschloo's test is by construction uniformly more powerful than Fisher's exact test. For small sample sizes (e.g. 10 per group) the power difference is large, ranging from 16 to 20 percentage points in the regarded cases. The power difference is smaller for greater sample sizes.

Exact ${\displaystyle Z}$-Pooled test

This test is based on the test statistic

${\displaystyle Z_{P}(x_{1},x_{0})={\frac {{\hat {p}}_{1}-{\hat {p}}_{0}}{\sqrt {{\tilde {p}}(1-{\tilde {p}})({\frac {1}{n_{1}}}+{\frac {1}{n_{0}}})}}},}$

where ${\displaystyle {\hat {p}}_{i}={\frac {x_{i}}{n_{i}}}}$ are the group event rates and ${\displaystyle {\tilde {p}}={\frac {x_{1}+x_{0}}{n_{1}+n_{0}}}}$ is the pooled event rate.

The power of this test is similar to that of Boschloo's test in most scenarios. In some cases, the ${\displaystyle Z}$-Pooled test has greater power, with differences mostly ranging from 1 to 5 percentage points. In very few cases, the difference goes up to 9 percentage points.

This test can also be modified by the Berger & Boos procedure. However, the resulting test has very similar power to the unmodified test in all scenarios.

Exact ${\displaystyle Z}$-Unpooled test

This test is based on the test statistic

${\displaystyle Z_{U}(x_{1},x_{0})={\frac {{\hat {p}}_{1}-{\hat {p}}_{0}}{\sqrt {{\frac {{\hat {p}}_{1}(1-{\hat {p}}_{1})}{n_{1}}}+{\frac {{\hat {p}}_{0}(1-{\hat {p}}_{0})}{n_{0}}}}}},}$

where ${\displaystyle {\hat {p}}_{i}={\frac {x_{i}}{n_{i}}}}$ are the group event rates.

The power of this test is similar to that of Boschloo's test in many scenarios. In some cases, the ${\displaystyle Z}$-Unpooled test has greater power, with differences ranging from 1 to 5 percentage points. However, in some other cases, Boschloo's test has noticeably greater power, with differences up to 68 percentage points.

This test can also be modified by the Berger & Boos procedure. The resulting test has similar power to the unmodified test in most scenarios. In some cases, the power is considerably improved by the modification but the overall power comparison to Boschloo's test remains unchanged.

Software

The calculation of Boschloo's test can be performed in following software:

• Packages Exact and exact2x2 of the programming language R
• StatXact

See also

• Fisher's exact test
• Barnard's test

References

1. Boschloo R.D. (1970). "Raised Conditional Level of Significance for the 2x2-table when Testing the Equality of Two Probabilities". Statistica Neerlandica. 24: 1–35. doi:10.1111/j.1467-9574.1970.tb00104.x.
2. Lydersen, S., Fagerland, M.W. and Laake, P. (2009). "Recommended tests for association in 2×2 tables". Statist. Med. 28 (7): 1159–1175. doi:10.1002/sim.3531. PMID 19170020.CS1 maint: multiple names: authors list (link)
3. Martín Andrés, A, and I. Herranz Tejedor (1995). "Is Fisher's exact test very conservative?". Computational Statistics and Data Analysis. 19 (5): 579–591. doi:10.1016/0167-9473(94)00013-9.CS1 maint: multiple names: authors list (link)
4. Finner, H, and Strassburger, K (2002). "Structural properties of UMPU-tests for 2x2 tables and some applications". Journal of Statistical Planning and Inference. 104: 103–120. doi:10.1016/S0378-3758(01)00122-7.CS1 maint: multiple names: authors list (link)
5. Berger, R L, and Boos, D D (1994). "P Values Maximized Over a Confidence Set for the Nuisance Parameter". Journal of the American Statistical Association. 89 (427): 1012–1016. doi:10.2307/2290928. JSTOR 2290928.CS1 maint: multiple names: authors list (link)
6. Mehrotra, D V, Chan, I S F, and Berger, R L (2003). "A cautionary note on exact unconditional inference for a difference between two independent binomial proportions". Biometrics. 59 (2): 441–450. doi:10.1111/1541-0420.00051. PMID 12926729.CS1 maint: multiple names: authors list (link)