The Athena Class of Risk-Limiting Ballot Polling AuditsThanks: This material is based upon work supported in part by NSF Awards 2015253 and 1421373

We refer to audits where decisions are taken after each ballot draw as ballot-by-ballot or B2 audits. The general audit, however, is a round-by-round or R2 audit where, in the $j^{th}$ round, some ballots are drawn, after which a decision is taken regarding whether to (a) stop the audit and declare the election outcome correct, (b) stop the audit and go to a manual recount, or (c) draw the $(j+1)^{th}$ round. A B2 audit is a special case of the R2 audit, when a single ballot is drawn in each round.

There are two ways to apply B2 audit rules to an R2 audit. Consider a total of $n_{j}$ ballots drawn after the $j^{th}$ round, of which $k_{j}$ are for the reported winner.

• •

  End-of-round: In this application, the B2 stopping rule for $k_{j}$ winner ballots in a sample of $n_{j}$ ballots determines whether the audit will stop.

• •

  Selection-ordered-ballots: In this application, ballot order is recorded and the B2 stopping condition is tested $\forall n\leq n_{j}$. The audit stops if the B2 condition is satisfied for any value of $n\leq n_{j}$.

Selection-ordered-ballots is generally more efficient than end-of-round as a means of applying B2 rules to R2 audits, but requires the significant additional effort of preserving enough information to be able to recreate the subtotals of winner ballots in selection order. End-of-round relies only on the tallies and does not require selection order. As our paper shows, neither is a most efficient R2 stopping rule.

One may view the problem we address as lying somewhere between (a) the problem solved by Neyman-Pearson [12]: derive a single-use binary hypothesis testing rule satisfying certain error criteria, and (b) the problem solved by Wald [15]: derive a stopping condition for sequential sampling, satisfying certain error criteria, where the condition is tested draw-by-draw. We address the problem of sequential sampling in rounds, where the condition is tested after multiple draws.

Our contributions are as follows:

1. 1.

   We derive analytical expressions for the risk and probability of stopping, given the history of rounds and the margin for the Bravo audit. Treating the B2 Bravo audit as an R2 audit with $n_{j}=j$, we verify that the expressions we derive predict the stopping percentiles originally obtained by Lindeman et al. using Bravo simulations [5, Table 1]. The average of the absolute value of the fractional difference between our results and those of [5] is 0.13%. The largest difference has value $190$ ballots, corresponding to a fractional difference of 0.41 %, in the estimate of the expected number of ballots drawn for a margin of 1%. This difference could be due to small inaccuracies in our computational approach (such as rounding off errors or the finiteness of summations involved in the computations) or the finiteness of the number of simulations used to generate the results of [5]. Our approach is easily extended to audits with stopping conditions that are monotone increasing in the number of ballots for the announced winner, such as Bayesian audits. The code for computing these expressions is available as a MATLAB library, released as open-source under the MIT License [13].

2. 2.

   We present the Athena class of R2 stopping rules for audits Minerva and Athena and prove that, if the round schedule is pre-determined (before the audit begins), Minerva and Athena are both risk-limiting and at least as efficient as the corresponding end-of-round Bravo stopping rule. Another audit from the Athena class, Metis, is out of scope for this draft.

3. 3.

   We provide experimental results and software to support the use of the proposed audits:

   • •

     To illustrate the efficiency improvements, we compute (without simulations, using the derived analytical expressions), for each state in the 2016 US Presidential election, risk limit $\alpha=0.1$ and a stopping probability of 0.9, first round sizes for end-of-round Bravo and Athena. We find that Athena requires about half the number of ballots, across all margins.

   • •

     We compute first round sizes for selection-ordered-ballots Bravo and find that Athena requires about $15-29\%$ fewer ballots for the data of the 2016 US Presidential election, with the improvement being better for smaller margins. Thus Athena is more efficient than selection-ordered-ballots Bravo and does not require the additional book-keeping of recording selection ballot order.

   • •

     We present the results of simulations supporting our predictions of first round stopping probabilities and the risk-limiting properties of Athena.

   • •

     Our code for the audits is available as MATLAB and Python libraries [17, 13, 6]. All code is released as open source under the MIT license.

   This contribution is important because a number of states have undertaken ballot polling pilots in the last year and plan to use ballot polling audits in November 2020. We hope that our results and code can help developers of auditing software. We also note that, in many scenarios, ballot comparison or batch comparison audits could be more feasible. One may also consider combinations of ballot comparison and ballot polling audits, such as described in [8].

4. 4.

   The Athena class of R2 stopping rules is a class of B2 rules when round size is one. Of theoretical interest, we prove that B2 Minerva (round size one) has the same stopping rule as B2 Bravo, as does B2 Athena for some values of its parameters.

We do not claim that the audits of the Athena class are the most efficient R2 audits with zero error of the second kind ($\beta=0$). The problem of finding the most efficient R2 audits is open.

Unlike the SPRT and other Martingale-based approaches, the stopping rules for audits of the Athena class use information about the history of round sizes. For this reason the stopping condition for rounds other than the first one does not depend only on the cumulative sample size and number of winner ballots drawn, but also on the history of individual round sizes.

We do not address some simple extensions of our work in this paper. For example, we do not consider audits with a limit on the total number of ballots drawn in the polling audit (if the audit fails to stop, a full sequential hand count would follow). Were we to do so, we could provide audits with larger stopping probabilities given the same risk limit.

This paper focuses exclusively on Bravo. However, we expect that our results extend to other risk-limiting SPRTs with $\beta=0$ and a stopping condition that is monotonic in the number of winner ballots drawn; examples include Bayesian audits.

Section 2 presents the model and related work. Section 3 motivates the problem with an example demonstrating that the application of B2 rules to an R2 audit results in inefficiencies. Section 4 introduces the Athena class of audits with examples and provides insight into why the audits are risk-limiting and more efficient than either R2 application of B2 Bravo. Section 5 describes the analytical approaches for computing probabilities for multiple-round audits. Section 6 presents the Minerva and Athena audits, and Section 7 presents rigorous claims of their risk-limiting and efficiency properties in the form of Theorems and Lemmas. Section 8 presents experimental results. Section 9 concludes. Proofs are in the Appendix.

We gratefully acknowledge comments on an early draft by: Matthew Bernhard, Amanda Glazer, Mark Lindeman, Jake Spertus, Mayuri Sridhar, Philip B. Stark, Damjan Vukcevic.

This version is updated from the previous one to reflect only a couple, and not all, of their valuable suggestions. Importantly, this draft notes that the risk-limiting property is proven only when round sizes are pre-determined. We have also made some of the changes, including some improvements to our notation, and the inclusion of the number of distinct ballots in tables listing the number of ballots required for a first round with 90% stopping probability for 2016 tallies. We plan to soon further update the manuscript to include estimated first round sizes for the 2021 tallies and address all other suggestions made by the reviewers.

This research was sponsored in part by NSF Awards 2015253 and 1421373.

We model the audit as a binary hypothesis test:

Null hypothesis $H_{0}$: The election outcome is the closest possible incorrect outcome: $w\neq w_{a}$ and the fractional vote count for $w_{a}$ is $\frac{1}{2}$. In particular, if the total number of valid votes is even, the election is a tie. If the total number of valid votes is odd, the margin is one in favor of $\ell_{a}$. In this case, we assume that the number of valid votes is large enough that the fractional vote count is sufficiently close to $\frac{1}{2}$. Henceforth, we will refer to both cases as being represented by a fractional vote count of $\frac{1}{2}$.

Alternate hypothesis $H_{a}$: The election outcome is correct: $w=w_{a}$ and the fractional vote count is as announced.

After each round the test $\mathcal{A}$ takes $X$ as input and outputs one of the following:

• •

  Correct: The test estimates that $w=w_{a}$ and the audit should stop.

• •

  Incorrect: The test estimates that $w\neq w_{a}$. We stop drawing votes and proceed to perform a complete hand count to determine $w$.

• •

  Undetermined (draw more samples): We need to draw more ballots to improve the estimate.

When the audit stops, it can make one of two kinds of errors:

1. 1.

   Miss: A miss occurs when $w\neq w_{a}$ but the audit misses this, and outputs Correct. We denote by $P_{M}$ the probability of a miss:

   $$P_{M}=Pr[\mathcal{A}(X)=\textit{Correct}\mid H_{0}]$$

   $P_{M}$ is the risk in risk limiting audits and the Type I error of the test.

2. 2.

   Unnecessary Hand Count: Similarly, if $w=w_{a}$, but the audit estimates that a hand count must follow, the hand count is unnecessary. We denote the probability of an unnecessary hand count by $P_{U}$:

   $$P_{U}=Pr[\mathcal{A}(X)=\textit{Incorrect}\mid H_{a}]$$

   $P_{U}$ is the Type II error.

Like the Bravo audit, this paper focuses on tests with $P_{U}=0$. The risk, on the other hand, is an important (generally) non-zero value characterizing the quality of the audit.

A risk-limiting audit (RLA) with risk limit $\alpha$—as described by, for example, Lindeman and Stark [4]—is one for which the risk is smaller than $\alpha$ for all possible (unknown) true tallies in the election. For convenience when we compare audits, we refer to this audit as an $\alpha$-RLA.

There are many audits that would satisfy the $\alpha$-RLA criterion, and not all would be desirable. For example, the constant audit which always outputs Incorrect always requires a hand count and is risk-limiting with $P_{M}=0<\alpha$, $\forall\alpha$, $\forall p$. However, $P_{U}=1$, and the audit examines all votes each time; this is undesirable.

An example of an $\alpha$-RLA with $P_{U}=0$ and drawing fewer ballots is the B2 Bravo audit [5] which specifies round size increments of one.

$\sigma(k,p,n)$ is the likelihood ratio of the drawn sequence $X$. The B2 $(\alpha,p)$-Bravo audit is an SPRT [15] with:

$H_{0}$, the null hypothesis: the election is a tie

$H_{a}$, the alternate hypothesis: the fractional tally for the winner is $p$.

Implicit in Definition 2 is the point that a sequence $X$ is tested only if it has not previously satisfied the test. If $\mathcal{A}(X_{*})~=~Correct$ for some sequence $X_{*}$, all extensions $X_{*}^{+}$ of $X_{*}$ are defined as having passed the test. Determining the stopping condition by evaluating $\mathcal{A}(X_{*}^{+})$ does not satisfy the assumptions of the test, and the properties of the test do not necessarily apply. As we shall see in Section 3, this is relevant to end-of-round Bravo. In fact, it is relevant to end-of-round applications of any B2 audit that is an SPRT.

B2 Bravo is a most efficient test given the hypotheses (if the stopping condition is satisfied exactly every time). Vora shows [14] that B2 $(\alpha,p)$-Bravo is an $\alpha$-RLA because it assumes a tie for $H_{0}$, which is the wrong election outcome that is hardest to distinguish from the announced one, and hence defines the worst-case risk [14].

Other approaches, such as Rivest’s CLIP Audit [9], improve on B2 Bravo’s efficiency subject to certain constraints (namely, of $\beta$ as defined in [9]).

A prototype of Athena mirrored the explicit risk allocation found in Stark’s Conservative Statistical Post-Election Audits [11] before ballots are examined for the audit, a list of increasing rounds $(n_{1},n_{2},...,n_{j})$, and a list of corresponding risks $(\alpha_{1},\alpha_{2},...,\alpha_{j})$ are generated. Dispensing with auditor flexibility in favor of a predetermined list of rounds and corresponding risks facilitated the investigation of the convolution procedure that underlies a fundamental improvement of Athena over Bravo.

There is a line of work on group sequential testing [16, 3, 1, 2] but all results that we were able to find begin with the assumption of a normal distribution and cannot be directly applied to the considered scenario of auditing elections.

We propose the Athena class of audits, which use the tails of the probability distribution functions to define the stopping condition. Here we provide an informal description of the simplest of the Athena class, the Minerva audit.

We will prove in Section 7 that the Minerva stopping condition is monotonic increasing as a function of $k_{1}$; one may understand this informally as follows. As explained in Section 3, $\sigma(k_{1},p,n_{1})$ is monotone increasing with $k_{1}$. The Minerva ratio $\tau_{1}(k_{1},p,n_{1})$ is a weighted average of the values of $\sigma(k,p,n_{1})$ for $k\geq k_{1}$ and is also, hence, monotone increasing with $k_{1}$. If a sample with $32$ winner ballots of a total of $50$ ballots were to satisfy the stopping condition, so would all samples with $k_{1}\geq 32$.

Smaller values of $k_{1}$ are associated with larger tails in both curves of Figure 2; and the tails denote the stopping probability (given $H_{a}$, the translucent blue tail of the solid blue curve) and the risk (given $H_{0}$, the solid red tail of the dashed red curve). The smaller the value of $k_{1}$, hence, the larger the associated stopping probability and risk. We could simply choose the smallest acceptable $k_{1}$ (denoted as $k_{min,1}$) so that the associated risk is $\alpha$, but then we could not plan to ever go to another round because we would have exhausted the risk budget in the first round. For the Minerva audit, we choose $k_{1}$ so that the risk is no larger than $\alpha$ times the stopping probability. This allows us to go on to an indefinite number of rounds.

Let $R_{j}$ and $S_{j}$ be informally defined as follows (more formal definitions follow in Sections 6 and 5):

and

We define $R_{j}$ and $S_{j}$ more carefully in section 7 and describe how to compute these values in section 5. Loosely speaking, they denote the risk associated with the $j^{th}$ round ($R_{j}$) and the stopping probability of the $j^{th}$ round ($S_{j}$) respectively.

The Minerva stopping condition is:

If $R$ is the risk of the audit and $S$ its stopping probability,

because $S$, the stopping probability of the audit, is no larger than $1$.

In other words, the total risk of the audit is the sum of the risks of each individual round. The stopping condition ensures that each of these risks is no larger than $\alpha$ times the corresponding stopping probability. Adding all the risks gives us the total risk, which is no larger than $\alpha$ times the total stopping probability. Because the total stopping probability cannot be larger than one, the total risk cannot be larger than $\alpha$, and Minerva is risk-limiting.

In this section, we further examine the audit of our previous examples to understand the behavior of the ratios of Bravo and Minerva, $\sigma(k_{1},p,n_{1})$ and $\tau_{1}(k_{1},p,n_{1})$ respectively.

The reason for Minerva stopping at smaller values of $k_{1}$ is as follows. Consider $k_{1}=32$. While

$\sigma(k_{1},0.75,50)$ can be much larger for larger values of $k_{1}$. The Minerva ratio, $\tau_{1}$, at $k_{1}=32$ is a weighted average of all the values of $\sigma(k_{1},0.75,50)$ for $k_{1}\geq 32$, allowing the larger values of $\sigma(k_{1},0.75,50)$ to “make up” for the smaller ones; in fact, $\tau_{1}=30.356$ for $k_{1}=32$.

In other words, because the end-of-round Bravo ratio increases as $k_{1}$ increases, the weighted average, the Minerva ratio, will always be larger than the end-of-round Bravo ratio except if $k$ is the largest possible number of winner votes, in which case the two ratios will be equal. Equivalently, the Minerva p-value will always be smaller except when $k$ is the largest possible number of winner votes, and the p-values are equal. Thus, Minerva is at least as efficient.

In this section we present an example introducing the Athena audit.

In our experiments we have observed samples satisfying Minerva but not Athena when the election margin is wide, as in our example. Hence, clearly, Athena is not as efficient as Minerva, because it imposes an additional condition. One may think of Minerva as determining whether the election outcome is correct, and Athena determining, in addition, if the election tally is close enough to the announced tally.

Given the B2 $(\alpha,p)$-Bravo test we define the corresponding R2 Minerva test by its stopping condition, which is a comparison test of the ratio of the complementary cdfs of samples that did not satisfy the stopping condition for any previous round. We expect that similar R2 Minerva tests can be defined for other SPRTs with zero error of the second kind whose probability ratio is a monotonic increasing function of $k$, such as Bayesian audits, but do not address these in this paper. Note that the round schedule is predetermined before the audit begins.

Clearly, for $j=0$ and the first round,

In addition to comparing the ratio of complementary cumulative distribution functions as in Minerva, the Athena audit also enforces a lower bound on the probability ratio, $\sigma$, of the B2 $(\alpha,p)$-Bravo test.

Clearly, for $j=0$ and the first round,

We further define the risk and stopping probability associated with each round.

In this section we provide an outline of the claims and proofs.

Using induction on the number of rounds, we prove a couple of interesting properties, including, at the core, that the likelihood ratio of $K_{j}$ (total number of winner ballots) in the $j^{th}$ round is

That is, for $k$ winner ballots in round $j$, when sequences are restricted to those that did not satisfy stopping conditions in previous rounds, the likelihood ratio is simply $\sigma(k,p,n)$, independent of any additional constraints on sequence order and the past or future round schedule. This property leads to the result that the test in the $j^{th}$ round is a comparison test for $k_{j}$.

For the base case, it is easily shown that the likelihood ratio in the first round is $\sigma(k,p,n)$, as there is no previous round:

$\sigma(k,p,n)$ is easily seen to be monotone increasing with $k$ because $p>\frac{1}{2}$.

The induction step proceeds as follows. Suppose the likelihood ratio for the number of winner ballots in round $j$ is $\sigma(k,p,n)$. The ratios used for the stopping conditions of the $j^{th}$ round in Minerva and Athena

and

respectively, are weighted sums of $\sigma(k_{j},p,n)$ for $k_{j}\geq k$. Because $\sigma(k_{j},p,n)$ is monotone increasing with $k_{j}$, the respective stopping conditions are monotone increasing with $k$, and can be expressed as comparison tests for $k$.

Athena has two conditions. The second one is a comparison test for the likelihood ratio, and hence also equivalent to a comparison test for the number of winner ballots. The overall comparison test is the stricter of the two, and is also a comparison test. Thus the stopping condition for round $j$ is a test of the form $k\geq k_{min,j}$ for a $k_{min,j}$ that depends on the audit, its parameters including previous round sizes, the election parameters and the risk limit. As described in section 4, one can use convolution to compute the probability distributions for $k_{j+1}$.

To determine the nature of the likelihood ratio for round $j+1$, we proceed as follows. The likelihood ratio for $k$ in round $j$ is assumed to be $\sigma(k,p,n)$. Hence, the distribution of $k$ in round $j$ given $H_{a}$ is a multiple of $p^{k}(1-p)^{n-k}$, and that given $H_{0}$ is the same multiple of $(\frac{1}{2})^{n}$. The multiplying factor itself is a function of $k$, current and previous round sizes, $p$ and $\alpha$, as it captures the previous comparison tests on the number of winner ballots. On convolution, when one obtains the distributions for round $j+1$, the multiplying factors change, but the one for the distribution given $H_{a}$ is the same as that for the distribution given $H_{0}$. Loosely speaking, the number of ways of obtaining a sequence with $k$ winner ballots in a round of size $n$, given previous round sizes (and hence previous comparison tests for winner ballots), is independent of the hypothesis. Thus the likelihood ratio of $k$ in round $j+1$ remains $\sigma(k,p,n)$.

It is now easy to show that the audits are risk-limiting. The tail (beginning at $k_{min,j}$) of the probability distribution of $K_{j}$ in the $j^{th}$ round conditional on hypothesis $H_{a}$ ($H_{0}$) is the stopping probability (risk) associated with the $j^{th}$ round. Both Minerva and Athena audits require that the tail corresponding to $H_{0}$ (the risk corresponding to round $j$) be no more than $\alpha$ times the tail corresponding to $H_{a}$ (the stopping probability), thus ensuring that the sum of all the risk contributions of the rounds is no more than $\alpha$ times the total stopping probability, and hence no more than $\alpha$.

Both Minerva and Athena are more efficient than end-of-round Bravo. This is because the ratios $\sigma$, $\tau_{j}$ and $\omega_{j}$ are all compared to the same value, $\frac{1}{\alpha}$. For $k$ winner ballots, the ratio for Bravo is $\sigma(k,p,n)$, which is monotone increasing with $k$. The ratios for Athena and Minerva, $\omega_{j}$ and $\tau_{j}$ respectively, are weighted sums of $\sigma(k_{j},p,n)$ for $k_{j}\geq k$ and hence strictly larger, unless $k$ is the largest value with non-zero probability (it would be value of of $k_{min}-1$ for the previous round, plus the size of the current draw), when they are equal. Thus, if a sample satisfies the end-of-round Bravo condition, it also satisfies the conditions on $\omega_{j}$ and $\tau_{j}$. The Athena audit includes a second test, which is a comparison test for $\sigma(k,p,n)$. If $\delta\geq\alpha$, this too is satisfied if the end-of-round Bravo condition is satisfied.

We state the above claims more formally in the rest of this section, and prove them in the Appendix. We also prove that the Minerva and Athena ($\delta\geq\alpha$) stopping conditions, when round increments are specified to be one, are equivalent to the B2 Bravo condition, though p-values are not the same except for samples where the audits stop.

We establish some shorthand notation which will be useful.

For ease of notation, when the audit and its parameters: round schedule $(n_{1},n_{2},\ldots)$, risk limit $\alpha$, fractional vote for the winner $w_{a}$ are fixed, we denote:

Thus $\frac{S_{j}(k_{j})}{R_{j}(k_{j})}$ is the ratio of the complementary cdfs in round $j$ when the number of winner ballots drawn is $k_{j}$, and the sequence did not satisfy the stopping condition in a previous round.

Similarly,

and

and $\frac{s_{j}(k_{j})}{r_{j}(k_{j})}$ is the likelihood ratio of $k_{j}$ winner ballots in round $j$ when the sequence did not satisfy the stopping condition in a previous round.

Note also the following simple observation: