Monday, May 18, 2020

Is the Burning Wheel's difficulty table consistent?

On page 15 of the Burning Wheel Gold Revised we see the table of difficulties shown in Fig. 1. (While "the" is technicaly part of the title, since references to Burning Wheel on its own website do not include "the", I will henceforth dispense with its usage, especially as it's often awkward sounding.) You can get this section of the book for free and see for yourself here. This is a handy-looking table for judging which rolls are of various levels of difficulty. But is it consistent? That is, do all "Easy'' rolls succeed with higher probability than all "Hard'' rolls and likewise, do all "Hard" rolls succeed with higher probability than "Too Hard'' rolls?

Fig. 1: Burning Wheel difficulties

To find out, we'll calculate all the relevant probabilities. In Burning Wheel, you roll a number of six-sided dice equal to the exponent of the skill, which may be increased by various factors. Note that this difficulty table refers to the exponent, not the total number of dice rolled, though. A roll using five dice is often abbreviated 5D. For a black shade skill each die that results in a 4, 5, or 6 is counted as a success. (There are three shades: black, gray, and white, which affect which values on each die count as a success.) If you roll a number of successes equal to the obstacle (abbreviated Ob), then you succeed at the roll. For each die that you roll, you get one success with probability 0.5 and no successes with probability 0.5.

Let's first calculate the probability mass function (pmf) of the number of successes rolled as a function of the number of dice rolled. I've already stated what it is for 1D. Now, let's expand that. Consider the case with two dice. Each die can get a success, so we'll get between 0 and 2 successes. Let's assign the probability of success on a single die as $p=0.5$. To get 0 successes, both dice must fail, which occurs with probability $(1-p)^2 = 0.5^2=0.25$. To get 2 successes, both dice must succeed, which occurs with probability $p^2 = 0.5^2=0.25$ also. All that remains is getting 1 success, which thus must occur with probability $1-0.25-0.25=0.5$. The other way to look at it is that we need one success and one failure, but they can come the two dice in any valid combination. This is occurs with probability $p \cdot (1-p) \cdot n_1$, where $n_1$ is the number of ways to select 1 success die and 1 traitor die (Burning Wheel terminology for a failed die). Since we are assigning all dice as either success or traitor, this is equivalent to picking just the successes (the traitors defined by all those which are not successes). We know that the number of ways to select 1 success die out of 2 dice is 2 choose 1, or $\binom{2}{1} = 2$. Thus, the probability of one success on two dice is,
\binom{2}{1} \cdot p \cdot (1-p) &= 2 \cdot 0.5 \cdot 0.5 \\
&= 0.5,
matching our previous calculation method. We can use this as a prototype for a general formula. In general for a roll of $n$ dice, where the random variable $S$ represents the number of successes, to get $s$ successes we'll have the general formula for the pmf, $f_S(s) = P(S=s)$, as follows.
f_S(s) &= \binom{n}{s} \cdot p^s \cdot (1-p)^{n-s}
Here we select $s$ dice to succeed out of $n$ total, which we can do in $\binom{n}{s}$ different ways. Then those $s$ dice must all succeed, which occurs with probability $p^s$, and the remaining $n-s$ dice must all fail, which occurs with probability $(1-p)^{n-s}$. This is known as a binomial distribution. For $n$ dice, we can get from 0 to $n$ successes, which is the range over which this equation is valid. For values above and below this, it's not possible to get that number of successes, and thus the probability is zero. This indicates that at least part of Burning Wheel's table is consistent: "Too Hard" refers to the cases where the probability is 0.

The next step is to compute the probability of getting enough successes on the dice rolled to pass the test. Unfortunately, I don't know a meaningful way to come up with a meaningful closed form solution for this. So, instead, we'll just have to sum the pmf over the range of values that pass. If the Ob is such that we need $o$ successes to pass a test with $n$ dice, then we n
eed to add up the pmf from $o$ to $n$. We need at least $o$ successes and can get at most $n$.
P(\text{pass}) = \sum_{s=o}^{n} f_S(s)
I've computed these and plotted them in Fig. 2 on top of a color coding matching the difficulty categories printed in Burning Wheel. Note the orientation is different, higher exponents are on top. This is both because I'm thinking of this as a plot instead of a table and because my plotting tool puts it in this order by default. As you can see the difficulties are not consistent. Probabilities of 0.5 are categorized as both easy and hard (Exp 1 vs.\ Ob 1, 3 Exp vs.\ Ob 2) as well as probabilities of 0.25 (Exp 2 vs.\ Ob 2, Exp 9 vs. Ob 6). Probability 0.25 is listed as "Easy" while probability 0.5 is "Hard''.

Fig. 2: Burning Wheel probabilities of passing tests colored by difficulty

In Fig. 3 is a modified figure colored by the probabilities, in which it's easier to see that the slope of constant probabilities is higher than that indicated by the Burning Wheel difficulty table. For example, if we track probably 0.5, we can see that for every increase of 1 in the obstacle, the number of dice needs to increase by 2 to maintain the same probability of passing the test. Is this a real phenomenon, or is it happenstance or just some artifact of rounding in the values shown? This is indeed the actual pattern. The pmf is symmetric, since $S$ is a sum of symmetric random variables (the individual die rolls), as the probability of getting a success or failure on each die is equal. The pmf of successes is the same as the pmf of traitors. There are two cases of symmetry for a discrete random variable: either there is a single center value or a pair of center values. In the case of a pair of center values, that means that the number of possible successes can be put into two groups of equal probability. Thus, when the pmf is summed from the higher center value to the maximum, the result is 0.5. Since the range of possible successes ranges from 0 to $n$, there are $n+1$ possible values. For odd $n$ this is an even number of possibilities, which can be evenly split into two groups. This is why we see that for all odd $n$, the probability of getting $\frac{n+1}{2}$ successes is 0.5.

Fig. 3: Burning Wheel probabilities of passing tests colored by probability

Given that the difficulties listed are not consistent, I see two possibilities. The first possibility, that the author, Luke Crane, doesn't know what the probabilities are. However, we know that this is not the case. In the Monster Burner, there is a table of probabilities (a supplement to an earlier edition of the game, the Codex may have something similar). These are listed as approximate fractions, so I've computed the decimal values (interpreting better than $\frac{9}{10}$ as 1), as shown in Fig. 4. It's hard to compare a large number of values displayed like this. To evaluate the error, I've calculated the difference between the passing probabilities that we calculated and the approximate values in the Monster Burner and plotted them in Fig. 5. The maximum error magnitude is about 0.07 and most are less, so that approximate values are quite good.

Fig. 4: Monster Burner approximate probabilities

Fig. 5: Monster Burner approximation error

The second possible explanation is that these difficulties are meant to have some meaning other than the probability of passing a test. By looking at the advancement table from page 41 we can see the same information presented differently and over a wider range of values. Certain numbers of each type of test are needed to advance skills, which may or may not map to equal probabilities of success depending on the desired properties of the game.

Fig. 6: Burning Wheel advancement

No comments:

Post a Comment