Conditional probability and Bayes' theorem

You hear that it is cloudy outside. Suddenly rain feels more likely than it did this morning. Your belief about rain changed because you gained information — and probability has a precise way to measure exactly how much information changes a belief. That measurement is called conditional probability.

This idea matters everywhere: a doctor interpreting a positive test result, a spam filter deciding if an email is junk, a factory tracing a defective part back to a machine. In all of these, you know something new (a test came back positive, an email contains certain words) and you want the probability of something you cannot directly observe. Bayes' theorem is the tool that reverses the direction of a probability, and it is one of the most used formulas in all of applied statistics.

What conditional probability means

The conditional probability of $A$ given $B$, written $P(A\mid B)$, answers: "if I know that $B$ happened, what is the chance that $A$ also happened?" Knowing $B$ shrinks the universe of possible outcomes down to only those where $B$ occurred, so we measure $A$ inside that smaller universe.

The definition is:

$$P(A\mid B)=\frac{P(A\cap B)}{P(B)},P(B)>0.$$

The numerator counts outcomes where both $A$ and $B$ happen; the denominator restricts attention to outcomes where $B$ happens. Rearranging gives the multiplication rule:

$$P(A\cap B)=P(B)P(A\mid B).$$

This says: the chance that both events happen equals the chance the first one happens, times the chance the second happens given the first. Note that $P(A\mid B)$ and $P(B\mid A)$ are generally different numbers — reversing the condition changes the question.

Example 1. A condition on two dice Two fair dice are rolled. Given that the sum is $8$, what is the probability that the first die shows an even number?

List the outcomes with sum $8$:

$(2,6),(3,5),(4,4),(5,3),(6,2)$

There are $5$ equally likely outcomes in the reduced universe. The first die is even in $(2,6)$, $(4,4)$, and $(6,2)$ — that is $3$ of them.

$$P(\text{first even}\mid \text{sum }8)=\frac{3}{5}.$$

As a check with the formula: $P(\text{first even}\cap \text{sum }8)=3/36$ and $P(\text{sum }8)=5/36$, so the ratio is $(3/36)/(5/36)=3/5$.

Chaining conditions: the multiplication rule in action

The multiplication rule extends to any number of events. For three events $A$, $B$, $C$:

$$P(A\cap B\cap C)=P(A)P(B\mid A)P(C\mid A\cap B).$$

This is the natural way to handle draws without replacement: after each draw, the probabilities for the next draw change because the composition of the pool changed. If the events do not influence each other — that is, $P(A\mid B)=P(A)$ — we call them independent, and the multiplication rule collapses to $P(A\cap B)=P(A)P(B)$. Conditional probability is the general concept; independence is the special case.

Example 2. Drawing without replacement A bag contains $4$ red and $6$ blue marbles. Two marbles are drawn without replacement. What is the probability that both are red?

Probability the first is red:

$$P(R_1)=\frac{4}{10}.$$

Given the first was red, $3$ red marbles remain among $9$ total:

$$P(R_2\mid R_1)=\frac{3}{9}=\frac{1}{3}.$$

Multiply:

$$P(R_1\cap R_2)=\frac{4}{10}\cdot \frac{3}{9}=\frac{12}{90}=\frac{2}{15}.$$

Check by counting: the number of ways to choose $2$ of the $4$ red marbles divided by ways to choose any $2$ of $10$ is $\left(\genfrac{}{}{0ex}{}{4}{2}\right)/\left(\genfrac{}{}{0ex}{}{10}{2}\right)=6/45=2/15$. It agrees.

Splitting the problem: the law of total probability

Often you cannot compute $P(A)$ directly, but you can compute it along each of several distinct scenarios. If $B_1,B_2,\dots ,B_n$ are mutually exclusive and cover everything (a partition), then:

$$P(A)=P(A\mid B_1)P(B_1)+P(A\mid B_2)P(B_2)+\cdots +P(A\mid B_n)P(B_n).$$

Intuitively: whatever happens to $A$ must happen inside one of the scenarios, so you add up the scenario-by-scenario contributions, each weighted by how likely that scenario is. This "weighted average of conditional probabilities" is the setup for Bayes' theorem, because it supplies the denominator.

Example 3. Defects from two machines Machine A produces $60\%$ of a factory's output with a $2\%$ defect rate; machine B produces the remaining $40\%$ with a $5\%$ defect rate. What is the probability that a randomly chosen item is defective?

Let $D$ be "defective", $A$ "made by A", $B$ "made by B". The machines form a partition.

Contribution from A:

$$P(D\mid A)P(A)=0.02\cdot 0.60=\mathrm{0.012.}$$

Contribution from B:

$$P(D\mid B)P(B)=0.05\cdot 0.40=\mathrm{0.020.}$$

Add them:

$$P(D)=0.012+0.020=\mathrm{0.032.}$$

So $3.2\%$ of all items are defective. As a sanity check, this lies between the two defect rates ($2\%$ and $5\%$), closer to $2\%$ because A makes more items — exactly what a weighted average should do.

Bayes' theorem: reversing the condition

Suppose you know $P(A\mid B)$ — how likely the evidence is under a hypothesis — but you want $P(B\mid A)$: how likely the hypothesis is given the evidence. Bayes' theorem flips the condition:

$$P(B\mid A)=\frac{P(A\mid B)P(B)}{P(A)}.$$

Each piece has a name worth knowing:

• $P(B)$ is the prior: how likely the hypothesis was before seeing the evidence.
• $P(A\mid B)$ is the likelihood: how strongly the hypothesis predicts the evidence.
• $P(A)$ is the evidence: the overall chance of seeing the evidence, computed by the law of total probability.
• $P(B\mid A)$ is the posterior: the updated probability after the evidence.

With two competing hypotheses $B$ and $\neg B$, the denominator expands:

$$P(B\mid A)=\frac{P(A\mid B)P(B)}{P(A\mid B)P(B)+P(A\mid \neg B)P(\neg B)}.$$

The key insight: a hypothesis that explains the evidence well can still be improbable if it started out improbable. The prior matters.

Example 4. A positive medical test A disease affects $1\%$ of a population. A test detects the disease $99\%$ of the time when it is present (sensitivity), and gives a false positive $5\%$ of the time when it is absent. A random person tests positive. What is the probability they actually have the disease?

Let $D$ = has the disease, $+$ = tests positive. Given:

$$P(D)=0.01,P(+\mid D)=0.99,P(+\mid \neg D)=\mathrm{0.05.}$$

Denominator by total probability:

$$P(+)=P(+\mid D)P(D)+P(+\mid \neg D)P(\neg D)=0.99\cdot 0.01+0.05\cdot \mathrm{0.99.}$$

$$P(+)=0.0099+0.0495=\mathrm{0.0594.}$$

Bayes' theorem:

$$P(D\mid +)=\frac{0.0099}{0.0594}=\frac{1}{6}\approx \mathrm{0.167.}$$

Only about a $17\%$ chance! This surprises almost everyone. The reason: the disease is rare, so among $10,000$ people about $99$ true positives appear alongside about $495$ false positives — false positives dominate. The check: $0.99\cdot 0.01/0.0594=0.0099/0.0594=1/6$, confirming the arithmetic.

A reliable workflow for Bayes problems

Bayes word problems become mechanical if you follow the same steps every time:

1. Name the events. Identify the hypothesis $B$ you care about and the evidence $A$ you observed.
2. List the givens. Write down the prior $P(B)$ and the likelihoods $P(A\mid B)$, $P(A\mid \neg B)$ (or for every partition member).
3. Compute the denominator with the law of total probability.
4. Apply Bayes' theorem and simplify.
5. Sanity-check. The posterior must lie between $0$ and $1$, and if the evidence is genuinely informative it should differ from the prior. If your answer is more extreme than the evidence justifies, re-check the base rates.

A useful shortcut for two hypotheses: the posterior odds are $\frac{P(B\mid A)}{P(\neg B\mid A)}=\frac{P(A\mid B)}{P(A\mid \neg B)}\cdot \frac{P(B)}{P(\neg B)}$ — likelihood ratio times prior odds. In the medical example, the likelihood ratio is $0.99/0.05\approx 19.8$, but the prior odds are $1:99$, so the posterior odds are about $1:5$ — matching $P(D\mid +)=1/6$.

Example 5. Tracing a defect back to its machine Using the factory from before (machine A: $60\%$ of output, $2\%$ defective; machine B: $40\%$ of output, $5\%$ defective), an item is found defective. What is the probability it came from machine A?

We already computed the denominator in the total-probability example:

$$P(D)=\mathrm{0.032.}$$

The numerator is the A-branch:

$$P(D\mid A)P(A)=0.02\cdot 0.60=\mathrm{0.012.}$$

Bayes' theorem:

$$P(A\mid D)=\frac{0.012}{0.032}=\frac{3}{8}=\mathrm{0.375.}$$

Check by substitution: $0.02\cdot 0.60/0.032=0.012/0.032=0.375$, and indeed $0.375+P(B\mid D)=0.375+0.625=1$, since a defective item must come from one of the two machines. Even though A makes more items, its low defect rate means a defective item is more likely from B.

Common mistakes

• Confusing $P(A\mid B)$ with $P(B\mid A)$ — the probability a test is positive given disease is not the probability of disease given a positive test; Bayes' theorem is exactly what converts one to the other.
• Ignoring the base rate and reporting the sensitivity as the answer — a $99\%$ sensitive test for a $1\%$-prevalence disease still yields only about a $17\%$ chance of disease after a positive result.
• Forgetting to shrink the denominator when conditioning — computing $P(A\mid B)$ as $P(A\cap B)$ without dividing by $P(B)$.
• Assuming $P(A\mid B)+P(A\mid \neg B)=1$ — the correct identity is $P(A\mid B)+P(\neg A\mid B)=1$; the two conditional probabilities with different conditions need not sum to $1$.
• Using replacement probabilities for without-replacement draws — after removing a red marble from 4 red and 6 blue, the next-draw probability of red is $3/9$, not $4/10$.

Practice

1. Given $P(A\cap B)=0.2$ and $P(B)=0.5$, find $P(A\mid B)$.
2. A fair six-sided die is rolled once. Given that the result is even, what is the probability that the result is greater than 3?
3. Two cards are drawn from a standard 52-card deck without replacement. Using the multiplication rule, find the probability that both cards are aces.
4. Machine A produces 60% of a factory's items with a 2% defect rate, and machine B produces the other 40% with a 5% defect rate. Using the law of total probability, what is the probability that a randomly chosen item is defective?
5. A disease affects 2% of a population. A test has sensitivity 90% (positive with probability 0.9 when the disease is present) and false positive rate 10% (positive with probability 0.1 when the disease is absent). A person tests positive. Using Bayes' theorem, what is the probability that the person has the disease?

Answers

1. $P(A\mid B)=0.2/0.5=0.4$, by the definition of conditional probability.
2. $2/3$, because among the even outcomes $\{2,4,6\}$, two of them ($4$ and $6$) exceed $3$.
3. $\frac{4}{52}\cdot \frac{3}{51}=\frac{1}{221}$, since after one ace is drawn, 3 aces remain among 51 cards.
4. $0.6\cdot 0.02+0.4\cdot 0.05=0.032$, the weighted average of the two defect rates.
5. $\frac{0.9\cdot 0.02}{0.9\cdot 0.02+0.1\cdot 0.98}=\frac{0.018}{0.116}=\frac{9}{58}\approx 0.155$, since the many false positives from the large healthy population swamp the true positives.

Next, random variables and distributions give us a way to package all these probabilities about an uncertain quantity into a single object.