In the previous chapter, we defined the average treatment effect (ATE) as the difference in average potential outcomes under treatment and control. This is simply the hypothetical difference between the average outcome if we assigned everyone to treatment and the average outcome if we assigned everyone to control.
But we also acknowledged a fundamental problem: we can never observe both potential outcomes for the same individual. So how can we design a procedure to estimate the ATE from data?
In this chapter, we develop a simple and intuitive procedure and estimator.
First, randomly assign individuals to treatment or control.
Second, compute the difference in average outcomes between the treatment and control groups
We show that this is a good procedure because it is an unbiased estimator of the ATE. Importantly, this result relies only on the design of the experiment (the random assignment), not on any assumptions about how outcomes are generated.
2.1 The Problem; A Solution?
2.1.1 Problem: We Can’t Observe the ATE
Recall from the previous chapter that we define the ATE as the difference between two averages: the average outcome if we assigned everyone to treatment minus the average outcome if we assigned everyone to control.
But here’s the problem: we can’t assign everyone to treatment and everyone to control. Each individual can only be in one condition. If we assign someone to treatment, we see their outcome under treatment (but not their outcome under control). If we assign someone to control, we see their outcome under control (but not their outcome under treatment).
This means we can never directly compute the ATE. So how do we get around this?
2.1.2 Solution?: Random Assignment
The key insight is that we don’t need to observe everyone’s outcome under treatment to learn about the average outcome under treatment. If we randomly assign individuals to treatment, then the individuals who happen to be assigned to treatment are—on average—just like the individuals who happen to be assigned to control. They are a simple random sample from the same population.
This is the logic of a completely randomized experiment. We take our group of individuals and randomly divide them into two groups: some go to treatment, some go to control.
The treated individuals are a random sample of the full group. So the average outcome we observe among treated individuals should be a good estimate of what the average outcome would have been if we had assigned everyone to treatment.
Similarly, the average outcome among control individuals should be a good estimate of what the average outcome would have been if we had assigned everyone to control.
In other words, random assignment lets us estimate the two quantities we need—the average outcome under treatment and the average outcome under control—even though we can’t observe them directly for everyone.
The difference between these two estimates gives us an estimate of the ATE. It won’t be perfect. Any particular randomization might, by chance, put more higher-outcome individuals in one group than the other. But on average, across all possible randomizations, this approach gets it exactly right.
2.2 The Observed Data
In a completely randomized experiment, we have \(N\) individuals. We randomly assign \(N_t\) individuals to treatment and the remaining \(N_c = N - N_t\) individuals to control, where \(1 < N_t < N\) (so at least one individual is in each group).
In practice, we tend to use a balanced design where \(N_t = N_c = \dfrac{N}{2}\), so that half of the respondents go into the treatment group and an equal number go into the control group.
We use the indicator variable \(D_i\) to denote the treatment assignment for individual \(i\).
\(D_i = 1\) if individual \(i\) is assigned to treatment
\(D_i = 0\) if individual \(i\) is assigned to control
The big sigma (\(\sum\)) just means “add up.” The expression \(\sum_{i=1}^{N} D_i\) says this: start with individual 1, then individual 2, then individual 3, and so on up to individual \(N\), and add up all their \(D_i\) values.
Since each \(D_i\) is either 0 (control) or 1 (treatment), adding them up counts how many 1’s there are—that is, how many individuals are in the treatment group. That’s \(N_t\).
The expression \((1 - D_i)\) just flips the indicator. If \(D_i = 1\), then \(1 - D_i = 0\). If \(D_i = 0\), then \(1 - D_i = 1\). So adding up all the \((1 - D_i)\) values counts how many 0’s there are in the original \(D_i\)’s—that is, how many individuals are in the control group. That’s \(N_c\).
Example. Suppose we have \(N = 5\) individuals with the following treatment assignments:
Individual \(i\)
\(D_i\)
\(1 - D_i\)
1
1
0
2
0
1
3
1
0
4
0
1
5
1
0
To count the treated individuals, we add up the \(D_i\) column: \(N_t = 1 + 0 + 1 + 0 + 1 = 3\).
To count the control individuals, we add up the \(1 - D_i\) column: \(N_c = 0 + 1 + 0 + 1 + 0 = 2\).
Three individuals are treated; two are in control.
For each individual, we observe only one potential outcome. We call this the observed outcome, denoted \(Y_i^{\text{obs}}\), so that
The notation \(Y_i(D_i)\) means “plug the value of \(D_i\) into the potential outcome function.” Since \(D_i\) is either 0 or 1, this gives us either \(Y_i(0)\) or \(Y_i(1)\)—whichever one actually happened.
The curly brace notation is just a compact way of writing “if-then” rules. It says: look at \(D_i\), and depending on its value, pick the corresponding potential outcome.
For the observed outcome \(Y_i^{\text{obs}}\), we get the potential outcome that matches the treatment the individual actually received. For the missing outcome \(Y_i^{\text{mis}}\), we get the other potential outcome—the one we can’t see.
Example. Suppose we have \(N = 5\) individuals with the following potential outcomes and treatment assignments:
Individual \(i\)
\(Y_i(0)\)
\(Y_i(1)\)
\(D_i\)
\(Y_i^{\text{obs}}\)
\(Y_i^{\text{mis}}\)
1
4
7
1
7
4
2
6
5
0
6
5
3
5
8
1
8
5
4
3
6
0
3
6
5
7
9
1
9
7
For individual 1, \(D_1 = 1\) (treated), so we observe \(Y_1^{\text{obs}} = Y_1(1) = 7\) and the missing outcome is \(Y_1^{\text{mis}} = Y_1(0) = 4\).
For individual 2, \(D_2 = 0\) (control), so we observe \(Y_2^{\text{obs}} = Y_2(0) = 6\) and the missing outcome is \(Y_2^{\text{mis}} = Y_2(1) = 5\).
2.2.1 What the Data Look Like
To make this concrete, consider again our hypothetical experiment with 10 respondents. Suppose we know all the potential outcomes:
Respondent
\(Y_i(0)\)
\(Y_i(1)\)
\(\tau_i\)
1
4
7
3
2
6
5
-1
3
5
5
0
4
3
9
6
5
7
10
3
6
5
4
-1
7
4
8
4
8
6
7
1
9
5
3
-2
10
4
6
2
We computed in the previous chapter that \(\text{ATE} = 1.5\) for these data.
Now suppose we run an experiment with \(N_t = 5\) and \(N_c = 5\). After randomization, respondents 1, 4, 5, 7, and 10 are assigned to treatment, while respondents 2, 3, 6, 8, and 9 are assigned to control. What do we actually observe?
Respondent
\(Y_i(0)\)
\(Y_i(1)\)
\(D_i\)
\(Y_i^{\text{obs}}\)
1
?
7
1
7
2
6
?
0
6
3
5
?
0
5
4
?
9
1
9
5
?
10
1
10
6
5
?
0
5
7
?
8
1
8
8
6
?
0
6
9
5
?
0
5
10
?
6
1
6
Notice that for each respondent, we observe exactly one potential outcome. The question marks represent the counterfactual outcomes we can never observe. We cannot compute \(\tau_i = Y_i(1) - Y_i(0)\) for any individual because we never have both values.
But here’s the key insight: even though we can’t compute individual treatment effects, we can compute averages within the treatment and control groups.
2.3 The Difference-in-Means Estimator
A natural approach is to estimate the ATE by comparing the average outcome among treated individuals to the average outcome among control individuals. We define \(\overline{Y}^{\text{obs}}_{t}\) and \(\overline{Y}^{\text{obs}}_{c}\) as
The bar over \(Y\) (as in \(\overline{Y}\)) means “average.” So \(\overline{Y}^{\text{obs}}_{t}\) is the average observed outcome in the treatment group, and \(\overline{Y}^{\text{obs}}_{c}\) is the average observed outcome in the control group.
The subscript \(i:D_i=1\) under the summation sign means “only add up the individuals where \(D_i = 1\)”—that is, only the treated individuals. Similarly, \(i:D_i=0\) means “only the control individuals.”
So the formula says this: add up the observed outcomes for all treated individuals, then divide by \(N_t\) (the number of treated individuals) to get the average. Do the same for control individuals, dividing by \(N_c\).
Example. Suppose we have \(N = 5\) individuals:
Individual \(i\)
\(D_i\)
\(Y_i^{\text{obs}}\)
1
1
7
2
0
6
3
1
8
4
0
3
5
1
9
There are \(N_t = 3\) treated individuals (individuals 1, 3, and 5) and \(N_c = 2\) control individuals (individuals 2 and 4).
For the treatment group, we add up only the outcomes where \(D_i = 1\), so that
The “hat” over ATE (as in \(\widehat{\text{ATE}}\)) means “estimate of.” We can’t compute the true ATE because we don’t observe all potential outcomes, so we estimate it using the data we have.
The formula says: take the average outcome among treated individuals and subtract the average outcome among control individuals. That’s it—just a difference of two averages.
Example. Using the same 5 individuals from before:
Individual \(i\)
\(D_i\)
\(Y_i^{\text{obs}}\)
1
1
7
2
0
6
3
1
8
4
0
3
5
1
9
We already computed \(\overline{Y}^{\text{obs}}_{t} = 8\) and \(\overline{Y}^{\text{obs}}_{c} = 4.5\).
Thus \(\widehat{\text{ATE}} = 8 - 4.5 = 3.5\). Our estimate is that the treatment increases outcomes by 3.5 units on average.
Therefore \(\widehat{\text{ATE}} = 8 - 5.4 = 2.6\). But recall that the true ATE (which we computed using all potential outcomes) is 1.5. So our estimate is off by 1.1 units.
Is this a problem? Not necessarily. Any particular estimate will differ from the true ATE due to the randomness in treatment assignment. The question is whether, on average across all possible randomizations, our estimator gets it right.
2.4 Why This Estimator?
Why is \(\widehat{\text{ATE}} = \overline{Y}^{\text{obs}}_{t} - \overline{Y}^{\text{obs}}_{c}\) a sensible estimator of the ATE?
The ATE is the difference between two population averages: the average of all potential outcomes under treatment and the average of all potential outcomes under control.
Our estimator replaces these population averages with sample averages:
Instead of averaging \(Y_i(1)\) over all individuals, we average \(Y_i^{\text{obs}}\) over individuals assigned to treatment
Instead of averaging \(Y_i(0)\) over all individuals, we average \(Y_i^{\text{obs}}\) over individuals assigned to control
This “plug-in” approach—replacing population quantities with their sample counterparts—is a fundamental strategy in statistics. Under random assignment, the treated individuals are a random sample from the population, so \(\overline{Y}^{\text{obs}}_{t}\) should be a good estimate of \(\overline{Y}(1)\). Similarly, the control individuals are a random sample, so \(\overline{Y}^{\text{obs}}_{c}\) should be a good estimate of \(\overline{Y}(0)\).
2.5 Design-Based Inference
Before proving that the estimator \(\widehat{\text{ATE}}\) is unbiased, we need to understand
that the estimate \(\widehat{\text{ATE}}\) is a noisy and random estimate and
what makes this estimate random in the first place.
In typical statistical settings, we imagine that data are drawn from some distribution \(Y_i \sim f(\theta)\). The randomness comes from imagining that the data could have been different if we had drawn a different sample from the population.
But in the potential outcomes framework for randomized experiments, we take a different view. The potential outcomes \(Y_i(0)\) and \(Y_i(1)\) are treated as fixed values, not random variables. They are simply the outcomes each individual would experience under each condition—there’s nothing random about the potential outcomes.
The only randomness in the experiment comes from the treatment assignment \(D_i\). And because we designed the randomization mechanism, we know exactly how this randomness works.
This is important: because we designed the randomization mechanism, we know exactly how this randomness works.
This is called design-based inference in which all uncertainty comes from the design of the experiment (i.e., the random assignment in this case), not from any model of how outcomes are generated.
To make this explicit, we can rewrite the estimator to highlight what is random and what is fixed, so that
This formula looks intimidating, but it’s just a clever rewriting of the difference-in-means estimator that separates the random part (\(D_i\)) from the fixed parts (potential outcomes).
Let’s break it down piece by piece:
\(D_i \cdot Y_i(1)\): This equals \(Y_i(1)\) when individual \(i\) is treated (\(D_i = 1\)) and equals 0 when individual \(i\) is in control (\(D_i = 0\)). So it “turns on” the treated potential outcome only for treated individuals.
\((1 - D_i) \cdot Y_i(0)\): This does the opposite—it equals \(Y_i(0)\) when \(D_i = 0\) and equals 0 when \(D_i = 1\). It “turns on” the control potential outcome only for control individuals.
\(N_t/N\): This is the fraction of individuals assigned to treatment (the treatment probability).
Dividing by \(N_t/N\) in the first term and \(N_c/N\) in the second term rescales each contribution so that the formula averages correctly over each group.
The key insight is that when you work through the algebra, this formula gives exactly the same answer as \(\overline{Y}^{\text{obs}}_{t} - \overline{Y}^{\text{obs}}_{c}\). But writing it this way makes clear that \(D_i\) is the only random quantity—the potential outcomes \(Y_i(1)\) and \(Y_i(0)\) are fixed numbers.
Example. Suppose \(N = 4\) with \(N_t = 2\) and \(N_c = 2\) and the following potential outcomes and randomization:
Individual \(i\)
\(Y_i(0)\)
\(Y_i(1)\)
\(D_i\)
1
4
7
1
2
6
5
0
3
3
8
1
4
5
6
0
For individual 1 (\(D_1 = 1\)): the contribution is \(\frac{1 \cdot 7}{2/4} - \frac{0 \cdot 4}{2/4} = \frac{7}{0.5} - 0 = 14\).
For individual 2 (\(D_2 = 0\)): the contribution is \(\frac{0 \cdot 5}{2/4} - \frac{1 \cdot 6}{2/4} = 0 - \frac{6}{0.5} = -12\).
For individual 3 (\(D_3 = 1\)): the contribution is \(\frac{1 \cdot 8}{0.5} - 0 = 16\).
For individual 4 (\(D_4 = 0\)): the contribution is \(0 - \frac{5}{0.5} = -10\).
Adding these and dividing by \(N = 4\): \(\widehat{\text{ATE}} = \frac{1}{4}(14 - 12 + 16 - 10) = \frac{8}{4} = 2\).
You can verify this equals the simple difference in means: \(\overline{Y}^{\text{obs}}_{t} = \frac{7 + 8}{2} = 7.5\) and \(\overline{Y}^{\text{obs}}_{c} = \frac{6 + 5}{2} = 5.5\), so \(\widehat{\text{ATE}} = 7.5 - 5.5 = 2\).
In this expression
\(Y_i(1)\) and \(Y_i(0)\) are fixed (the potential outcomes)
\(N\), \(N_t\), and \(N_c\) are fixed (determined by design)
\(D_i\) is random (the only source of uncertainty)
The subscript notation \(E_D[\cdot]\) indicates that we are taking expectations with respect to the randomization distribution—the distribution induced by the random assignment of individuals to treatment and control.
2.6 Unbiasedness of \(\widehat{\text{ATE}}\)
We now prove the key result: \(\widehat{\text{ATE}}\) is an unbiased estimator of \(\text{ATE}\). Informally, this means that our estimator “gets it right on average”. It means that if we could repeat the randomization many times, our estimates would exactly equal the ATE in the long-run.
To state this result precisely, we need to introduce some notation. The notation is tedious, but the underlying idea is straightforward, important, and powerful.
2.6.1 The Expectation Operator \(E[\cdot]\)
The expectation operator \(E[\cdot]\) is just a way of writing “long-run average.”
The expectation of a random variable is its average value across all possible outcomes, weighted by how likely each outcome is. You can think of \(E[X]\) as asking: “If I could repeat this random process infinitely many times, what would \(X\) average out to?”
For our purposes, we need four simple rules about expectations.
Rule 1: The expectation of a constant is that constant.
If \(c\) is a fixed number (not random), then \(E[c] = c\).
If something never changes, its “average” is just itself. For example, \(E[5] = 5\).
Rule 2: Constants can be pulled outside the expectation.
If \(c\) is a constant and \(X\) is random, then \(E[c \cdot X] = c \cdot E[X]\).
The average of “5 times something random” equals 5 times the average of that random thing. If on average you earn $100 per day, then on average you earn $500 per five-day week.
Rule 3: The expectation of a sum is the sum of the expectations.
If \(X\) and \(Y\) are random variables, then \[
E[X + Y] = E[X] + E[Y].
\]
The average of “two random things added together” equals the sum of their averages. If on average you earn $100 per day from your job and $20 per day from side work, then on average you earn $120 per day total. You do not need \(X\) and \(Y\) to be independent for this rule to hold.
Rule 4: The expectation of a binary (0/1) random variable equals its probability of being 1.
If \(X\) can only be 0 or 1, then \(E[X] = \Pr(X = 1)\).
This is the key rule for our proof. And it’s intuitive. If you flip a fair coin and let \(X = 1\) for heads and \(X = 0\) for tails, what’s the average value of \(X\)? Half the time you get 1, half the time you get 0, so the average is \(\frac{1}{2}(1) + \frac{1}{2}(0) = 0.5\). And \(\Pr(X = 1) = 0.5\). They match!
More generally, if you repeat the process many times, the fraction of 1’s you observe will approach the probability of getting a 1. So the average value equals the probability.
Why does Rule 4 work? Well, remember that the expectation of a random variable is defined as the sum of each possible value times its probability, so that
The zero term disappears, and we’re left with just the probability that \(X\) equals 1.
Example. Suppose you roll a die and let \(X = 1\) if you roll a 6, and \(X = 0\) otherwise. Then \(\Pr(X = 1) = 1/6\), so \(E[X] = 1/6 \approx 0.167\). If you rolled the die 600 times, you’d expect about 100 sixes, and your average value of \(X\) would be about \(100/600 = 0.167\).
2.6.2 The Theorem
Now we can state the key result precisely. When we write \(E_D[\widehat{\text{ATE}}]\), we mean “the average value of our estimate across all possible random assignments.” (The subscript \(D\) reminds us that the only randomness comes from treatment assignment.)
Theorem
Under random assignment, \(\widehat{\text{ATE}} = \overline{Y}^{\text{obs}}_{t} - \overline{Y}^{\text{obs}}_{c}\) is an unbiased estimator of the ATE. This mean that \[
E_D[\widehat{\text{ATE}}] = \text{ATE}.
\]
If we average our estimate over all the different ways we could have randomly assigned individuals to treatment and control, we get exactly the true ATE.
2.6.3 The Proof
Proof. We want to show that \(E_D[\widehat{\text{ATE}}] = \text{ATE}\).
Step 1: Write the estimator to separate random from fixed.
First, we write \(\widehat{\text{ATE}}\) in a form that clearly separates the random part (\(D_i\)) from the fixed parts (the potential outcomes \(Y_i(1)\) and \(Y_i(0)\)). We borrow the representation above, where
Here’s where the magic happens. Because the potential outcomes are fixed numbers and only \(D_i\) is random, we can move the expectation operator inside the formula until it includes only\(D_i\) and \(1 - D_i\). Rule 2 above allows us to do this.
Think of it this way: if you’re computing the average of “5 times something random,” you can compute it as “5 times the average of that random thing” (that’s Rule 2). The \(\frac{1}{N}\), the \(\sum\), the potential outcomes \(Y_i(1)\) and \(Y_i(0)\), the fractions \(N_t/N\) and \(N_c/N\) are all fixed numbers that pass right through the expectation. The expectation only “grabs onto” the random part, \(D_i\).
Now we need to find \(E_D[D_i]\). Remember that \(D_i\) is a binary variable that equals 1 if individual \(i\) is assigned to treatment, and 0 if individual \(i\) is assigned to control. By Rule 4, the expectation of a binary variable equals its probability of being 1, so that
\[
E_D[D_i] = \Pr(D_i = 1).
\]
Under completely randomized assignment, every individual has the same chance of being assigned to treatment. If we’re assigning \(N_t\) individuals to treatment out of \(N\) total individuals, then each individual has probability \(N_t/N\) of being treated so that
\[
E_D[D_i] = \Pr(D_i = 1) = \frac{N_t}{N}.
\]
Step 5: Compute the expectation of \(1 - D_i\).
Similarly, \(1 - D_i\) is also binary. It equals 1 when \(D_i = 0\) (individual in control) and equals 0 when \(D_i = 1\) (individual in treatment). Then we have
\[
\begin{align}
E_D[\widehat{\text{ATE}}] &= \frac{1}{N} \sum_{i=1}^{N} \left( \frac{(N_t/N) \cdot Y_i(1)}{N_t/N} - \frac{(N_c/N) \cdot Y_i(0)}{N_c/N} \right).
\end{align}
\] Then something magical happens. In the first term, we have \(N_t/N\) in both the numerator and denominator, so they cancel out.
The same thing happens with \(N_c/N\) in the second term, so that
This is just the average of individual treatment effects across all individuals. This is exactly the definition of the ATE, so that
\[
E_D[\widehat{\text{ATE}}] = \text{ATE}.
\]
This completes the proof and shows that \(\widehat{\text{ATE}}\) is an unbiased estimator of \(\widehat{\text{ATE}}\). \(\square\)
Tip
The key insight is that the fractions \(N_t/N\) and \(N_c/N\) cancel perfectly. This happens because (1) the probability of assignment to treatment equals \(N_t/N\), and (2) we divide by \(N_t\) when averaging over treated individuals. The “selection” into treatment is perfectly balanced by the “weighting” in our estimator.
2.6.4 What Does Unbiasedness Mean?
Unbiasedness means that on average, across all possible randomizations, our estimator equals the true ATE. Any particular estimate might be too high or too low, but there’s no systematic tendency to over- or under-estimate.
In our example, we got \(\widehat{\text{ATE}} = 2.6\) when the true ATE is 1.5. This particular randomization overestimated. But if we had assigned different individuals to treatment, we might have gotten a different estimate. Some randomizations would overestimate, some would underestimate, and on average we would get it right.
2.6.5 No Model Required
Notice what we did not assume in this proof:
We did not assume any particular distribution for the outcomes
We did not assume the treatment effects are constant
We did not assume the outcomes are normally distributed
We did not assume any functional form for how treatment affects outcomes
We did not assume a “large” sample
The only assumption is that treatment was randomly assigned. This is why randomized experiments are so powerful. Randomization provides unbiased estimates (and other good things to be discussed later) without requiring strong modeling assumptions.
2.7 A Different Randomization
To reinforce that \(\widehat{\text{ATE}}\) depends on the randomization, consider what happens with a different random assignment. Suppose instead that individuals 2, 3, 6, 8, and 9 are assigned to treatment, while individuals 1, 4, 5, 7, and 10 are assigned to control.
This randomization gives us \(\widehat{\text{ATE}} = 0.4\), which underestimates the true ATE of 1.5. The first randomization overestimated (2.6), this one underestimates (0.4). Unbiasedness tells us that if we averaged over all possible randomizations, we would get exactly 1.5.
2.8 Computing \(\widehat{\text{ATE}}\) in R
So far, we have computed \(\widehat{\text{ATE}}\) by hand. In practice, we use software. In this section, we walk through two approaches in R: (1) using mean() to compute the difference in means directly, and (2) using lm() to fit a linear regression. Both approaches give the same answer, but mean() is a little more intuitive and lm() is more useful in the practice.
2.8.1 The Mock Rasinski Data
We collected data that mimics Rasinski’s experiment. Each respondent was randomly assigned to see either the “welfare” wording or the “assistance to the poor” wording. The respondent then indicated whether they thought government spending on this program was “Too little,” “About right,” or “Too much.”
First, let’s load the data and prepare it for analysis. For convenience, I’ve written a little tribble() as a convenient way to hold the data (rather than loading from a CSV).
# load packageslibrary(tidyverse)library(tibble)library(forcats)# load the mock rasinski datarasinski <-tribble(~desc, ~response,"assistance to the poor", "Too little","welfare", "Too little","assistance to the poor", "Too little","assistance to the poor", "Too little","welfare", "Too little","welfare", "Too little","assistance to the poor", "Too little","assistance to the poor", "Too little","welfare", "Too little","assistance to the poor", "About right","assistance to the poor", "Too little","assistance to the poor", "About right","welfare", "Too little","welfare", "Too little","assistance to the poor", "Too little","assistance to the poor", "Too little","welfare", "About right","welfare", "About right","assistance to the poor", "Too little","welfare", "Too little","welfare", "Too much","welfare", "About right","assistance to the poor", "Too little","assistance to the poor", "Too little","assistance to the poor", "Too much","welfare", "Too little","welfare", "About right","welfare", "About right","welfare", "About right","assistance to the poor", "Too little","assistance to the poor", "Too little","welfare", "Too little","assistance to the poor", "Too little","assistance to the poor", "Too little","welfare", "Too little","assistance to the poor", "Too little","welfare", "Too little","assistance to the poor", "Too little","welfare", "Too little")# take a quick lookglimpse(rasinski)
Rows: 39
Columns: 2
$ desc <chr> "assistance to the poor", "welfare", "assistance to the poor"…
$ response <chr> "Too little", "Too little", "Too little", "Too little", "Too …
The desc variable contains the experimental condition. It takes on the value "welfare" (control) or "assistance to the poor" (treatment). The response variable contains the respondent’s answer.
Following Rasinski’s coding scheme from the previous chapter, we code the outcome numerically:
We also need to create a treatment indicator variable. Following our convention, treatment (\(D_i = 1\)) is the “assistance to the poor” wording and control (\(D_i = 0\)) is the “welfare” wording.
# create treatment indicatorrasinski <- rasinski |>mutate(treatment =if_else(desc =="assistance to the poor", 1, 0) ) |>glimpse()
We can compute this directly using R’s mean() function. The key is to subset the data to get the outcomes for each group separately.
Step 1: Identify the treated and control observations.
We use logical indexing to select rows where the treatment indicator equals 1 (treated) or 0 (control).
We can write compactly in a single line.
# direct computation of difference in meansate_hat <-mean(rasinski$outcome[rasinski$treatment ==1]) -mean(rasinski$outcome[rasinski$treatment ==0])ate_hat
[1] -0.2210526
The estimated ATE is -0.221. Since lower values indicate more support for spending (“Too little” = 1), a negative estimate means the “assistance to the poor” wording increases support for spending compared to the “welfare” wording.
This Is Only an Estimate
Remember that \(\widehat{\text{ATE}}\) is an estimate of the true \(\text{ATE}\), not the \(\text{ATE}\) itself. If we had randomized respondents differently—assigning different people to treatment and control—we would have observed different outcomes and computed a different estimate. Perhaps it would have been a bit larger, perhaps a bit smaller, or perhaps even the opposite sign.
This is a fundamental feature of randomized experiments: the estimate we get depends on the particular randomization we happened to draw. The true \(\text{ATE}\) is fixed (it’s determined by the potential outcomes in the population), but our estimate of it varies with the randomization.
In future chapters, we will address this uncertainty head-on. Our framework allows us to say a great deal about how good our estimate is—that is, how close \(\widehat{\text{ATE}}\) is likely to be to the true \(\text{ATE}\). For now, just keep in mind that the number we computed is our best guess, but it comes with uncertainty that we haven’t yet quantified.
2.8.3 Approach 2: Using lm()
Setting Up the Model
Linear regression with lm() provides another way to compute the same estimate. This might seem surprising at first. After all, regression is usually taught as a way to model relationships between variables. But it turns out that a simple regression of the outcome on a treatment indicator gives us exactly the difference in means.
When can fit the model
\[
Y_i = \beta_0 + \beta_1 D_i + \epsilon_i,
\]
where \(D_i\) is the treatment indicator (1 = treated, 0 = control).
The estimated coefficients have a direct interpretation:
\(\hat{\beta}_0\) is the mean outcome in the control group (\(\overline{Y}^{\text{obs}}_{c}\))
\(\hat{\beta}_1\) is the difference in means (\(\overline{Y}^{\text{obs}}_{t} - \overline{Y}^{\text{obs}}_{c}\))
Fitting the Model
First, we need to ensure that “welfare” (control) is the reference category for our treatment variable. When lm() encounters a factor variable, it creates a dummy variable using the first level as the reference. By default, R orders factor levels alphabetically, which would make “assistance to the poor” the reference—the opposite of what we want.
We use fct_relevel() from the forcats package to set “welfare” as the reference category.
# create a factor with "welfare" as the reference categoryrasinski <- rasinski |>mutate(condition =fct_relevel(desc, "welfare") )# verify: "welfare" should be firstlevels(rasinski$condition)
[1] "welfare" "assistance to the poor"
Now we fit the linear model.
# fit the modelfit <-lm(outcome ~ condition, data = rasinski)# extract the coefficientscoef(fit)
(Intercept) conditionassistance to the poor
1.4210526 -0.2210526
Let’s verify that these coefficients match our manual calculations.
# the slope should equal the ATE estimatecoef(fit)[2]
conditionassistance to the poor
-0.2210526
ate_hat
[1] -0.2210526
The values match exactly. Using mean() makes the calculation transparent. You can see exactly what quantities are being computed: the mean in each group and their difference. This is valuable for understanding and teaching the concept. Using lm() will be more valuable in applied work. We’ll use lm() moving forward.
2.9 Key Terms
Completely randomized experiment
Observed outcome
Missing outcome
Difference-in-means estimator
Design-based inference
Unbiasedness
2.10 Exercises
2.10.1 Conceptual Understanding
Observed vs. Missing Outcomes. In your own words, explain the difference between \(Y_i^{\text{obs}}\) and \(Y_i^{\text{mis}}\). Why is one observable and the other not?
Solution
\(Y_i^{\text{obs}}\) is the observed outcome—the potential outcome that corresponds to the treatment condition the individual actually received. If individual \(i\) was assigned to treatment (\(D_i = 1\)), then \(Y_i^{\text{obs}} = Y_i(1)\). If individual \(i\) was assigned to control (\(D_i = 0\)), then \(Y_i^{\text{obs}} = Y_i(0)\).
\(Y_i^{\text{mis}}\) is the missing outcome—the potential outcome that corresponds to the treatment condition the individual did not receive. It is the counterfactual: what would have happened if the individual had been assigned to the other condition.
We can observe \(Y_i^{\text{obs}}\) because it is the outcome that actually occurred. Once we assign individual \(i\) to treatment, we see how they respond to treatment. But we cannot observe \(Y_i^{\text{mis}}\) because we cannot simultaneously assign the same individual to both treatment and control. The individual can only be in one condition at a time, so we only ever see one of the two potential outcomes.
This is the fundamental problem of causal inference applied to the observed data.
Design-Based Inference. Explain what it means to say that inference in randomized experiments is “design-based” rather than “model-based.” What is the source of randomness in each approach?
Solution
In design-based inference, the potential outcomes \(Y_i(0)\) and \(Y_i(1)\) are treated as fixed values—they are simply the outcomes each individual would experience under each condition. The only source of randomness comes from the treatment assignment \(D_i\). Because we designed the randomization mechanism, we know exactly how this randomness works.
In model-based inference, the outcomes themselves are treated as random draws from some probability distribution. The randomness comes from imagining that the data could have been different if we had drawn a different sample from the population, or from random variation in how outcomes are generated.
The key difference is where the uncertainty comes from:
Design-based: Uncertainty comes from the random assignment. The potential outcomes are fixed; we just don’t know which ones we’ll observe because we don’t know the randomization in advance.
Model-based: Uncertainty comes from the data-generating process. The outcomes themselves are random, drawn from some distribution with unknown parameters.
Design-based inference is powerful because we don’t need to make assumptions about how outcomes are generated. We only rely on the randomization we controlled.
The Meaning of Unbiasedness. A student says, “Since \(\widehat{\text{ATE}}\) is unbiased, my estimate must equal the true ATE.” Explain why this is incorrect. What does unbiasedness actually guarantee?
Solution
The student’s statement is incorrect. Unbiasedness does not mean that any single estimate equals the true ATE.
Unbiasedness is a property about averages over repeated randomizations. Specifically, \(\widehat{\text{ATE}}\) is unbiased means:
\[E_D[\widehat{\text{ATE}}] = \text{ATE}\]
In plain language: if we could repeat the randomization infinitely many times and compute \(\widehat{\text{ATE}}\) each time, the average of all those estimates would equal the true ATE.
Any particular estimate will almost certainly differ from the true ATE. Some randomizations will, by chance, put more high-outcome individuals in the treatment group, leading to overestimates. Other randomizations will put more high-outcome individuals in control, leading to underestimates. Unbiasedness tells us that these over- and under-estimates balance out on average—there is no systematic tendency to err in one direction.
The chapter example illustrates this: one randomization gave \(\widehat{\text{ATE}} = 2.6\) (overestimate), another gave \(\widehat{\text{ATE}} = 0.4\) (underestimate), but the true ATE was 1.5. Neither estimate equaled the truth, but they averaged out correctly.
Why Randomization Matters. Suppose instead of randomly assigning treatment, we let participants choose whether to receive treatment. Why would \(\widehat{\text{ATE}} = \overline{Y}^{\text{obs}}_{t} - \overline{Y}^{\text{obs}}_{c}\) no longer be unbiased for the ATE?
Solution
If participants choose whether to receive treatment, the people who select into treatment may be systematically different from those who select into control. This is called self-selection bias.
For example, in a job training program where participants choose whether to enroll:
People who are more motivated, more confident, or have better prospects might be more likely to enroll
These same characteristics might also lead to better outcomes regardless of training
In this case, the treatment group would have higher average outcomes than the control group even if the training itself had no effect. The difference in means would reflect both (1) any true effect of training and (2) pre-existing differences between the groups.
Mathematically, the proof of unbiasedness relies on the fact that \(E_D[D_i] = N_t/N\) for all individuals \(i\)—every individual has the same probability of being treated. With self-selection, this no longer holds. Individuals with higher potential outcomes might have higher probabilities of selecting treatment, which means \(E_D[D_i]\) would vary across individuals and would be correlated with potential outcomes.
When treatment assignment is correlated with potential outcomes, the difference-in-means estimator captures both the treatment effect and the selection effect, making it biased for the ATE.
2.10.2 Computational Practice
Computing the Estimate. Consider the following observed data from a randomized experiment:
Individual
\(D_i\)
\(Y_i^{\text{obs}}\)
1
1
12
2
0
8
3
1
15
4
0
7
5
0
9
6
1
11
What are \(N_t\) and \(N_c\)?
Compute \(\overline{Y}^{\text{obs}}_{t}\) and \(\overline{Y}^{\text{obs}}_{c}\).
Compute \(\widehat{\text{ATE}}\).
Solution
Part a. Count the individuals in each group by looking at the \(D_i\) column:
Individuals with \(D_i = 1\) (treatment): Individuals 1, 3, and 6
Individuals with \(D_i = 0\) (control): Individuals 2, 4, and 5
So \(N_t = 3\) and \(N_c = 3\).
Part b. Compute the mean outcome in each group.
For the treatment group (individuals where \(D_i = 1\)):
Our estimate is that treatment increases outcomes by about 4.67 units on average.
Different Randomizations. Suppose the true potential outcomes for 4 individuals are:
Individual
\(Y_i(0)\)
\(Y_i(1)\)
1
2
5
2
4
6
3
3
4
4
5
7
Compute the true ATE.
If individuals 1 and 3 are assigned to treatment (\(D_1 = D_3 = 1\)) and individuals 2 and 4 are assigned to control (\(D_2 = D_4 = 0\)), compute \(\widehat{\text{ATE}}\).
If individuals 2 and 4 are assigned to treatment and individuals 1 and 3 are assigned to control, compute \(\widehat{\text{ATE}}\).
Average your two estimates from (b) and (c). How does this compare to the true ATE?
Solution
Part a. First, compute the individual treatment effects:
This equals the true ATE exactly! The first randomization underestimated (0 vs. 2), and the second overestimated (4 vs. 2), but they balanced out. This illustrates unbiasedness: on average, the estimator gets it right.
Verifying Unbiasedness. Using the same potential outcomes from Exercise 6, list all possible ways to assign exactly 2 individuals to treatment and 2 to control. For each, compute \(\widehat{\text{ATE}}\). Then compute the average of all these estimates and verify that it equals the true ATE.
Solution
With 4 individuals and 2 assigned to treatment, there are \(\binom{4}{2} = 6\) possible randomizations. Let’s list each one and compute \(\widehat{\text{ATE}}\).
Recall the potential outcomes:
Individual
\(Y_i(0)\)
\(Y_i(1)\)
1
2
5
2
4
6
3
3
4
4
5
7
Randomization 1: Individuals 1, 2 treated; Individuals 3, 4 control
The average of all possible estimates equals 2, which is exactly the true ATE we computed in Exercise 6. This confirms that \(\widehat{\text{ATE}}\) is unbiased: averaged across all possible randomizations, it equals the truth.
2.10.3 Application to Rasinski’s Experiment
Rasinski’s Experiment Revisited. Return to the hypothetical Rasinski data from the previous chapter. Suppose we randomly assign 5 respondents to the “assistance to the poor” wording (treatment) and 5 to the “welfare” wording (control).
Respondent
\(Y_i(0)\) (“welfare”)
\(Y_i(1)\) (“assistance”)
1
3 (Too much)
1 (Too little)
2
3 (Too much)
2 (About right)
3
2 (About right)
1 (Too little)
4
3 (Too much)
1 (Too little)
5
2 (About right)
1 (Too little)
6
3 (Too much)
2 (About right)
7
2 (About right)
1 (Too little)
8
3 (Too much)
1 (Too little)
9
2 (About right)
2 (About right)
10
3 (Too much)
1 (Too little)
We know from the previous chapter that the true ATE is -1.3. If respondents 1, 3, 5, 7, and 9 are assigned to treatment, compute \(\widehat{\text{ATE}}\). How close is it to the true ATE?
If respondents 2, 4, 6, 8, and 10 are assigned to treatment instead, compute \(\widehat{\text{ATE}}\). How close is it to the true ATE?
Average your estimates from (a) and (b). What do you notice?
Solution
Part a. With respondents 1, 3, 5, 7, and 9 assigned to treatment:
Treatment group (sees “assistance to the poor”): Respondents 1, 3, 5, 7, 9
The average equals the true ATE exactly! This is a concrete illustration of unbiasedness. The first randomization overestimated the effect (in magnitude), and the second underestimated it by the same amount. When we average them, we get exactly the truth.
Note: These two randomizations are not the only possible ones—there are \(\binom{10}{5} = 252\) ways to assign 5 of 10 respondents to treatment. If we computed \(\widehat{\text{ATE}}\) for all 252 and averaged them, we would get exactly \(-1.3\).
What the Researcher Sees. In Rasinski’s actual experiment, the researcher does not observe the potential outcomes table—they only observe the outcomes under the assigned condition.
Using the randomization from Exercise 8a (respondents 1, 3, 5, 7, 9 assigned to treatment), write out the data table that the researcher would actually observe. Use “?” for unobserved potential outcomes.
Can the researcher compute the true ATE from this observed data? Why or why not?
What does the researcher compute instead, and why is this a reasonable substitute?
Solution
Part a. With respondents 1, 3, 5, 7, 9 assigned to treatment, the researcher observes:
Respondent
\(Y_i(0)\)
\(Y_i(1)\)
\(D_i\)
\(Y_i^{\text{obs}}\)
1
?
1
1
1
2
3
?
0
3
3
?
1
1
1
4
3
?
0
3
5
?
1
1
1
6
3
?
0
3
7
?
1
1
1
8
3
?
0
3
9
?
2
1
2
10
3
?
0
3
For each respondent, the researcher sees only one potential outcome—the one corresponding to their assigned condition. The other potential outcome is forever unknown.
Part b. No, the researcher cannot compute the true ATE from this observed data.
The true ATE requires knowing all potential outcomes:
But the researcher is missing \(Y_i(0)\) for treated respondents (respondents 1, 3, 5, 7, 9) and \(Y_i(1)\) for control respondents (respondents 2, 4, 6, 8, 10). Without these counterfactual outcomes, the true ATE cannot be calculated.
Part c. The researcher computes the difference-in-means estimator:
It uses only observable quantities (no counterfactuals needed)
Under random assignment, it is unbiased for the true ATE
The average of \(Y_i^{\text{obs}}\) among treated individuals estimates what the average of \(Y_i(1)\) would be for everyone
The average of \(Y_i^{\text{obs}}\) among control individuals estimates what the average of \(Y_i(0)\) would be for everyone
Random assignment ensures that the treated and control groups are, on average, comparable samples from the same population. This allows us to use observed group means as stand-ins for the unobservable population means.
Interpreting the Estimate in Context. Using the estimate from Exercise 8a (\(\widehat{\text{ATE}} = -1.8\)):
Write a one-sentence interpretation of this estimate in the context of Rasinski’s question-wording experiment. Be sure to explain what the negative sign means substantively.
A colleague says, “A change of 1.8 points on a 3-point scale is huge!” Is this a fair characterization? What context would help evaluate the magnitude?
Solution
Part a. Based on this experiment, respondents gave answers that were on average 1.8 points lower on the 1-to-3 scale (where 1 = “Too little” and 3 = “Too much”) when asked about “assistance to the poor” compared to “welfare,” indicating that the “assistance to the poor” framing substantially increases expressed support for government spending.
Part b. The colleague’s characterization captures the right intuition—this is a large effect—but deserves more context:
What makes this effect large:
The outcome scale only has three categories (1, 2, 3), so the maximum possible effect is 2 points (from “Too much” to “Too little” or vice versa)
An effect of 1.8 points is 90% of the maximum possible effect
Many respondents are shifting by a full category or more based solely on word choice
Context that helps evaluation:
The treatment is minimal—just changing one phrase in a question. The fact that word choice alone produces such a large shift suggests question wording is extremely consequential for survey results.
If this effect generalizes to the broader population, it means public opinion polls could show dramatically different levels of support for the same policy depending on how the question is worded.
In political terms, this could mean the difference between a policy appearing popular or unpopular.
So yes, the effect is substantively large, and the colleague’s reaction is warranted. Changing two words shifts responses by nearly the full width of the scale.
2.10.4 Critical Thinking
Unbiasedness vs. Accuracy. An estimator can be unbiased but still give estimates that are far from the true value.
How is this possible? What determines how far individual estimates might be from the truth?
What might make \(\widehat{\text{ATE}}\) more or less variable across different randomizations?
Solution
Part a. Unbiasedness only guarantees that the estimator is correct on average—it says nothing about any single estimate.
An unbiased estimator can produce estimates far from the truth because of sampling variability. Each randomization produces a different assignment of individuals to treatment and control. Some randomizations might, by chance, put individuals with unusually high potential outcomes into the treatment group, leading to overestimates. Other randomizations might do the opposite, leading to underestimates.
Think of it like flipping a fair coin 10 times. On average, you get 5 heads. But any particular sequence might give you 3 heads, 7 heads, or even 0 heads. The “estimator” (count the heads) is unbiased, but individual realizations vary.
What determines how far individual estimates might be from the truth:
The variance of the estimator—how spread out the distribution of possible estimates is
This variance depends on the sample size, the variability in potential outcomes, and the proportion assigned to treatment vs. control
Part b. Several factors affect how variable \(\widehat{\text{ATE}}\) is across randomizations:
Factors that increase variability:
Smaller sample sizes (\(N\)): Fewer individuals means each randomization can produce more extreme group compositions
More heterogeneous potential outcomes: If individuals differ greatly in their outcomes, the groups can be quite different depending on which individuals land where
Unbalanced designs: Having very few individuals in one group (e.g., \(N_t = 2\) out of \(N = 100\)) increases variability because the small group’s mean is very sensitive to which specific individuals are included
Factors that decrease variability:
Larger sample sizes: With many individuals, the law of large numbers kicks in—each group is more likely to be representative of the population
More homogeneous potential outcomes: If all individuals have similar outcomes, it matters less which ones end up in which group
Balanced designs: Having roughly equal numbers in treatment and control (\(N_t \approx N_c\)) tends to minimize variance
Breaking Unbiasedness. The proof of unbiasedness relies on the fact that \(E_D[D_i] = N_t/N\) for all individuals \(i\).
Give an example of an assignment mechanism where this would not hold.
If we used such a biased assignment mechanism, would \(\widehat{\text{ATE}}\) still be unbiased? Why or why not?
Solution
Part a. Here are examples of assignment mechanisms where \(E_D[D_i] = N_t/N\) would not hold for all individuals:
Example 1: Selection based on a characteristic
A researcher assigns older participants to treatment with probability 0.8 and younger participants with probability 0.2. If age is associated with the outcome, \(E_D[D_i]\) differs across individuals based on their age.
Example 2: Self-selection
Participants choose whether to receive treatment. Motivated participants might be more likely to choose treatment (\(E_D[D_i]\) is higher for motivated people).
Example 3: Researcher discretion
A researcher “randomly” assigns treatment but subconsciously tends to assign sicker patients to the new treatment. Sicker patients have higher \(E_D[D_i]\).
Example 4: Sequential assignment with learning
A researcher stops assigning to treatment once they see a few bad outcomes, making later individuals less likely to be treated.
Part b. No, \(\widehat{\text{ATE}}\) would generally not be unbiased under such mechanisms.
The proof of unbiasedness relied on a key cancellation: the probability of treatment (\(E_D[D_i] = N_t/N\)) canceled with the weighting in the estimator (dividing by \(N_t\)). This worked because every individual had the same probability of treatment.
When treatment probabilities vary across individuals, this cancellation breaks down. Individuals with higher probabilities of treatment contribute more to the treatment group mean, and individuals with lower probabilities contribute less. If these probabilities are correlated with potential outcomes, the estimator becomes biased.
For example, if high-outcome individuals have higher treatment probabilities:
The treatment group will be enriched with high-outcome individuals
The control group will be enriched with low-outcome individuals
\(\overline{Y}^{\text{obs}}_{t} - \overline{Y}^{\text{obs}}_{c}\) will overestimate the true ATE
The estimator would be capturing both the treatment effect and the selection effect, leading to bias.
Sample Size Considerations. Consider two experiments estimating the same ATE. Experiment A has \(N = 10\) with \(N_t = 5\). Experiment B has \(N = 1000\) with \(N_t = 500\).
Are both estimates unbiased for the ATE?
In which experiment would you expect \(\widehat{\text{ATE}}\) to be closer to the true ATE? Why?
What property of estimators (besides unbiasedness) captures this difference?
Solution
Part a. Yes, both estimates are unbiased for the ATE.
Unbiasedness depends only on the randomization procedure, not on the sample size. As long as treatment is randomly assigned (with each individual having equal probability), the estimator is unbiased. This is true whether \(N = 10\) or \(N = 1,000,000\).
The proof of unbiasedness holds for any \(N > 1\) with \(1 < N_t < N\).
Part b. We would expect \(\widehat{\text{ATE}}\) from Experiment B (the larger experiment) to be closer to the true ATE.
With only 10 individuals, random chance can produce quite unrepresentative groups. Maybe by luck, 4 of the 5 highest-outcome individuals all end up in treatment, producing a large overestimate. With 1000 individuals, such extreme imbalances are far less likely. The law of large numbers ensures that with more individuals, each group’s mean converges toward the population mean of that potential outcome.
To use a simple analogy: if you flip a coin 10 times, getting 8 heads (80%) wouldn’t be surprising. If you flip it 1000 times, getting 800 heads (80%) would be extremely unlikely. Larger samples are more stable.
Part c. The property that captures this difference is the variance (or equivalently, the standard error) of the estimator.
Both estimators are unbiased, but the estimator from the larger experiment has smaller variance—its sampling distribution is more concentrated around the true ATE. We often describe this by saying the larger experiment is more precise or has more efficiency.
Related concepts include:
Standard error: The standard deviation of the sampling distribution of \(\widehat{\text{ATE}}\)
Efficiency: An estimator with smaller variance is more efficient
Consistency: As \(N \to \infty\), the estimator converges to the true value (related to variance shrinking to zero)
2.10.5 R Practice
Computing \(\widehat{\text{ATE}}\) with mean(). Suppose you have a data frame called experiment with columns treatment (1 = treated, 0 = control) and outcome (a numeric outcome variable).
Write R code to compute \(\overline{Y}^{\text{obs}}_{t}\) using logical indexing.
Write R code to compute \(\overline{Y}^{\text{obs}}_{c}\) using logical indexing.
Write R code to compute \(\widehat{\text{ATE}}\) as the difference.
Solution
Part a. To compute \(\overline{Y}^{\text{obs}}_{t}\), the mean outcome among treated individuals:
Understanding lm() Output. A researcher fits the model lm(outcome ~ treatment, data = experiment) and gets the following output:
(Intercept) treatment
4.200 1.350
What is the mean outcome in the control group?
What is \(\widehat{\text{ATE}}\)?
What is the mean outcome in the treatment group?
Solution
Part a. The mean outcome in the control group is 4.200.
The intercept represents the predicted outcome when treatment = 0. Since the treatment variable is binary (0 or 1), plugging in 0 gives:
\[\hat{Y} = 4.200 + 1.350 \times 0 = 4.200\]
This is \(\overline{Y}^{\text{obs}}_{c}\).
Part b. The \(\widehat{\text{ATE}}\) is 1.350.
The coefficient on treatment represents how much the predicted outcome changes when treatment goes from 0 to 1. This is exactly the difference in means:
We can verify: \(\overline{Y}^{\text{obs}}_{t} - \overline{Y}^{\text{obs}}_{c} = 5.550 - 4.200 = 1.350\), which matches the coefficient.
Reference Categories. Explain why we used fct_relevel(desc, "welfare") in the R code for the mock Rasinski data. What would happen if we had not done this? Would the \(\widehat{\text{ATE}}\) computed using mean() and lm() still match?
Solution
We used fct_relevel(desc, "welfare") to set “welfare” as the reference category for the factor variable.
Why this matters for lm():
When lm() encounters a factor variable, it creates dummy (indicator) variables. R uses the first level of the factor as the reference category—the category that gets coded as 0. By default, R orders factor levels alphabetically. Since “assistance to the poor” comes before “welfare” alphabetically, it would be the reference category by default.
With “assistance to the poor” as reference:
The intercept would be the mean for “assistance to the poor” (the treatment group)
The coefficient would be the change when moving to “welfare” (the control group)
The coefficient would be \(\overline{Y}^{\text{obs}}_{c} - \overline{Y}^{\text{obs}}_{t}\), which is the negative of our usual definition
By using fct_relevel(desc, "welfare"), we make “welfare” the reference:
The intercept is the mean for “welfare” (control group)
The coefficient is the change when moving to “assistance to the poor” (treatment)
The coefficient equals \(\overline{Y}^{\text{obs}}_{t} - \overline{Y}^{\text{obs}}_{c}\), matching our definition
Would mean() and lm() still match?
The magnitudes would still match, but the signs might differ depending on how we set up the calculation:
If we compute mean() as treatment minus control and lm() uses the alphabetical default, the coefficient would have the opposite sign of our mean() calculation
The numerical values would still be correct in absolute value
In practice, for the two approaches to give the same number (not just the same magnitude), we need to ensure the reference category in lm() matches which group we subtract in the mean() calculation.
Connecting mean() and lm(). In your own words, explain why regressing the outcome on a binary treatment indicator produces a coefficient that equals the difference in means. What role does the intercept play in this relationship?
Solution
When we regress outcome on a binary treatment indicator, OLS (ordinary least squares) finds the coefficients that minimize the sum of squared residuals—the differences between observed outcomes and predicted outcomes.
The key insight:
For a binary predictor, the predictions can only take two values:
When \(D_i = 0\): \(\hat{Y}_i = \hat{\beta}_0\)
When \(D_i = 1\): \(\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1\)
OLS chooses \(\hat{\beta}_0\) and \(\hat{\beta}_1\) to make predicted values as close as possible to observed values. The best prediction for a group of observations is their mean (this minimizes squared errors within that group).
Therefore:
\(\hat{\beta}_0\) is set to the mean outcome in the control group, because that’s the best prediction for control individuals
\(\hat{\beta}_0 + \hat{\beta}_1\) is set to the mean outcome in the treatment group, because that’s the best prediction for treated individuals
This is exactly the difference in means—our \(\widehat{\text{ATE}}\).
The role of the intercept:
The intercept (\(\hat{\beta}_0\)) serves as the “baseline”—the predicted outcome for the reference group (control). It anchors the predictions. The coefficient (\(\hat{\beta}_1\)) then represents how much we add or subtract from this baseline when an individual is treated. This additive structure means the coefficient must equal the difference between the two group means.