Week 4
Causal Inference

Sociβ€”316

Sakeef M. Karim
Amherst College

SOCIAL RESEARCH

Does X \to Y?
September 22nd

An Update

Qualtrics Survey Assignment

Assignment instructions are live.

Advancing Causal Claims

Cause and Effect

Social scientists are often interested in drawing causal inferences.

X \to Y

Cause and Effect

About what, exactly?

There are a dizzying array of examples.

Here are three questions that invite causal claims:

Three Causal Questions

  1. Do smaller class sizes improve pedagogical outcomes?

  2. Will investments in family planning programs improve economic and health outcomes for women in low-income settings?

  3. Did the cultural grievances of the β€œmiddle class” trigger the rise of fascist politics in interwar Europe?

Cause and Effect

Let’s pause for a second.

Try to frame your research question in causal terms.

Rearrange the text in the box below to arrive at your question.

Your Research Question
10:00
0 words

Cause and Effect

We can, of course, deploy quantitative tools (e.g., estimators, weighting, causal diagrams, experiments) to β€œresolve” causal questionsβ€”or at least adduce evidence grounded in statistical reasoning.

For an overview of how causality β€œworks” in ethnographic research, see Small (2013).

Cause and Effect

Let’s assume that our outcomes of substantive interest are approximately linear. Can we estimate linear regression models to β€œresolve” our causal questions?

y = \beta_0 + \beta_1 x + \epsilon

The Answer

It depends.

Cause and Effect

The Class Size Example

Cause and Effect

The Class Size Example

How can we deal with confoundingβ€”i.e., to estimate the causal effect of class size at Amherst
on post-graduate outcomes?

Randomization is the gold standard.

We will return to this point in a second.

Some Technical Details

Treatments and Outcomes

Causal relationships refer to the cause-and-effect connection between two variables … We refer to the variable where the change originates as the treatment variable. We refer to the variable that may change in response to the change in the treatment variable as the outcome variable.

(Llaudet and Imai 2023:28, EMPHASIS ADDED)

Treatments and Outcomes

Under the Assumption of a Binary Treatment

X_i= \begin{cases} 1 & \text{if individual } i \text{ receives the treatment}\\ 0 & \text{if individual } i \text{ does not receive the treatment} \end{cases}

small_i= \begin{cases} 1 & \text{if individual } i \text{ assigned to small classes at Amherst}\\ 0 & \text{if individual } i \text{ was not assigned to small classes at Amherst} \end{cases}

Treatments and Outcomes

When estimating the causal effect of X on Y, we attempt to quantify the change in the outcome variable Y that is caused by a change in the treatment variable X … In mathematical notation, we represent change with \Delta (the Greek letter Delta), and thus, we represent a change in the outcome as \Delta Y. To measure this change in the outcome Y, ideally we would compare two potential outcomes: the outcome when the treatment is present and the outcome when the treatment is absent.

(Llaudet and Imai 2023:29–30, EMPHASIS ADDED)

Treatments and Outcomes

  • Y_{i}(X_{i}=1) is the potential outcome under the treatment condition for individual i (the value of Y_{i} if X_{i}=1).

  • Y_{i}(X_{i}=0) is the potential outcome under the control condition for individual i (the value of Y_{i} if X_{i}=0).

(Llaudet and Imai 2023:30, EMPHASIS ADDED)

\Delta income_{i} = \underbrace{income_{i}\!\left(small_{i}=1\right)}_{\text{Treatment}} - \underbrace{income_{i}\!\left(small_{i}=0\right)}_{\text{Control}}

Treatments and Outcomes

If we could observe both potential outcomes

\textit{Individual effect}_i \;=\; \Delta Y_i \;=\; {\color{#856cb0}{Y_i(X_i{=}1)}} \;-\; {\color{#8a8494}{Y_i(X_i{=}0)}}

where

\Delta Y_i

The change in the outcome individual i would have experienced by receiving the treatment, compared to not receiving it.

{\color{#856cb0}{Y_i(X_i{=}1)}}
{\color{#8a8494}{Y_i(X_i{=}0)}}

The two potential outcomes for the same individual i, under the treatment and control conditions, respectively.

Adaptation of summary box in Llaudet and Imai (2023:30).

Alternate Worlds

Unfortunately, this kind of analysis is not possible. In the real world, we never observe both potential outcomes for the same individual. Instead, we observe only the factual outcome, which is the potential outcome under whichever condition (treatment or control) was received in reality. We can never observe the counterfactual outcome, which is the potential outcome that would have occurred under whichever condition (treatment or control) was not received in reality. As a result, we cannot compute causal effects at the individual level.

(Llaudet and Imai 2023:32, EMPHASIS ADDED)

Alternate Worlds

The fundamental problem of causal inference

To measure causal effects, we need to compare the factual outcome with the counterfactual outcome,

but we can never observe the counterfactual outcome.

Adaptation of summary box in Llaudet and Imai (2023:33).

Alternate Worlds

Alas, this can’t happen in real life for person i

Image can be retrieved here.

Approximating Counterfactuals

Average Treatment Effects (ATEs)

To get around the fundamental problem of causal inference, we must find good approximations for the counterfactual outcomes. To accomplish this, we move away from individual-level effects and focus on the average causal effect across a group of individuals.

The average causal effect of the treatment X on the outcome Y, also known as the average treatment effect, is the average of all the individual causal effects of X on Y within a group. Since each individual causal effect is the change in Y caused by a change in X for a particular individual, the average causal effect of X on Y is the average change in Y caused by a change in X for a group of individuals.

(Llaudet and Imai 2023:33, EMPHASIS ADDED)

Average Treatment Effects (ATEs)

If we could observe both potential outcomes

{\color{#856cb0}{\overline{\textit{Individual effects}}}} \;=\; \frac{\sum_{i=1}^{n} \textit{Individual effects}_{i}}{n}

where

{\color{#856cb0}{\overline{\textit{Individual effects}}}}

The average treatment effect for the observations in the study.

\textit{Individual effects}_{i}

The individual treatment effect for observation i.

\sum_{i=1}^{n} \textit{Individual effects}_{i}

The sum of all \textit{Individual effects}_{i}, from i=1 to i=n.

n

The number of observations in the study.

Adaptation of summary box in Llaudet and Imai (2023:34).

Average Treatment Effects (ATEs)

How can we obtain good approximations for the counterfactual outcomes, which by definition cannot be observed? As we will see in detail soon, we must find or create a situation in which the treated observations and the untreated observations are similar with respect to all the variables that might affect the outcome other than the treatment variable itself. The best way to accomplish this is by conducting a randomized experiment.

(Llaudet and Imai 2023:34, EMPHASIS ADDED)

Random Assignment

In a randomized experiment, also known as a randomized controlled trial (RCT), researchers decide who receives the treatment based on a random process …

  • the treatment group consists of the individuals who received the treatment (the group of observations for which X_{i}=1)
  • the control group consists of the individuals who did not receive the treatment (the group of observations for which X_{i}=0)

(Llaudet and Imai 2023:34, EMPHASIS ADDED)

Random Assignment

In Sociβ€”316

Random Assignment

The Class Size Example

All students
Treatmentsmall classes,
Controlregular classes,
Major
Treatment
Control
Difference, pp (bar spans Β±10)
Mean difference

Random Assignment

When treatment assignment is randomized, the only thing that distinguishes the treatment group from the control group, besides the reception of the treatment, is chance. This means that although the treatment and control groups consist of different individuals, the two groups are comparable to each other, on average, in all respects other than whether or not they received the treatment.

Random treatment assignment makes the treatment and control groups on average identical to each other in all observed and unobserved pre-treatment characteristics. Pre-treatment characteristics are the characteristics of the individuals in a study before the treatment is administered.

(Llaudet and Imai 2023:36, EMPHASIS ADDED)

Random Assignment

Randomization of treatment assignment

By randomly assigning treatment, we ensure that treatment and control groups are, on average,

identical to each other in all observed and unobserved pre-treatment characteristics.

Adaptation of summary box in Llaudet and Imai (2023:36).

Random Assignment

With randomization, this holds for two populations, C (X_{i}=0) and  T (X_{i}=1)

Image can be retrieved here.

Random Assignment and ATEs

If the treatment and control groups were comparable before the treatment was administered … we can use the factual outcome of one group as an approximation for the counterfactual outcome of the other. In other words, we can assume that the average outcome of the treatment group is a good estimate of the average outcome of the control group, had the control group received the treatment. Similarly, we can assume that the average outcome of the control group is a good estimate of the average outcome of the treatment group, had the treatment group not received the treatment. As a result, we can approximate the average treatment effect by computing the difference in the average outcomes between the treatment and control groups. Since both of these average outcomes are observed, this is an analysis we are able to perform.

(Llaudet and Imai 2023:37, EMPHASIS ADDED)

Random Assignment and ATEs

If groups were comparable before the treatment was administered

\widehat{\textit{ATE}} \;=\; {\color{#856cb0}{\overline{Y}_{\text{Treatment group},\,T}}} \;-\; {\color{#8a8494}{\overline{Y}_{\text{Control group},\,C}}}

where

\widehat{\textit{ATE}}

The estimated average treatment effect. The hat marks it as an estimate.

{\color{#856cb0}{\overline{Y}_{\text{Treatment group},\,T}}}

The observed average outcome for the treatment group.

{\color{#8a8494}{\overline{Y}_{\text{Control group},\,C}}}

The observed average outcome for the control group.

Adaptation of summary box in Llaudet and Imai (2023:37).

Random Assignment and ATEs

Estimating average causal effects using randomized experiments and the difference-in-means estimator

By using random treatment assignment, we can assume that the treatment and control groups were comparable before the administration of the treatment. As a result,

we can rely on the difference-in-means estimator to provide a valid estimate of the average treatment effect.

Adaptation of summary box in Llaudet and Imai (2023:38).

A Class Conversation

The Challenges of Causal Inference

As Llaudet and Imai (2023:38) note, social scientists often run into ethical, logistical, and financial obstacles that ensure that fielding a randomized experiment is challengingβ€”if not impossible. With this in mind, think aboutβ€”and refineβ€”your own causal question.

Orienting Questions

Can you randomly sort respondents into C (X_{i}=0) and  T (X_{i}=1)? How might you go about doing so? What are some challenges you may run into?

Your Updated Research Question
10:00
0 words

Experiments, DAGsβ€”
September 24th

Your Next Assignment

Qualtrics Survey Assessment

An Invitation from LJST

Click to Expand Invitation

High-Level Overview Experiments

Experiments

An experiment is a research method where the researcher manipulates one or more independent variables to determine the effect(s) on a dependent variable. Experiments share three key features: (1) manipulation of the independent variable, (2) random assignment of participants to experimental and control conditions, and (3) experimental control of other factors that could influence the outcome of the experiment.

(Carr et al. 2020:230, EMPHASIS ADDED)

Experiments

Experiments provide leverage in three important ways.

Three Key Affordances

  1. Experiments are ideal for adjudicating causal claims.

  2. Experiments compel researchers to test specific posited mechanisms, i.e. beyond reporting descriptive associations.

  3. Experiments can be used to test highly abstract theoriesβ€”generally through fine control of the experimental setting.

A Question

What are some of the weaknesses of experiments?

Experiments

Experimental Method
Strengths
Weaknesses
Laboratory experiments
Researcher has maximum control over the experimental setting and can ensure that the manipulation of the independent variable was salient to all participants.
The artificial setting allows researchers to test more abstract theories and to disentangle complex social phenomena.
Most laboratory experiments rely on nonrandom samples, so results are less generalizable.
Because the laboratory environment is highly artificial, lab experiments are less effective for exploring more applied, β€œreal world” questions.
Field experiments
Because field experiments take place in a real-world setting, they are good at evaluating more applied questions and for assessing whether an outcome, such as discrimination, occurs.
Researchers have less control over the experimental setting and frequently rely on nonrandom or regional samples.
Less suited for evaluating the processes that produce an outcome, and less useful for testing abstract theories.
Survey experiments
Survey experiments often use representative samples, so results are generalizable.
Researchers have less control over the experimental setting.
Often rely on self-reports, which do not always correspond to actual behavior.
Natural experiments
Realistic setting allows researchers to draw strong conclusions about specific settings.
Good for assessing whether an outcome occurs.
Researchers cannot randomly assign participants to conditions.
Settings are often so realistic that the results are tied to the setting in which the study was conducted and are not generalizable.
Less suited for evaluating the processes that produce an outcome.

Adapted from Table 8.1 in Carr et al. (2020)

Experiments

An Example

  1. 01Vignette
  2. 02Outcomes

John is a 44-year-old white man living in a suburb of Grand Rapids, Michigan. He lives with his wife and their two teenage children. He works as a facilities manager for a regional healthcare network. He coaches youth soccer on weekends and attends church a few times a year. He owns his home and follows local and national news when he can. He voted in the last two general elections but tends to skip the primaries. John is a .

Counterfactual arm Β· not the one this respondent saw

Q1How likely is it that John has a college degree?

Very unlikely
Very likely

Q2How similar is John to people like you?

Not at all similar
Very similar

Experiments

Let’s pause for a second.

Design a survey experiment linked to your research question.

Rearrange the text in the box below to arrive at your pitch.

An Experimental Pitch
10:00
0 words

High-Level Overview Directed Acyclic Graphs (DAGs)

DAGs

Directed acyclic graphs (DAGs) offer a powerful visual framework for representing and reasoning about causal relationships. At their core, DAGs capture the directional flow of causalityβ€”each arrow represents an immediate cause-and-effect relationship between variables, and the absence of cycles reflects the unidirectional nature of time and causation. While social science theories often allow for reverse causality or feedback loops, DAGs require breaking these cycles into sequential events, making causality more explicit and analyzable.

(Cunningham 2026, EMPHASIS ADDED)

DAGs

Despite their simplicity, DAGs are grounded in the same counterfactual framework that underpins potential outcomes. A causal effect is still defined as a comparison between two states of the world: the observed outcome under one intervention and the counterfactual outcome under another. This connection makes DAGs a natural complement to potential outcomes by offering a graphical representation of various assumptions we must make when identifying effects. In fact, this is precisely one of the strengths of this approach β€” it forces the researcher to think long and hard about the treatment assignment mechanisms, which are one of the main factors that dictate whether one research design is more suitable than another.

(Cunningham 2026, EMPHASIS ADDED)

DAGs

Example from Cunningham (2026)

DAGs

Correlation \neq Causation

DAGs

Do Not Condition on a Collider

DAGs

In the next 5-10 minutes, come up with a DAG related to your research project!

Enjoy the Weekend

References

Carr, Deborah S., Elizabeth Heger Boyle, Benjamin Cornwell, Shelley J. Correll, Robert Crosnoe, et al. 2020. The Art and Science of Social Research. Second Edition. New York: W. W. Norton & Company, Inc.
Cunningham, Scott. 2026. Causal Inference: The Remix. 2nd ed. New Haven, CT: Yale University Press.
Llaudet, Elena, and Kōsuke Imai. 2023. Data Analysis for Social Science: A Friendly and Practical Introduction. Princeton (N.J.) Oxford: Princeton University press.
Small, Mario Luis. 2013. β€œCausal Thinking and Ethnographic Research.” American Journal of Sociology 119(3):597–601. doi: 10.1086/675893.