Impact Evaluation (Midterm 3)

Front 1

Before and After Designs for Causal Inference

Back 1

• Investigators compare the same individual before and after the treatment is administered
  
• Intuitive designs that controls for between person confounding (as in the case-crossover designs
  

Front 2

Limitations to Before and After Design for Causal Inference

Back 2

B&A does not account for time-varying factors that could produce a spurious or biased association

Front 3

Enrolled and Not Enrolled Designs for Causal Inference

Back 3

Comparison groups are persons who are enrolled in the study and those that are not

Front 4

Limitations to Enrolled and Not Enrolled Designs for Causal Inference

Back 4

• Groups differ by more than just their enrollment status
  
• Unknown confounders could very well distort the relationship of interest
  

Front 5

Difference-in-Differences Design for Causal Inference

Back 5

• This method combines before and after as well as enrolled and not enrolled.
  
• Two subtractions are done, one to remove the temporal effects and the other to estimate the program's effect
  
• The method relies on generating a counterfactual slope for one group based on the other; but if this slope does not accurately reflect what would have happened, bias will occur
  

Front 6

Randomized Offering/ Promotion

Back 6

• The comparison groups are those that were offered the program and those that were not
  
• The randomization must occur at the smallest viable unit possible for program implementation
  
• pilot testing needed b/c of complexity of the study

Front 7

IV Estimator

Back 7

• change in Y / change in enrollment %
  
• gives an estimate of local average treatment effects, LATE.
  
• used when controlling for confounders is impossible due to the nature of the study/intervention, or when a controlled experiment is not possible.
  

Front 8

Randomized Allocation

Back 8

• Randomize the treatment, if ethically permissible and logistically manageable.
  
• As long as the groups are large enough and the randomization works, it is likely that exchangeability between the groups will hold.
  

Front 9

Discontinuity Design

Back 9

• A continuous eligibility index or criterion is used to admit persons into a program.
  
• Units just above and just below the eligibility criterion are assumed to be relatively comparable across a wide variety of factors, so they serve as appropriate comparison groups over time.
  
• This design leads to an unbiased estimate of the local treatment effect, as it can only give unbiased inference for the groups just above and just below the cutoff point.
  

Front 10

Matching

Back 10

• Investigators could match participant and non-participants on the basis observed characteristics, but this might lead to bias as a result of unmeasured factors. So instead, match persons on the basis of their propensity score.
  
• To do this successfully, a large number of units is usually required.
  
• Note that matching after the fact is inadvisable, as information at baseline might be missing, and matching on ex-post variables might lead to severe bias.
  

Front 11

Stepped Wedge / Phased Introduction / Phased Implementation / Step Wedge

Back 11

• This design is used in the context of a randomized, controlled, prospective program evaluation under the conditions that:The intervention is beneficial to the population (Note: This is NOT the assumption of equipoise!)Logistics would prevent rollout of the program in a single moment (i.e. a parallel arm trial is not logistically possible)
  
• The outcomes at the different times constitute the major units of analysis for this design
  

Front 12

Wanting the persons in the program and thecomparison group to have…

– Identical characteristics

– Except for benefiting from the intervention

Back 12

is the same thing as wanting exchangeabilityto hold. So everything we learned aboutcausal inference applies here.

Front 13

Exchangeability: Before and After

Back 13

Does not hold because there may be secular or other temporaltrends over the study period which introduce bias

Front 14

Exchangeability: Enrolled and Not Enrolled

Back 14

Does not hold because the two groups could be non-comparablebecause of confounders, most of which are unknown

Front 15

Calculate diff-in-diff impact

Back 15

Estimateprogram effects first,then remove the timeeffects

or

Remove the time effects first, thenestimate programeffects

Front 16

Running variable

Back 16

In discontinuity design, determines who gets the intervention and who does not

Those just above/below the threshold are similar and act as a counterfactual of each other

Estimate is only a local effect

Front 17

Assumptions required for causalinterpretation of IV estimates

Back 17

• Instrument must be related to exposure

• Must have random assignment of theinstrument

• SUTVA (stable unit treatment valueassumption)

• Monotonicity

• Exclusion restriction

Front 18

Validity

Back 18

Repeated measures center on the correctvalue. That is, a valid test is an unbiased one

Front 19

Reliability

Back 19

Repeated measures cluster together.