Matching and weighting for quasi-experimental policy evaluation: a primer

Sessions and Times

One session, 09:30 to 12:30. 

Overview

Policymakers need to know whether social programmes and policies make a difference, and often when randomised experiments are unethical or unfeasible. Causal matching and weighting, types of quasi-experiments, can satisfy this need. Using a range of examples, this course will begin by exploring intuitive understandings of how matching and weighting methods work and progress towards an introduction to key concepts that underpin them. The course will demonstrate that a robust theory from the substantive topic area is an essential ingredient for quasi-experiments to deliver the conclusions that policymakers demand.

Topics Covered

  • The case for programme and policy evaluation and why evaluation matters and what it can offer policymakers.
  • A running example: estimating the average causal effect of an intervention when there is one comparison group and data from two time points, baseline and endpoint.
  • What a quasi-experiment is trying to estimate, couched in terms of the potential outcomes framework, and what you lose when you can’t run an RCT.
  • Exact matching, why it usually fails, and how to fix it through coarsening.
  • The idea of a multivariate distance and how distance metrics (Euclidean and Mahalanobis) can be used for matching.
  • The idea of a propensity score and how it can be used for matching and weighting.
  • Common themes when assessing matching/weighting analyses, regardless of the approach used.

Learning Outcomes

By the end of the course participants will be able to:

  • know where quasi-experiments fit into the broader landscape of evaluations.
  • make a compelling case for running quasi-experiments and how they differ from RCTs.
  • explain the potential outcomes framework and how it applies to quasi-experiments.
  • know what a causal estimand is and be able to choose an appropriate estimand for a given evaluation question.
  • understand the steps of estimating propensity scores and common ways to use them for matching and weighting.
  • understand a selection of approaches of matching directly on covariates (coarsened exact matching and distance-based matching).
  • appraise the quality of key elements of a quasi-experimental analysis that each of the methods introduced have in common.
  • engage with literature in the field with more confidence.

Target Audience

Participants should have a firm grasp of the foundation of quantitative methods used in social science, particularly linear and logistic regression and confidence intervals.  Attendees are encouraged Participants should have a specific idea for a quasi-experiment they would like to run.

Cost

This course is currently only available for in-house delivery, for further information and rates please contact us on natcenlearning@natcen.ac.uk.