Subgroup Analysis After Propensity Score Matching Using R

Today I’m going to demonstrate performing a subgroup analysis after propensity score matching using R. Subgroup analysis, also known as moderation analysis or the analysis of effect modification, concerns the estimation of treatment effects within subgroups of a pre-treatment covariate. This post assumes you understand how to do propensity score matching. For a general introduction to propensity score matching, I recommend Austin (2011) and the {MatchIt} introductory vignette. If you understand inverse probability weighting but aren’t too familiar with matching, I recommend my article with Liz Stuart (Greifer and Stuart 2021). For an introduction to subgroup analysis with propensity scores, you can also check out Green and Stuart (2014). Here, I’ll mainly try to get to the point.

The dataset we’ll use today is the famous Lalonde dataset, investigating the effect of a job training program on earnings. We’ll use the version of this dataset that comes with the {MatchIt} package.

data("lalonde", package = "MatchIt")
head(lalonde)
##      treat age educ   race married nodegree re74 re75       re78
## NSW1     1  37   11  black       1        1    0    0  9930.0460
## NSW2     1  22    9 hispan       0        1    0    0  3595.8940
## NSW3     1  30   12  black       0        0    0    0 24909.4500
## NSW4     1  27   11  black       0        1    0    0  7506.1460
## NSW5     1  33    8  black       0        1    0    0   289.7899
## NSW6     1  22    9  black       0        1    0    0  4056.4940

The treatment is treat, the outcome in the original study was re78 (1978 earnings), and the other variables are pretreatment covariates that we want to adjust for using propensity score matching. In this example, I’ll actually be using a different outcome, re78_0, which is whether the participant’s 1978 earnings were equal to 0 or not, because I want to demonstrate the procedure for a binary outcome. So, we hope the treatment effect is negative, i.e., the risk of 0 earnings decreases for those in the treatment.

lalonde$re78_0 <- as.numeric(lalonde$re78 == 0)

Our moderator will be race, a 3-category factor variable.

with(lalonde, table(race))
## race
##  black hispan  white 
##    243     72    299

Our estimand will be the subgroup-specific and marginal average treatment effect on the treated (ATT), using the risk difference as our effect measure.

Packages You’ll Need

We’ll need a few R packages for this analysis. We’ll need {MatchIt} and {optmatch} for the matching, {cobalt} for the balance assessment, {marginaleffects} for estimating the treatment effects, and {sandwich} for computing the standard errors. You can install those using the code below:

install.packages(c("MatchIt", "optmatch", "cobalt",
                   "marginaleffects", "sandwich"))

Let’s get into it!

Step 1: Subgroup Matching

Our first step is to perform the matching. Although there are a few strategies for performing matching for subgroup analysis, in general subgroup-specific matching tends to work best, though it requires a little extra work.

We’ll do this by splitting the dataset by race and performing a separate matching analysis within each one.

#Splitting the data
lalonde_b <- subset(lalonde, race == "black")
lalonde_h <- subset(lalonde, race == "hispan")
lalonde_w <- subset(lalonde, race == "white")

Here we’ll use full matching because 1:1 matching without replacement, the most common (but worst) way to do propensity score matching, doesn’t work well in this dataset. The process described below works exactly the same for 1:1 and most other kinds of matching as it does for full matching. We’ll estimate propensity scores in each subgroup, here using probit regression, which happens to yield better balance than logistic regression does.

library("MatchIt")

#Matching in race == "black"
m.out_b <- matchit(treat ~ age + educ + married + nodegree + re74 + re75,
                   data = lalonde_b, method = "full", estimand = "ATT",
                   link = "probit")

#Matching in race == "hispan"
m.out_h <- matchit(treat ~ age + educ + married + nodegree + re74 + re75,
                   data = lalonde_h, method = "full", estimand = "ATT",
                   link = "probit")

#Matching in race == "black"
m.out_w <- matchit(treat ~ age + educ + married + nodegree + re74 + re75,
                   data = lalonde_w, method = "full", estimand = "ATT",
                   link = "probit")

Step 2: Assessing Balance within Subgroups

We need to assess subgroup balance; we can do that using summary() on each matchit object, or we can use functions from {cobalt}.

Below are examples of using summary() and cobalt::bal.tab() on one matchit object at a time1:

summary(m.out_b)
## 
## Call:
## matchit(formula = treat ~ age + educ + married + nodegree + re74 + 
##     re75, data = lalonde_b, method = "full", link = "probit", 
##     estimand = "ATT")
## 
## Summary of Balance for All Data:
##          Means Treated Means Control Std. Mean Diff. Var. Ratio eCDF Mean eCDF Max
## distance        0.6587        0.6121          0.4851     0.7278    0.1134   0.1972
## age            25.9808       26.0690         -0.0121     0.4511    0.0902   0.2378
## educ           10.3141       10.0920          0.1079     0.5436    0.0336   0.0807
## married         0.1859        0.2874         -0.2608          .    0.1015   0.1015
## nodegree        0.7244        0.6437          0.1806          .    0.0807   0.0807
## re74         2155.0132     3117.0584         -0.1881     0.9436    0.0890   0.2863
## re75         1490.7221     1834.4220         -0.1043     1.0667    0.0480   0.1441
## 
## Summary of Balance for Matched Data:
##          Means Treated Means Control Std. Mean Diff. Var. Ratio eCDF Mean eCDF Max Std. Pair Dist.
## distance        0.6587        0.6579          0.0078     1.0457    0.0094   0.0705          0.0379
## age            25.9808       27.3671         -0.1899     0.3647    0.1147   0.2276          1.4295
## educ           10.3141       10.1746          0.0677     0.6920    0.0178   0.0485          1.0432
## married         0.1859        0.1825          0.0088          .    0.0034   0.0034          0.6579
## nodegree        0.7244        0.7259         -0.0033          .    0.0015   0.0015          0.7593
## re74         2155.0132     2959.6631         -0.1574     0.7828    0.0525   0.2156          0.7246
## re75         1490.7221     2158.7331         -0.2026     0.8901    0.0877   0.1983          0.8399
## 
## Sample Sizes:
##               Control Treated
## All             87.       156
## Matched (ESS)   36.64     156
## Matched         87.       156
## Unmatched        0.         0
## Discarded        0.         0
library("cobalt")
bal.tab(m.out_b, un = TRUE, stats = c("m", "ks"))
## Call
##  matchit(formula = treat ~ age + educ + married + nodegree + re74 + 
##     re75, data = lalonde_b, method = "full", link = "probit", 
##     estimand = "ATT")
## 
## Balance Measures
##              Type Diff.Un  KS.Un Diff.Adj KS.Adj
## distance Distance  0.4851 0.1972   0.0078 0.0705
## age       Contin. -0.0121 0.2378  -0.1899 0.2276
## educ      Contin.  0.1079 0.0807   0.0677 0.0485
## married    Binary -0.1015 0.1015   0.0034 0.0034
## nodegree   Binary  0.0807 0.0807  -0.0015 0.0015
## re74      Contin. -0.1881 0.2863  -0.1574 0.2156
## re75      Contin. -0.1043 0.1441  -0.2026 0.1983
## 
## Sample sizes
##                      Control Treated
## All                    87.       156
## Matched (ESS)          36.64     156
## Matched (Unweighted)   87.       156

We can also get a clearer sense of balance overall using bal.tab() by directly supplying the matching weights.

#Initialize the weights
fm_weights <- numeric(nrow(lalonde))

#Assign the weights based on the subgroup
fm_weights[lalonde$race == "black"] <- m.out_b$weights
fm_weights[lalonde$race == "hispan"] <- m.out_h$weights
fm_weights[lalonde$race == "white"] <- m.out_w$weights

bal.tab(treat ~ age + educ + married + nodegree + re74 + re75,
        data = lalonde, weights = fm_weights, cluster = "race",
        stats = c("m", "ks"), abs = TRUE, cluster.summary = TRUE)
## Balance by cluster
## 
##  - - - Cluster: black - - - 
## Balance Measures
##             Type Diff.Adj KS.Adj
## age      Contin.   0.1899 0.2276
## educ     Contin.   0.0677 0.0485
## married   Binary   0.0034 0.0034
## nodegree  Binary   0.0015 0.0015
## re74     Contin.   0.1574 0.2156
## re75     Contin.   0.2026 0.1983
## 
## Effective sample sizes
##            Control Treated
## Unadjusted   87.       156
## Adjusted     36.64     156
## 
##  - - - Cluster: hispan - - - 
## Balance Measures
##             Type Diff.Adj KS.Adj
## age      Contin.   0.2306 0.1848
## educ     Contin.   0.2868 0.2783
## married   Binary   0.0564 0.0564
## nodegree  Binary   0.1089 0.1089
## re74     Contin.   0.1028 0.3034
## re75     Contin.   0.1422 0.2343
## 
## Effective sample sizes
##            Control Treated
## Unadjusted   61.        11
## Adjusted     26.51      11
## 
##  - - - Cluster: white - - - 
## Balance Measures
##             Type Diff.Adj KS.Adj
## age      Contin.   0.4541 0.2084
## educ     Contin.   0.4598 0.1982
## married   Binary   0.0180 0.0180
## nodegree  Binary   0.1736 0.1736
## re74     Contin.   0.2537 0.4014
## re75     Contin.   0.0872 0.1476
## 
## Effective sample sizes
##            Control Treated
## Unadjusted   281.       18
## Adjusted      50.4      18
##  - - - - - - - - - - - - - - 
## 
## Balance summary across all clusters
##             Type Mean.Diff.Adj Max.Diff.Adj Mean.KS.Adj Max.KS.Adj
## age      Contin.        0.2916       0.4541      0.2069     0.2276
## educ     Contin.        0.2714       0.4598      0.1750     0.2783
## married   Binary        0.0259       0.0564      0.0259     0.0564
## nodegree  Binary        0.0946       0.1736      0.0946     0.1736
## re74     Contin.        0.1713       0.2537      0.3068     0.4014
## re75     Contin.        0.1440       0.2026      0.1934     0.2343
## 
## Total effective sample sizes across clusters
##            Control Treated
## Unadjusted  429.       185
## Adjusted    113.55     185

Using the cluster argument produces balance tables in each subgroup and, because we specified cluster.summary = TRUE, a balance table summarizing across subgroups. To suppress display of the subgroup-specific balance tables (which may be useful if you have many subgroups), you can specify which.cluster = .none.

To make a plot displaying the balance statistics visually, we can use cobalt::love.plot():

love.plot(treat ~ age + educ + married + nodegree + re74 + re75,
        data = lalonde, weights = fm_weights, cluster = "race",
        stats = c("m", "ks"), abs = TRUE,
        which.cluster = .none, agg.fun = "max")
## Warning: Standardized mean differences and raw mean differences are present in the same plot. 
## Use the 'stars' argument to distinguish between them and appropriately label the x-axis.

See the {cobalt} vignette on customizing love.plot() to see how to finely control the appearance of the plot.

From this output, we can see that balance is actually pretty bad; the greatest standardized mean difference (SMD) across subgroups after matching is around .46, which is way too big. In a realistic scenario, we would try different matching methods, maybe resorting to weighting, until we found good balance across the subgroups. In order to validly interpret the subgroup-specific effects and tests for moderation, we need to achieve balance in each subgroup, not just overall. We didn’t get good balance here, but to stay focused on the rest of the procedure, we’ll move forward as if we did.

Step 3: Fitting the Outcome Model

Next, we’ll fit the outcome model. It’s important to remember that the outcome model is an intermediate step for estimating the treatment effect; no quantity estimated by the model needs to correspond to the treatment effect directly. We’ll be using a marginal effects procedure to estimate the treatment effects in the next section.

First, we’ll extract the matched datasets from the matchit objects. We can’t just use the matching weights we extracted earlier because we also need subclass (i.e., pair) membership. We’ll use match.data() from {MatchIt} to extract the matched datasets, which contain the matching weights and subclass membership in the weights and subclass columns, respectively, and use rbind() to bind them into a single combined dataset2.

#Extract the matched datasets
matched_data_b <- match.data(m.out_b)
matched_data_h <- match.data(m.out_h)
matched_data_w <- match.data(m.out_w)

#Combine them using rbind()
matched_data <- rbind(matched_data_b,
                      matched_data_h,
                      matched_data_w)

names(matched_data)
##  [1] "treat"    "age"      "educ"     "race"     "married"  "nodegree" "re74"     "re75"     "re78"     "re78_0"   "distance" "weights"  "subclass"

Next, we can fit the outcome model. The choice of which model to fit should depend primarily on the best model for the outcome, though in some cases it can be useful to parameterize the model to get a quantity you want for free. We’ll focus on using a good basic model; because we have a binary outcome, we’ll use logistic regression.

It’s usually a good idea to include covariates in the outcome model. It’s also usually a good idea to allow the treatment to interact with the covariates in the outcome model. It’s also usually a good idea to fit separate models within each subgroup. Combining this all yields a pretty complicated model, which is why it will be so important to use a marginal effects procedure rather than trying to interpret the model’s coefficients. Here’s how we fit this model:

fit <- glm(re78_0 ~ race * (treat * (age + educ + married + nodegree +
                                       re74 + re75)),
           data = matched_data, weights = weights,
           family = "quasibinomial")

We’re not even going to look at the output of this model, which has 42 parameters. If the model doesn’t fit with your dataset, you can remove interactions between the treatment and some covariates or remove the covariates altogether.

For a linear model, you can use lm() and remove the family argument. We used family = "quasibinomial" because we want logistic regression for our binary outcome but we are using the matching weights, which otherwise create a (harmless but annoying) warning when run with family = "binomial".

Step 4: Estimate the Treatment Effects

Finally, we can estimate the treatment effects. To do so, we’ll use an average marginal effects procedure as implemented in {marginaleffects}. First, we’ll estimate the average marginal effect overall, averaging across the subgroups. Again, we’re hoping for a negative treatment effect, which indicates the risk of having zero income decreased among those who received the treatment. Because we are estimating the ATT, we need to subset the data for which the average marginal effects are computed to just the treated units, which we do using the newdata argument (which can be omitted when the ATE is the target estimand). We also need to supply pair membership to ensure the standard errors are correctly computed, which we do by supplying the subclass variable containing pair membership to the vcov argument. In general, we need to supply the weights to the wts argument of comparisons() as well (though, in this case, because we are estimating the ATT and all weights are 1 for the treated group, it doesn’t make a difference).

library("marginaleffects")

#Estimate the overall ATT
comparisons(fit, variables = "treat",
            newdata = subset(matched_data, treat == 1),
            vcov = ~subclass, wts = "weights") |>
  summary()
##    Term Contrast  Effect Std. Error z value Pr(>|z|)    2.5 % 97.5 %
## 1 treat    1 - 0 0.02305    0.04369  0.5275  0.59785 -0.06259 0.1087
## 
## Model type:  glm 
## Prediction type:  response

The estimated risk difference is 0.02305 with a high p-value and a confidence interval containing 0, indicating no evidence of an effect overall. (Note: this doesn’t mean there is no effect! The data are compatible with effects anywhere within the confidence interval, which includes negative and positive effects of a moderate size!)

New, let’s estimate the subgroup-specific effects by supplying the subgrouping variable, "race", to the by argument:

comparisons(fit, variables = "treat",
            newdata = subset(matched_data, treat == 1),
            vcov = ~subclass, wts = "weights",
            by = "race") |>
  summary()
##    Term          Contrast   race   Effect Std. Error z value Pr(>|z|)    2.5 %  97.5 %
## 1 treat mean(1) - mean(0)  black  0.05871    0.05122   1.146 0.251709 -0.04168  0.1591
## 2 treat mean(1) - mean(0) hispan -0.19461    0.07756  -2.509 0.012098 -0.34661 -0.0426
## 3 treat mean(1) - mean(0)  white -0.15304    0.04834  -3.166 0.001545 -0.24777 -0.0583
## 
## Model type:  glm 
## Prediction type:  response

Here, we see that actually there is evidence of treatment effects within subgroups! In the subgroups hispan and white, we see moderately sized negative effects with small p-values and confidence intervals excluding 0, suggesting that there treatment effects in these subgroups.

We can also test whether the treatment effects differ between groups using the hypothesis argument of comparisons():

comparisons(fit, variables = "treat",
            newdata = subset(matched_data, treat == 1),
            vcov = ~subclass, wts = "weights",
            by = "race", hypothesis = "pairwise") |>
  summary()
##            Term   Effect Std. Error z value  Pr(>|z|)   2.5 %   97.5 %
## 1 Row 2 - Row 1 -0.25332    0.09294 -2.7255 0.0064204 -0.4355 -0.07115
## 2 Row 3 - Row 1 -0.21175    0.07043 -3.0066 0.0026422 -0.3498 -0.07371
## 3 Row 3 - Row 2  0.04157    0.09139  0.4549 0.6491737 -0.1375  0.22068
## 
## Model type:  glm 
## Prediction type:  response

We can see evidence that the treatment effect differs between the black (Row 1) and hispan (Row 2) groups, and between the black and white (Row 3) groups. With many subgroups, it might be useful to adjust your p-values for multiple comparisons, which we can do using p.adjust(), e.g.,

p.adjust(comp$p.value, method = "holm")

if comp contained the comparisons() |> summary() output above.

Congratulations! You’ve done a subgroup analysis!

Step 5: Reporting Your Results

A fair bit needs to be included when reporting your results to ensure your analysis is replicable and can be correctly interpreted by your audience. The key things to report are the following:

  • The method of estimating the propensity score and performing the matching (noting that these were done within subgroups), including the estimand targeted and whether that estimand was respected by the procedure (using, e.g., a caliper changes the estimand from the one you specify). This should also include the packages used and, even better, the functions used. If you’re using {MatchIt}, the documentation should also tell you which papers to cite.
  • A quick summary of other methods you might have tried and why you went with the one you went with (i.e., because it yielded better balance, a greater effective sample size, etc.).
  • Covariate balance, measured broadly; this can include a balance table, a balance plot (like one produced by cobalt::love.plot()), or a summary of balance (like providing the largest SMD and KS statistic observed across subgroups). Make sure your description of balance reflects the subgroups, e.g., by having separate tables or plots for each subgroup or clarifying that the statistics presented are averages or the worst case across subgroups.
  • The outcome model you used, especially specifying the form of the model used and how/whether covariates entered the model. Also mention the method used to compute the standard errors (e.g., cluster-robust standard errors with pair membership as the clustering variable).
  • Details of the marginal effects procedure used, including the package used, and the method to compute the standard errors (in this case, the delta method, which is the only method available in {marginaleffects}).
  • The treatment effect estimates along with their p-values and confidence intervals, both overall and within subgroups.

References

Austin, Peter C. 2011. “An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies.” Multivariate Behavioral Research 46 (3): 399–424. https://doi.org/10.1080/00273171.2011.568786.
Green, Kerry M., and Elizabeth A. Stuart. 2014. “Examining Moderation Analyses in Propensity Score Methods: Application to Depression and Substance Use.” Journal of Consulting and Clinical Psychology, Advances in Data Analytic Methods, 82 (5): 773–83. https://doi.org/10.1037/a0036515.
Greifer, Noah, and Elizabeth A Stuart. 2021. “Matching Methods for Confounder Adjustment: An Addition to the Epidemiologist’s Toolbox.” Epidemiologic Reviews, June, mxab003. https://doi.org/10.1093/epirev/mxab003.

  1. You might notices the mean differences for binary variables differ between the two outputs; that’s because summary() standardizes the mean differences whereas bal.tab() does not for binary variables. If you want standardized mean differences for binary variables from bal.tab(), just add the argument binary = "std".↩︎

  2. Note: rbind() must be used for this; functions from other packages, like dplyr::bind_rows(), will not correctly preserve the subclass structure.↩︎

Noah Greifer
Noah Greifer
Statistical Consultant and Programmer