“What To Do (and Not to Do) with Causal Panel Analysis: Parallel Trends, Heterogeneous Treatment Effects, and Diagnostics”

Two-way fixed effects (TWFE) models are a popular choice for causal panel analysis in political science. Recent developments in the methodological literature repudiate the TWFE estimator in the presence of heterogeneous treatment effects (HTE) and violations of the parallel trends assumption; to diagnose or alleviate these issues, researchers have proposed a variety of new estimators and testing procedures. How much do these problems matter for empirical political science research and how do new proposals compare in practice? We answer these two questions by reviewing, replicating, and reanalyzing 34 papers recently published in American Political Science Review, American Journal of Political Science, and Journal of Politics that use observational panel data with binary treatments. Leveraging both visual and statistical tests, we find that parallel trends violations are a much bigger threat than HTE and that four newly-proposed HTE-robust estimators yield qualitatively similar findings when a strong pretrend is not observed. Moreover, underestimation of uncertainties, which leads to uncontrolled Type-I errors, remains a prominent issue in empirical panel analysis. Overall, we find that claims in one-third of the studies we review are strongly supported by data. We provide recommendations for empirical researchers to improve practice.