New working paper: "Do Modern Difference-in-Differences Estimators Change Published Findings in Business Research?"
Peter Hull's recent meta-analysis found that in economics, two-way fixed effects and heterogeneity-robust DiD estimators rarely disagree. Using Claude, I ran his protocol on seven business journals, screening 1,829 papers, and re-estimated every eligible design in Management Science and Marketing Science, the two with public replication archives. That gave 355 published staggered-adoption specifications from 40 papers, re-estimated with Callaway-Sant'Anna (CS) and Borusyak-Jaravel-Spiess (BJS). On the comparisons that pass Hull's leverage screen, from 28 (CS) and 29 (BJS) papers:
- Correlations between TWFE and modern estimates are 0.61 and 0.70, against 0.88 and 0.93 in economics.
- Estimates change sign in 25 and 18 percent of comparisons, and 10 and 7 percent show a statistically significant difference.
- Modern standard errors are 1.6 and 1.3 times those of TWFE.
- Disagreement concentrates in unbalanced panels and in designs without never-treated units.
Every table and figure rebuilds from the public repository; re-estimating a specification from scratch requires the journal's replication package.
Paper: lnkd.in/gXmtvfBC
Code, protocol, and estimates: lnkd.in/gZijx_dy
Peter Hull's recent meta-analysis found that in economics, two-way fixed effects and heterogeneity-robust DiD estimators rarely disagree. Using Claude, I ran his protocol on seven business journals, screening 1,829 papers, and re-estimated every eligible design in Management Science and Marketing Science, the two with public replication archives. That gave 355 published staggered-adoption specifications from 40 papers, re-estimated with Callaway-Sant'Anna (CS) and Borusyak-Jaravel-Spiess (BJS). On the comparisons that pass Hull's leverage screen, from 28 (CS) and 29 (BJS) papers:
- Correlations between TWFE and modern estimates are 0.61 and 0.70, against 0.88 and 0.93 in economics.
- Estimates change sign in 25 and 18 percent of comparisons, and 10 and 7 percent show a statistically significant difference.
- Modern standard errors are 1.6 and 1.3 times those of TWFE.
- Disagreement concentrates in unbalanced panels and in designs without never-treated units.
Every table and figure rebuilds from the public repository; re-estimating a specification from scratch requires the journal's replication package.
Paper: lnkd.in/gXmtvfBC
Code, protocol, and estimates: lnkd.in/gZijx_dy