Policy Impact Modeling

🔍 Evaluating Policy Impacts with Synthetic Control Methods: Recent Advances and Tools How do we know if the policies we create have the effects we desire? What if a policy is enacted in only one place (like a state) but not others? Enter the Synthetic Control Method (SCM) which offers a useful framework for these kinds of problems, especially when randomized experiments are infeasible (which is most of the time). 📘 Foundations of Synthetic Controls Introduced by Abadie and Gardeazabal (2003) and further developed by Abadie, Diamond, and Hainmueller (2010), SCM constructs a weighted combination of control units to approximate the counterfactual of a treated unit. If you are not familiar with SCM, a great place to start is Scott Cunningham's excellent book Causal Inference: The Mixtape: https://lnkd.in/g7YMb7KT. If you're interested in more advanced methods, keep reading. 1. Augmented Synthetic Control Method (ASCM): Combines SCM with regression adjustments to improve estimation accuracy. Reference: Ben-Michael, Feller, & Rothstein (2021). 2. Generalized Synthetic Control (GSC): Extends SCM to accommodate multiple treated units and time-varying effects. Reference: Xu (2017). 3. Synthetic Diff in Diff (synthdid): Accounts for staggered adoption or variable treatment times. Reference: Arkhangelsky et al. (2021). 🛠️ R Packages for Implementation Synth: https://lnkd.in/gQAc4NU3 augsynth: https://lnkd.in/gqzqs6EV gsynth: https://lnkd.in/gmAzuS3k synthdid: https://lnkd.in/gWVkRUhT tidysynth: https://lnkd.in/gSScV-ya 📚 Further Reading Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program. Journal of the American Statistical Association, 105(490), 493–505. https://lnkd.in/g_jwdfri Arkhangelsky, D., et al. (2021). Synthetic Difference-in-Differences. American Economic Journal: Applied Economics, 13(2), 1–35. https://lnkd.in/gnWhCX4y Ben-Michael, E., Feller, A., & Rothstein, J. (2021). The Augmented Synthetic Control Method. Journal of the American Statistical Association, 116(536), 1789–1803. https://lnkd.in/gfSNPYih Xu, Y. (2017). Generalized Synthetic Control Method: Causal Inference with Interactive Fixed Effects Models. Political Analysis, 25(1), 57–76. Link Hazlett, C., & Xu, Y. (2018). Trajectory Balancing: A General Reweighting Approach to Estimating Treatment Effects in Synthetic Control Designs.  https://lnkd.in/g9d4_895 #CausalInference #SyntheticControl #PolicyEvaluation #RStats

🌍 New paper out in Nature Climate Change on a critical question for climate policy: How does policy sequencing impact energy decarbonization? Led by Huilin Luo and Wei Peng, with Allen Fawcett, Jessica Green, Gokul Iyer, Jonas Nahm and David G. Victor Our team used advanced energy modeling to examine "carrots" (subsidies like those in the Inflation Reduction Act) vs. "sticks" (carbon pricing) - and crucially, the ORDER in which they're deployed. Key findings: ✅ Carrots alone don't achieve deep decarbonization – sticks are needed ✅ Near-term impacts of carrots vary widely by sector and consistency ✅ Timing is critical: delaying carbon pricing by 20 years (vs. 10) increases the eventual price needed by 40% ✅ Carrots boost green industries but don't significantly phase out fossil fuels - sticks are essential for that ✅ With rapid innovation, carrots followed quickly by sticks can be nearly as cost-effective as leading with carbon pricing The research bridges political science and energy modeling to analyze real-world policy tradeoffs. While carbon taxes are economically "first-best," political reality often requires starting with industrial policy - making the transition strategy crucial. Check out Mark Purdon’s great commentary on the paper: Green Industrial Policy Is Not Enough for Deep Decarbonization https://lnkd.in/gURqCbUE Read the full paper: https://lnkd.in/gW52_bcC #ClimatePolicy #EnergyTransition #ClimateScience #InflationReductionAct #Decarbonization

𝗙𝗼𝗿 𝟮𝟬 𝘆𝗲𝗮𝗿𝘀, 𝘄𝗲 𝗽𝗿𝗲𝗽𝗮𝗿𝗲𝗱 𝗞𝗣𝗜𝘀. Now, this is only half of the job done. Erasmus+ Youth applications 𝗶𝗻 𝟮𝟬𝟮𝟲 𝘄𝗶𝗹𝗹 𝗹𝗼𝗼𝗸 𝘃𝗲𝗿𝘆 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁. The policy roadmap is already here. If you want to predict the future of your project, 𝘆𝗼𝘂 𝗻𝗲𝗲𝗱 𝘁𝗼 𝗿𝗲𝗮𝗱 𝘁𝗵𝗲 𝗳𝗶𝗻𝗲 𝗽𝗿𝗶𝗻𝘁 of today's policy frameworks. It tells. 𝗙𝗼𝗿𝗴𝗲𝘁 𝘀𝘁𝗮𝗻𝗱𝗮𝗿𝗱 𝗞𝗣𝗜𝘀! The shift from "𝗰𝗼𝘂𝗻𝘁𝗶𝗻𝗴 𝗵𝗲𝗮𝗱𝘀" to "p𝗿𝗼𝘃𝗶𝗻𝗴 𝘀𝘆𝘀𝘁𝗲𝗺𝗶𝗰 𝗰𝗵𝗮𝗻𝗴𝗲" is explicitly documented in three key resources: 𝟭 𝗘𝗨 𝗬𝗼𝘂𝘁𝗵 𝗦𝘁𝗿𝗮𝘁𝗲𝗴𝘆 (𝟮𝟬𝟭𝟵–𝟮𝟬𝟮𝟳): Explicitly demands evidence-based policy making and "participatory evaluation" methods. 𝟮 𝗥𝗔𝗬 𝗡𝗲𝘁𝘄𝗼𝗿𝗸 𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴 (𝗥𝗔𝗬-𝗠𝗢𝗡/𝗟𝗧𝗘): The EU’s data backbone now prioritises verified impact over simple satisfaction scores. 𝟯 𝗖𝗼𝘂𝗻𝗰𝗶𝗹 𝗥𝗲𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻 𝗼𝗻 𝗬𝗼𝘂𝘁𝗵 (𝟮𝟬𝟮𝟮–𝟮𝟬𝟮𝟳): Calls for systemic activity evaluation that links local project results to European policy goals. 𝗪𝗵𝗮𝘁 𝗰𝗵𝗮𝗻𝗴𝗲𝘀 𝗮𝗿𝗲 𝘁𝗵𝗲𝘀𝗲 𝗯𝗿𝗶𝗻𝗴𝗶𝗻𝗴? We are seeing a move away from purely quantitative KPIs. The "tick-box" era of evaluation is ending. The Commission is signalling a 𝗻𝗲𝗲𝗱 𝗳𝗼𝗿 𝗻𝗮𝗿𝗿𝗮𝘁𝗶𝘃𝗲-𝗯𝗮𝘀𝗲𝗱 𝗶𝗺𝗽𝗮𝗰𝘁 𝗿𝗲𝗽𝗼𝗿𝘁𝗶𝗻𝗴—they want to know the "story" of the change, not just the number of participants. Reread it! We will need to 𝘁𝗲𝗹𝗹 𝘁𝗵𝗲 𝘀𝘁𝗼𝗿𝘆 𝗼𝗳 𝘄𝗵𝗮𝘁 𝗵𝗮𝗽𝗽𝗲𝗻𝗲𝗱, not the number of people! 𝗪𝗵𝗮𝘁 𝗱𝗼𝗲𝘀 𝘁𝗵𝗶𝘀 𝗺𝗲𝗮𝗻 𝗶𝗻 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗲? To align with this shift for 2026, we need to upgrade our evaluation toolkits now. A hybrid model is emerging as the new gold standard: • 𝗠𝗼𝘀𝘁 𝗦𝗶𝗴𝗻𝗶𝗳𝗶𝗰𝗮𝗻𝘁 𝗖𝗵𝗮𝗻𝗴𝗲 (𝗠𝗦𝗖) 𝗠𝗼𝗱𝗲𝗹: To capture the qualitative, human stories of empowerment. • 𝗢𝘂𝘁𝗰𝗼𝗺𝗲 𝗛𝗮𝗿𝘃𝗲𝘀𝘁𝗶𝗻𝗴: To map those specific stories directly to the EU Youth Goals. • If you can prove the l͟i͟n͟k͟ ͟b͟e͟t͟w͟e͟e͟n͟ ͟a͟ ͟p͟a͟r͟t͟i͟c͟i͟p͟a͟n͟t͟'͟s͟ ͟p͟e͟r͟s͟o͟n͟a͟l͟ ͟s͟t͟o͟r͟y͟ ͟a͟n͟d͟ ͟a͟ ͟m͟a͟j͟o͟r͟ ͟E͟U͟ ͟p͟o͟l͟i͟c͟y͟ ͟o͟b͟j͟e͟c͟t͟i͟v͟e͟,͟ your proposal becomes incredibly difficult to reject. Are you preparing your evaluation frameworks for this shift?

Integrated #Evaluation Framework Theory of Change → Program Design → Impact Evaluation Pipeline → Causal Inference Analysis → Evidence-Based Policy #ToC [evaluators test whether each step actually occurred.] Problem / Context → Inputs (Funding, Staff, Technology) → Activities (Training, Awareness Campaigns, Infrastructure, Policy Actions) → Outputs (Services delivered, People trained, Resources distributed) → Outcomes (Behavior change, Improved skills, Access to services) → Impact (Long-term social change: poverty reduction, better health, economic growth) #Impact Evaluation Pipeline [Did the program work?  How much impact did it create?] Program Intervention → Baseline Data Collection (surveys, administrative data) → Treatment & Control Groups → Implementation of Program → Follow-up Data Collection (endline survey) → Impact/ Contribution Estimation (RCT, DiD, PSM, Regression, IV) → Results & Interpretation → Policy Learning & Program Improvement #Causal Inference Framework [Causal inference tries to isolate the true effect of an intervention while controlling for other influences.] Observed Outcome (Y) → Treatment Variable (Program / Policy) → Control Variables (age, income, education, environmental factors) → Counterfactual Estimation "What would have happened without the intervention?" → Causal Estimation Methods [• RCT • Difference-in-Differences • Propensity Score Matching • Regression • Instrumental Variables] → Estimated Treatment Effect (Program Impact) #ToC+ #Impact+ #Causal Example [Public health vaccination program:] #Vaccination campaign → Increased immunization coverage → Reduced disease incidence → Improved population health ·       Impact evaluation measures whether disease reduction was caused by the program. ·       Causal inference methods estimate the magnitude of the effect. #Food security program: farmers Training/ credit/ policy support → better farming practices → higher crop yield → Increased Production → improved Household income → reduced poverty and better food security

🎆 Excited to share that our paper, “Modeling the impacts of policy sequencing on energy decarbonization,” is out today in Nature Climate Change 🎉 Many countries have embraced climate policy strategies that emphasize large subsidies to deploy green technologies (‘carrots’) with the anticipation that more punitive policies (‘sticks’) may follow. But what does this sequencing mean for long-term decarbonization? Using the US as a case study, we explore different policy pathways—carrots only, sticks only, and various carrot-then-stick approaches—to understand how policy sequencing influences energy decarbonization. Main takeaways: - Carrots help with near-term mitigation, but sticks remain essential for long-term deep decarbonization - A carrots-first strategy works best when it speeds up innovation and is quickly followed by credible sticks - Policy durability matters: inconsistent carrots make decarbonization more costly and slower This work is part of our broader effort to bring political economy considerations into energy system modeling. I’m deeply grateful to our political science colleagues who have shaped my thinking—David G. Victor, Jonas Meckling Jonas Nahm Jessica Green—and to my fellow modelers—Gokul Iyer Allen Fawcett—for embracing the challenge of bringing politics into energy models. Special shoutout to Huilin Luo for publishing her first lead-author paper. Thanks also to Alfred P. Sloan Foundation for supporting this effort! - View-only full text: https://rdcu.be/eVPwH - Paper link from the journal: https://lnkd.in/eZ3x7Xvb

Modeling the world isn’t the same as making decisions in it! A common trap in applied AI, OR, and digital transformation is to spend months building a perfect simulator… a beautiful digital twin with clean architecture, smooth animations, and every physical nuance modeled. And yet… when it comes time to actually make decisions? No policy. No framework. Just dashboards and “what-if” buttons. Here’s the core mistake: We confuse modeling the system with modeling the decisions. A simulator helps you observe behavior. A policy helps you choose actions. They serve different purposes. At Toyota North America, I always separate the two: 🔹 Modeling the system (the physics, flows, stochastic processes) gives you a sandbox to play in. 🔹 Designing the policy means deciding how you’ll act over time, based on what you observe in the system. Want to optimize shuttle routing at a port? Great. Simulate vehicle movements, fueling stations, labor shifts, arrival patterns. But then design a policy that says when the shuttle leaves, who it picks up, and how it adjusts when demand surges. 🚫 A simulator is not a decision model. ✅ A simulator is the environment. Your policy is the intelligence. Throughout my career I’ve had entire projects spin in circles because nobody took the time to define: • What’s the decision? • When is it made? • Based on what information? • Using what logic? These four questions can help drive policy design and be the difference between pretty analytics and real ROI. Build your digital twin if you need to. But don’t forget to teach it how to act!