Experimental Design In Science

A Machine Learning Framework to Defy Hidden Confounding? The overwhelming majority of data collected in the world is observational. In most cases, such data cannot be directly used to inform decision-making. This is due to the high risk of hidden confounding: the presence of unobserved variables that impact both the treatment of interest and the outcome, making the estimates of potential effects of one variable on another untrustworthy at best and catastrophically misleading at worst. In their brand-new paper, Konstantin Hess and colleagues propose a novel method for finding optimal policies from observational data under hidden confounding. The method extends the marginal sensitivity model and provides us with a well-behaved estimator for personalized policy learning under hidden confounding. The estimator returns bounds (rather than a point estimate) that are sharp (cannot be narrower), is unbiased, and achieves the lowest possible variance (efficiency). The authors demonstrate through a series of experiments on synthetic and real-world data that their method significantly outperforms existing approaches. Although the results are dependent on the specification of assumed confounding strength, the robustness checks provided by the authors are reassuring—the method appears to be more than reasonably robust to the confounding strength misspecification. Although bounds are not guaranteed to be always useful, they can be immensely helpful in decision-making under uncertainty. A very important contribution and a must-read for anyone interested in robust decision-making under hidden confounding. Python code available on GitHub (all links in the comments)

This reflection paper by the European Medicines Agency provides guidance on the methodological aspects of using real-world data (#RWD) in non-interventional studies (#NIS) to generate real-world evidence (#RWE) for #regulatory purposes. It emphasizes the importance of carefully selecting study designs based on the research question. Studies are classified as either descriptive, focusing on patient characteristics and patterns, or causal, aiming to infer treatment effects. The framework of target #trial_emulation is recommended to improve the internal validity of causal studies by mimicking randomized controlled trials as closely as possible using observational data. A key concern throughout is addressing sources of #bias and #confounding. The paper highlights different types of bias—selection bias, information bias, and time-related bias—and stresses the need to prevent or mitigate these issues during study design rather than trying to correct them post hoc. Special attention is given to confounding, recommending careful identification and handling of confounders through study design choices like active comparators and new user designs, and the use of control exposures or outcomes where appropriate. Effect modification is also discussed in the context of ensuring the generalisability of findings. #Governance and #transparency are critical elements, with the paper advocating adherence to the ENCePP Code of Conduct and EU data protection regulations. It calls for clear study registration, public disclosure of protocols and results, and sharing of analytical codes to enhance reproducibility and trust. #Data_quality is another major focus, with emphasis on evaluating both the reliability (accuracy, completeness, credibility) and relevance (fitness for the research question) of RWD sources. The use of data quality frameworks, transparent validation of data elements, and careful handling of data linkage across sources are recommended to ensure robust evidence generation. Finally, the paper outlines expectations for statistical analyses, stressing pre-specified analysis plans, model transparency, and robust handling of missing data, confounding, and heterogeneity. It advises moving beyond p-values to focus on effect estimates and their clinical relevance, using sensitivity and stratified analyses to assess the robustness of findings. By integrating these principles, the paper aims to improve the quality and regulatory acceptability of RWE derived from NIS. Disclaimer: The opinions shared are solely my own and not express the views or opinions of any of my employers.

Standard regression breaks the moment your confounder is affected by prior treatment. And in longitudinal data — HIV therapy, dialysis, statins, ventilator management — that is the default. You measure CD4 at every visit. You adjust for it in a time-varying Cox model. You think you controlled for confounding. For the total causal effect, you likely did not. You may have blocked part of the pathway you are trying to estimate. --- The problem A time-varying covariate can be: → Confounder of future treatment → Mediator of past treatment → Predictor of the outcome Condition on it → bias Ignore it → confounding Standard regression has no clean solution. --- The fix: Marginal Structural Models Do not condition. Weight. 1️⃣ Model treatment over time 2️⃣ Build stabilized IPTW weights 3️⃣ Create a pseudo-population where treatment ⟂ history 4️⃣ Run weighted regression → causal effect --- Assumptions (non-negotiable) ✓ Consistency ✓ Sequential exchangeability ✓ Positivity ✓ No interference If these fail, no method will save you. --- Estimation layer ⚠️ IPTW needs a well-specified treatment model ✓ TMLE is doubly robust --- Bottom line ❌ Time-varying Cox / GEE → biased total effect ✅ MSM + IPTW → recovers causal effect --- This is Post in my Causal Inference Visual Guide series. Previously: KM → PSM → IPW → IV → DiD → RDD → DAGs → TTE → Synthetic Control → MSM ♻️ Repost if this clarified longitudinal confounding 💾 Save for your next study design #CausalInference #MarginalStructuralModels #Epidemiology #Biostatistics #PublicHealth

Could AI drafts—even imperfect ones—be a time-saver for radiologists when interpreting CT scans? Our pilot study using simulated AI reports found a 24% faster workflow, with accuracy intact. Q: What makes this study's approach unique? A: Instead of building an AI system, we used GPT-4 to simulate what AI-generated draft reports might look like. We deliberately introduced 1-3 errors in half the drafts to study how radiologists would handle imperfect AI assistance - a "Wizard of Oz" approach to prototype the future workflow. Q: How was the simulation study structured? A: We conducted a 3-reader crossover study with 20 chest CT cases. Each case was read twice: once with standard templates, and once with our simulated AI drafts. This controlled design let us directly compare the workflows. Q: What efficiency gains did you see with the simulated drafts? A: Median reporting time dropped from 573 to 435 seconds (p=0.003) - a 24% reduction. Two readers showed major improvements (717→398s and 361→322s), while one showed an increase (947→1015s). Q: Did the intentionally flawed drafts impact accuracy? A: Surprisingly, even with deliberately introduced errors in half the simulated drafts, the AI-assisted workflow showed slightly fewer clinically significant errors (0.27±0.52) compared to standard workflow (0.38±0.78). While not statistically significant, this suggests radiologists maintained their vigilance even with imperfect drafts. Q: How did radiologists respond to working with these simulated drafts? A: All 3 readers found the prototype system easy to use and well-integrated into their workflow. Two reported somewhat less mental effort, while one reported significantly reduced effort. Their likelihood to recommend it varied (scores of 5, 9, and 10 out of 10). Q: What's next? A: While these simulation results are encouraging, these are small scale pilot studies setting the stage for deeper validation. Link to short paper: https://lnkd.in/d-4aTJ69 Congratulations to stellar team of Julián Nicolás Acosta, Siddhant Dogra, Subathra Adithan, Kay Wu, MD 💫, Michael Moritz, Stephen Kwak

✍ 𝗗𝗲𝘀𝗶𝗴𝗻 𝗼𝗳 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁 (𝗗𝗢𝗘) ✍ is a critical component within the Product Development Report (PDR) for an Abbreviated New Drug Application (ANDA). DOE is used to systematically plan, conduct, and analyze experiments to optimize the formulation and manufacturing process of the generic drug. 𝗣𝘂𝗿𝗽𝗼𝘀𝗲 𝗼𝗳 𝗗𝗢𝗘 𝗶𝗻 𝗣𝗗𝗥 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻: DOE helps in identifying the optimal conditions for the formulation and manufacturing process by evaluating the effects of multiple variables simultaneously. 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆: It reduces the number of experiments needed by using statistical methods to design the experiments, thus saving time and resources. 𝗤𝘂𝗮𝗹𝗶𝘁𝘆: Ensures that the product meets the required quality standards by understanding the relationship between input variables (e.g., excipients, process parameters) and output responses (e.g., drug stability, dissolution rate). 𝗞𝗲𝘆 𝗘𝗹𝗲𝗺𝗲𝗻𝘁𝘀 𝗼𝗳 𝗗𝗢𝗘 𝗙𝗮𝗰𝘁𝗼𝗿𝘀: These are the variables that are changed during the experiment. In drug development, factors could include temperature, pH, mixing speed, etc. 𝗟𝗲𝘃𝗲𝗹𝘀: The different values or settings for each factor. For example, temperature might be tested at 25°C, 30°C, and 35°C. 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗲𝘀: The outcomes measured to determine the effect of the factors. This could include drug potency, dissolution rate, and stability. 𝗜𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻𝘀: DOE helps in understanding how different factors interact with each other and their combined effect on the responses. 𝗦𝘁𝗲𝗽𝘀 𝗶𝗻 𝗖𝗼𝗻𝗱𝘂𝗰𝘁𝗶𝗻𝗴 𝗗𝗢𝗘 𝗗𝗲𝗳𝗶𝗻𝗲 𝗢𝗯𝗷𝗲𝗰𝘁𝗶𝘃𝗲𝘀: Clearly state what you aim to achieve with the experiment. 𝗦𝗲𝗹𝗲𝗰𝘁 𝗙𝗮𝗰𝘁𝗼𝗿𝘀 𝗮𝗻𝗱 𝗟𝗲𝘃𝗲𝗹𝘀: Choose the factors to be studied and the levels at which they will be tested. 𝗗𝗲𝘀𝗶𝗴𝗻 𝘁𝗵𝗲 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁: Use statistical software to create an experimental design, such as a full factorial or fractional factorial design. 𝗖𝗼𝗻𝗱𝘂𝗰𝘁 𝘁𝗵𝗲 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁: Perform the experiments as per the design. 𝗔𝗻𝗮𝗹𝘆𝘇𝗲 𝗗𝗮𝘁𝗮: Use statistical methods to analyze the data and determine the significance of the factors and their interactions. 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲: Identify the optimal conditions based on the analysis. 𝗕𝗲𝗻𝗲𝗳𝗶𝘁𝘀 𝗼𝗳 𝗗𝗢𝗘 𝗶𝗻 𝗣𝗗𝗥 𝗜𝗺𝗽𝗿𝗼𝘃𝗲𝗱 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴: Provides a deeper understanding of the process and formulation. 𝗥𝗼𝗯𝘂𝘀𝘁 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝘀: Helps in developing robust products that are less sensitive to variations in manufacturing conditions. 𝗥𝗲𝗴𝘂𝗹𝗮𝘁𝗼𝗿𝘆 𝗖𝗼𝗺𝗽𝗹𝗶𝗮𝗻𝗰𝗲: Demonstrates a scientific approach to product development, which is crucial for regulatory approval. 𝗗𝗢𝗘 𝗶𝘀 𝗮 𝗽𝗼𝘄𝗲𝗿𝗳𝘂𝗹 𝘁𝗼𝗼𝗹 𝗶𝗻 𝘁𝗵𝗲 𝗣𝗗𝗥 𝘁𝗵𝗮𝘁 𝗲𝗻𝘀𝘂𝗿𝗲𝘀 𝘁𝗵𝗲 𝗴𝗲𝗻𝗲𝗿𝗶𝗰 𝗱𝗿𝘂𝗴 𝗶𝘀 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗱 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁𝗹𝘆 𝗮𝗻𝗱 𝗺𝗲𝗲𝘁𝘀 𝗮𝗹𝗹 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝘀𝘁𝗮𝗻𝗱𝗮𝗿𝗱𝘀 #Pharmaceuticals #DrugDevelopment #DOE #Quality #Innovation #GenericDrugs

𝐓𝐡𝐞 𝐈𝐧𝐭𝐮𝐢𝐭𝐢𝐨𝐧 𝐁𝐞𝐡𝐢𝐧𝐝 𝐃𝐨𝐮𝐛𝐥𝐞 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 In real-world Causal Inference problems, we often face complex and partially hidden confounders (variables that affect both treatment and outcome). While some of them can be measured, others remain unknown. 𝐃𝐨𝐮𝐛𝐥𝐞 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 (𝐃𝐌𝐋) helps address the first challenge by flexibly adjusting for observed confounders using modern ML models. 🧩 𝐓𝐡𝐞 𝐈𝐧𝐭𝐮𝐢𝐭𝐢𝐯𝐞 𝐈𝐝𝐞𝐚: "𝐒𝐞𝐩𝐚𝐫𝐚𝐭𝐞 𝐚𝐧𝐝 𝐂𝐥𝐞𝐚𝐧" Imagine you want to measure the effect of a new education program on student performance. But you also know that many other factors such as income, prior grades, attendance etc influence both participation and performance. 𝐃𝐌𝐋 𝐡𝐚𝐧𝐝𝐥𝐞𝐬 𝐭𝐡𝐢𝐬 𝐢𝐧 𝐭𝐡𝐫𝐞𝐞 𝐢𝐧𝐭𝐮𝐢𝐭𝐢𝐯𝐞 𝐬𝐭𝐞𝐩𝐬: 1️⃣ 𝐏𝐫𝐞𝐝𝐢𝐜𝐭 𝐭𝐰𝐢𝐜𝐞: ▶️ first, use ML to predict performance (Y) based on student characteristics (X). ▶️ then, use another ML model to predict the probability of participating in the program (T) based on the same X. 2️⃣ 𝐂𝐥𝐞𝐚𝐧 (𝐨𝐫𝐭𝐡𝐨𝐠𝐨𝐧𝐚𝐥𝐢𝐳𝐞): subtract these predictions from the actual values to get 𝐫𝐞𝐬𝐢𝐝𝐮𝐚𝐥𝐬 or the parts of Y and T unexplained by X. These residuals are now cleaned from confounding effects. 3️⃣ 𝐄𝐬𝐭𝐢𝐦𝐚𝐭𝐞 𝐭𝐡𝐞 𝐜𝐚𝐮𝐬𝐚𝐥 𝐞𝐟𝐟𝐞𝐜𝐭: run a regression of the cleaned outcome on the cleaned treatment. The coefficient now reflects the true causal impact, isolated from noise and bias. 🔐 𝐖𝐡𝐲 𝐈𝐭'𝐬 𝐑𝐨𝐛𝐮𝐬𝐭: 𝐍𝐞𝐲𝐦𝐚𝐧 𝐎𝐫𝐭𝐡𝐨𝐠𝐨𝐧𝐚𝐥𝐢𝐭𝐲 Even if one of the ML models (say, the one predicting Y) isn't perfectly specified, DML stays reliable. This is thanks to a mathematical property called 𝐍𝐞𝐲𝐦𝐚𝐧 𝐨𝐫𝐭𝐡𝐨𝐠𝐨𝐧𝐚𝐥𝐢𝐭𝐲, which ensures that 𝐬𝐦𝐚𝐥𝐥 mistakes in the first-stage models don't distort the final causal estimate. In other words, DML is robust to imperfection. You don't need perfect ML predictions to get valid causal insights. 🔁 𝐂𝐫𝐨𝐬𝐬-𝐅𝐢𝐭𝐭𝐢𝐧𝐠: 𝐀𝐯𝐨𝐢𝐝𝐢𝐧𝐠 𝐎𝐯𝐞𝐫𝐟𝐢𝐭𝐭𝐢𝐧𝐠 𝐁𝐢𝐚𝐬 DML also uses cross-fitting, meaning the data is split into folds so that the models predicting Y and T never see the same data points used for the final causal regression. This prevents overfitting and ensures the estimate generalizes well. ⚠️ 𝐍𝐨𝐭 𝐚 𝐒𝐢𝐥𝐯𝐞𝐫 𝐁𝐮𝐥𝐥𝐞𝐭 Despite its strengths, DML is not a magic solution. It relies on a key assumption: that all relevant confounding variables are observed, which must be justified using strong domain knowledge. It can also be computationally expensive and data-hungry, especially when using complex ML models with cross-fitting. 👉 Still, DML remains a powerful and flexible framework that helps researchers and data scientists move from detecting correlations to uncovering true causal relationships. #CausalInference #MachineLearning #DoubleMachineLearning #PredictiveAnalysis #DataScience

As academics, we all want our research to be trusted, reproducible, and strong enough to withstand review. Yet most of the problems we face during publication come from one place: weak statistical foundations and unclear experimental design. This is why I want to give you a quick, practical guide you can use to strengthen any study you are planning or refining. These principles are simple, but they prevent the most common errors I see across manuscripts, reviews, and collaborations. 1. Statistics is not about numbers. It is about reasoning. Each test, each calculation, tells a story about your data and what it truly means. 2. Experimental design begins with purpose. Define your objective clearly before you begin collecting data. The design should flow naturally from the research question. 3. Randomization protects integrity. Assign treatments randomly to eliminate bias and ensure valid comparisons. 4. Replication increases confidence. Repeating experiments strengthens conclusions and helps distinguish real effects from noise. 5. Control groups matter. They provide the baseline that gives your results meaning. Without controls, interpretation becomes speculation. 6. Choose tests based on data, not habit. Understand whether your variables are categorical, continuous, or ordinal. Then select the statistical method that fits the data, not the one that feels familiar. 7. Interpret, do not just report. Numbers are not the end of the story. Explain what they mean, why they matter, and how they support or challenge your hypothesis. 8. Visuals clarify understanding. Use tables and graphs to reveal patterns and relationships, but keep them clean, accurate, and purposeful. 9. Ethical analysis is non-negotiable. Never manipulate data to fit a narrative. Transparency and honesty sustain the credibility of your research. 10. Statistics and design are partners. Good design minimizes errors. Good statistics reveal the truth within them. One without the other cannot stand. These principles are not theoretical. They are the difference between a study that moves quickly through review and a study that struggles with rejection, uncertainty, or inconsistent conclusions. Download the full PDF below. Do you think your current research would benefit from this guide? Reply and tell me. I would love to know. ______________________________ 📌 This is Prof. Samira Hosseini. I’ve helped 12,000+ ambitious academics go from struggling with publishing papers in Q1 journals, limited visibility, and poor citation records to building a solid research trajectory and high 𝘩-index. Book a free Strategy Call, and we can dive into your challenges in top-tier journal publication and citation and see how I can best assist you: https://lnkd.in/ezqV64dX

Delve into our collaborative publication between Pfizer and Massachusetts Institute of Technology (Klavs Jensen). 📚 Dynamic Flow Experiments for Bayesian Optimization of a Single Process Objective, Reaction Chemistry & Engineering, 2025. 🔬 Key insights: * 🔄 A new method, named dynamic experiment optimization (DynO), is developed for the chemical reaction optimization by leveraging for the first time both #Bayesian optimization and data-rich dynamic experimentation in flow chemistry. * ⚙️ The algorithm is able to guide the user from initialization (using steady or dynamic experiments) to the end of the optimization procedure thanks to useful convergence criteria, which were proposed for the first time together with an estimate of the regret reached. * 🚀 DynO is readily implementable in #automated systems and it is augmented with simple stopping criteria to guide non-expert users in fast and reagent-efficient optimization campaigns. * 📊 The developed algorithms is compared in silico with the algorithm Dragonfly (and a random optimizer), showing remarkable performance in terms of experiment time saving and reagent volume reduction. * 🧬 DynO is validated with an ester hydrolysis reaction at Pfizer on an automated platform, showing that DynO can be easily implemented experimentally and allows to evaluate optimal reaction conditions with a limited number of experiments. Thanks to all collaborators for this insightful research! Federico Florit, Dr. Kakasaheb Nandiwale, Cameron T. Armstrong, Katharina Grohowalski, Angel Diaz, Jason Mustakis, Steven Guinness, and Klavs Jensen. Congratulations to the authors!🎉 #Bayesian #Optimization #Datarich #Dynamic #collaboration #PfizerProud 📚 Article link: https://lnkd.in/efic5nSc

🤓 Understanding and Estimating Propensity Scores in Causal Inference 🤓 Assessing the effectiveness of an intervention often involves handling confounding variables—factors influencing both treatment assignment and outcomes. Propensity scores simplify confounder control, offering an efficient solution. Let’s explore what propensity scores are, how to calculate them, and their practical use. ℹ️ What is a Propensity Score? A propensity score, e(x) = P(T|X), represents the likelihood of receiving treatment given observable variables X. Instead of controlling for all confounders, which can be complex, propensity scores act as a balance score, ensuring comparability between treatment and control groups. ℹ️ Why Use Propensity Scores? 1️⃣ Dimensionality Reduction: Condition on a single score rather than multiple variables. 2️⃣ Blocking Backdoor Paths: Adjust for the score to prevent bias from confounders. 3️⃣ Comparability: Subjects with similar scores differ only in treatment, mimicking random assignment. ➡️ Estimating Propensity Scores In real-world scenarios, the true propensity score is unknown and must be estimated. Logistic regression is commonly used for binary treatments. 🪩 Example: E-commerce Discount Campaign An e-commerce platform runs a discount campaign to boost sales. Customers with higher purchase history or engagement might be more likely to receive the discount, creating bias. To evaluate the campaign’s effectiveness, we can use propensity scores to ensure treated and control groups are comparable. 1️⃣ Estimate Propensity Scores: Use logistic regression or ML to predict the probability of receiving the discount based on covariates like purchase history, average order value, customer tenure, and browsing frequency. 2️⃣ Match or Weight Customers: Create balanced groups by matching or weighting based on propensity scores. 3️⃣ Compare Outcomes: Adjust for propensity scores to isolate the campaign’s effect on sales. For binary treatments, logistic regression can be implemented using statsmodels: ``` m = smf.logit("discount_received ~ purchase_history + avg_order_value + customer_tenure + browsing_frequency", data=df).fit(disp=0) ps = df.assign(propensity_score=m.predict(df)) data_ps[["discount_received", "sales", "propensity_score"]].head() ``` Example Output | Discount Received | Sales | Propensity Score | |-------------------|-------|------------------| | 0         | 150  | 0.40       | | 1         | 200  | 0.55       | | 0         | 130  | 0.35       | | 1         | 220  | 0.60       | 🤔 Conclusion Propensity scores are vital for causal inference, enabling researchers to control for confounders efficiently. Whether using logistic regression or ML, ensuring accurate score estimation is key to deriving reliable conclusions about interventions. This approach enhances decision-making and understanding of treatment effects. __ For a deeper dive, consider Causal Inference in Python by Matheus Facure: https://shorturl.at/SMHcv.

We frequently mistake metrics for truth and build whole interventions on quantitative sand while ignoring the qualitative bedrock of human behaviour. You can meticulously measure a statistic, yet fail to recognise the cultural nuance or historical mistrust that underpins that figure. True systems thinking requires an architecture where we stop viewing storytelling as soft science and start analysing lived experience as structural evidence, equal to any clinical trial. Without this design rigour, our most expensive strategies are little more than well-funded guesses that fail to address the root causes of the problem. As you finalise your annual reports this season, resist the temptation to rely solely on the comforts of executive summaries and pristine charts. This is the precise moment design thinking becomes critical because a spreadsheet is incapable of capturing the complex reasoning behind why a community accepts or rejects a solution. You must treat narratives as primary data points and return to the field to listen. Before you publish your findings, interview the beneficiaries of your program; if their raw quotes contradict your polished graphs, have the professional courage to trust them.