Feedback Sensitivity Analysis

The concept of sensitivity analysis can often be shrouded in mystery. For many new to research, it's imagined as one specific type of analysis. However, sensitivity analysis isn't one singular test—it's about assessing how robust our findings are.💡 When we say that estimates are "robust," we mean that the results remain stable even when assumptions are changed. If results change drastically with even small changes in assumptions, it means they’re not robust. Here are key tips: 1️⃣ Measures of Central Tendency: If you use the mean in your main analysis, consider using the median in sensitivity analysis 📊 2️⃣ Contextual Definitions: For constructs without a universal definition, you might use a widely accepted definition for your primary analysis and test it with contextual modifications as part of sensitivity analysis 🧠 3️⃣ Exposure Variables: When exposure thresholds differ, try using various thresholds to define exposure. The main analysis could use the most commonly applied threshold, while sensitivity analysis explores others ⚖️ 4️⃣ Coherence with Outcomes: Looking at different outcomes measuring related aspects can strengthen your conclusions 📈 5️⃣ Outcome Specificity: If your primary outcome is less specific, explore secondary outcomes that may be more specific. For instance, looking at deaths due to smoking (more specific) vs all-cause mortality (less specific) 💀 6️⃣ Assessment Methods: If you have multiple methods for assessing the same outcome (e.g., self-reported vs. biomarker data), you can use the more accurate method as the main analysis and the less accurate one for sensitivity testing. 7️⃣ Handling Missing Values: How does your result change when you adjust for missing data? Test using different approaches like multiple imputation, listwise deletion, or inverse proportional weighting 📉 8️⃣ Model Assumptions: Test how your results hold when adjusting key model assumptions (e.g., linearity, independence) 🔧 9️⃣ Outlier Handling: Consider how sensitive your results are to extreme values. Does removing outliers or using robust methods change the outcome? 🚨 🔟 Timeframe Adjustments: For time-dependent data, check how your results change with different observation periods ⏳ 1️⃣1️⃣ Data Transformation: Examine how sensitive your findings are to data transformations (e.g., log-transformation vs Box-Cox transformation) 🔄 1️⃣2️⃣ Aggregation Level: Assess how results change when aggregating or disaggregating data (e.g., regional or demographic groupings) 🌍 1️⃣3️⃣ Uncertainty in Input Parameters: Monte Carlo simulations are a great way to test the range of possible outcomes with varying input assumptions 🎲 Bottomline: Sensitivity analysis isn’t a one-size-fits-all process—it’s context-driven. It’s not about fixing a "bad" analysis; rather, it’s about assessing how well a well-conducted analysis holds up under different assumptions and conditions💪 Please reshare ♻️ #Chisquares #VillageSchool #SensitivityAnalysis

"You cannot carry out causal analysis without controlling for *all* hidden confounders" For the brave, here's a sensitivity analysis case at Booking[.]com The idea of unverifiable causal assumptions may lead data practitioners to believe that causal inference from non-randomized data is not possible in practice. Well, at least not in any meaningful sense. But is that true? In their new paper Phillip Bach (FU Berlin), Victor Chernozhukov (MIT), Martin Spindler (Uni of Hamburg) and colleagues present a real-world use case for causal analysis where the no-hidden-confounding assumption is unlikely to hold. Here's a little TLDR based on the last week's issue of Causal Python Weekly: Why it matters? The authors demonstrate that a trustworthy causal analysis is not only possible, but also directly applicable in industrial settings. They highlight the relevance of sensitivity analysis -- an essential, but often underrepresented tool in any causal practitioner's toolbox. What to expect? The authors use general non-parametric bounds on biases caused by potential omitted variables using the double machine learning framework. Interested in trying it for yourself? A Python notebook walking through the case is available here: https://vist.ly/4i9id Paper: https://vist.ly/4i9ie Causal Python Weekly: https://vist.ly/4i9if (it's FREE)

The recent paper by Cinelli & Hazlett on Sensitivity Analysis is not only fascinating but might improve causal inference / econometrics claims significantly in an accessible way. First, let me do a quick refresher. When you do causal inference, you have to capture the effect of all the confounders in avoid bias (Omitted Variable Bias). However, you are never sure that you captured all the confounders (maybe you forgot some and if you present your work in a seminar in economics, anyway, some people in the audience will come up with very creative ideas about potential remaining confounders). So here comes Sensitivity analysis to the rescue. It is about quantifying the risk of remaining confounders. Basically it allows to make some assumption and quantify how likely/unlikely it is that a remaining confounder would change your conclusions. Cinelli et Hazlett introduced a very convenient way to do this which requires little information (standard info reported in most regression tables). This is incredibly useful as they suggest to 1. make this a standard in reporting 2. compute this quickly from "any" regression table. Link to the paper in comment

Designing a clinical trial or writing a SAP and not sure if you should include a sensitivity analysis? Here is why we do them and how the FDA views and considers them: The primary goal of a sensitivity analysis is to demonstrate the robustness of a clinical result. We want to show the primary results are consistent across a range of assumptions and methodologies. It's the same analysis that we originally performed, but with one or two items changed and we then look at how different the results are. In the eyes of the FDA (specified in their guidance documents 😎) sensitivity analyses are a way researchers can give the FDA confidence in results that are heavily dependent on assumptions or heavily adjusted models. The FDA DOES consider sensitivity analyses during the NDA and BLA submission process and I personally have been part of submissions where the FDA has placed HEAVY consideration on sensitivity analyses (one submission even had the FDA requesting additional sensitivity analyses to evaluate the effect missing data data had on the primary endpoints for the two pivotal trials). It does not matter if you are a biostatistician writing your first statistical analysis plan or if you are a CMO looking to submit an NDA - understanding what sensitivity analyses are and when they are needed is critical for you and your trial's success. If you would like to learn more about sensitivity analyses and when/how to implement them, please feel free to reach out. I would love to explain in detail and provide guidance. Better Biostatistics Better Research Happy Monday