Niloofar Jalali, PhD

I’m excited to share our latest research published in Statistics in Medicine: “Clustering-Informed Shared-Structure Variational Autoencoder for Missing Data Imputation in Large-Scale Healthcare Data” The Challenge: Missing data in electronic health records (EHR) isn't just a data quality issue; it's a major barrier to personalized medicine and accurate clinical decision-making. Traditional methods often struggle with: a)      Complex, nonlinear relationships in patient data b)      Different types of missing data mechanisms (especially MNAR) c)      Computational efficiency at scale Our Solution: CISS-VAE We developed a novel deep learning approach that: 1.      Identifies hidden missingness patterns using unsupervised clustering 2.      Builds pattern-specific variational autoencoders (VAE) with shared structures 3.      Captures complex temporal relationships in longitudinal data 4.      Employs an innovative iterative learning algorithm that continuously refines imputations, substantially improving accuracy over traditional VAE training 5.      Handles even the trickiest missing data scenarios (MNAR) Real-World Impact: We applied CISS-VAE to EHR data from early-stage breast cancer patients at Memorial Sloan Kettering Cancer Center, demonstrating superior accuracy compared to traditional methods like MissForest and contemporary approaches like GAIN. This work represents a step forward in leveraging AI to improve healthcare analytics, treatment monitoring, and ultimately, patient outcomes. Code available at: https://lnkd.in/eEswSCYx Huge thanks to my collaborators Kenneth Seier, Katherine S. Panageas, Danielle Vaithilingam, Mithat Gönen, and Yuan Chen! 🔗 Read the full paper: [https://lnkd.in/egq7vpPg] #HealthcareAI #MachineLearning #DataScience #ClinicalResearch #PersonalizedMedicine #DeepLearning #Biostatistics

25

Wondering about the complexity of using Perforated Backpropagation? Check out our founder Rorry Brenner's new Medium article with a step-by-step guide for implementation. A five minute read to get you started with this groundbreaking optimization technique. https://lnkd.in/gA_zGzme #pytorch #machinelearning #deeplearning #neuralnetworks #mnist

1

🧑‍💻 Soumith Chintala – Co-Creator of PyTorch and Pioneer in Generative Deep Learning ✅ Co-author of the original PyTorch paper, helping build the imperative deep learning framework adopted globally in research and industry ✅ Led the development of PyTorch's dynamic graph design, making model building more flexible and intuitive ✅ Co-authored the seminal DCGAN paper, introducing convolutional GANs for unsupervised representation learning ✅ Contributed to the influential Wasserstein GAN, stabilizing GAN training and improving generative quality ✅ Advanced deep learning infrastructure with work on distributed training, GPU performance, and scalable systems 🌍 A driving force in open-source AI, his contributions laid the foundation for today’s generative models and accelerated deep learning innovation worldwide #AI #DeepLearning #PyTorch #GANs #MachineLearning #SoumithChintala #OpenSourceAI

1

When NVIDIA's DGX Spark Benchmarks Meet Production Reality WARNING: Tech Heavy Post I spent 6+ days running intensive ML workloads on an NVIDIA DGX Spark—training multiple models, fine-tuning with LoRA, and benchmarking inference. Here's what NVIDIA's 82,739 tokens/sec benchmarks don't tell you: What Worked: ✅ Training performance matches claims (10-12hr for 7 LoRA experiments) ✅ Inference speed scales as expected (80 tok/sec via Ollama) ✅ 4-bit quantization quality is excellent (<1% degradation confirmed) What Didn't: ❌ GPU inference completely broken (inf/nan errors, empty responses) ❌ Memory fragmentation causes hard reboots after 3-8 hours ❌ llama.cpp direct invocation fails (ARM64 + Blackwell issues) The Plot Twist: I spent 15+ hours debugging "training failures" across 7 experiments. Turns out all the models trained perfectly—GPU inference was the problem. Testing on CPU revealed every model worked beautifully. Production Workarounds Required: - torch.cuda.empty_cache() every 50 steps - Maximum 2-3 hour training sessions - CPU inference for evaluation - Ollama for production serving My Verdict: NVIDIA's benchmarks are technically accurate. They're just not the whole truth. The hardware is powerful, but requires expert-level knowledge to navigate current limitations. Cautiously recommended for ML engineers willing to implement workarounds. Wait 6-12 months if you need stability. Read the full deep-dive with all experimental data, code samples, and production recommendations: https://bit.ly/43AGKQ8 #MachineLearning #AI #DGX #NVIDIA #ProductionML #LLM #FineTuning #MLOps #DeepLearning #GPU

87

7 Comments

🚀 Excited to share Drug Causality BERT v2 – an end‑to‑end NLP pipeline for automated adverse drug reaction causality assessment in clinical text. Built with BioBERT/Transformers, PyTorch and Streamlit, this version improves: - Classification performance for drug–event causality (suspected / concomitant / not related) - Handling of real‑world safety narratives and noisy clinical language - Usability for safety reviewers through an interactive web app (highlighted spans, probabilities, and thresholds) Key features: - Fine‑tuned biomedical language model on ADE Corpus V2 and pharmacovigilance-style reports - Token- and document-level causality scores to support case evaluation workflows - Ready for integration into pharmacovigilance and literature monitoring pipelines (e.g., safety databases, signal detection tools, medical review dashboards) This is a step towards reducing manual workload for safety teams and making signal detection and case review faster, more consistent, and more scalable. 🔗 GitHub / Demo / Model card: • GitHub: https://lnkd.in/gk7nhvsS • Live Demo: https://lnkd.in/gpD9PZB6 • Model: https://lnkd.in/g9q27DHz Open to feedback and ideas from: - Pharmacovigilance and safety science professionals - NLP / ML engineers working on real‑world clinical text - Teams building drug safety automation tools #NLP #Pharmacovigilance #DrugSafety #MachineLearning #BioBERT #HealthcareAI

7

🚀 A/B Testing vs Causal Inference A/B testing and causal inference solve different identification problems. 👉 A/B testing estimates causal effect under randomized controlled trials. Random assignment breaks the link between treatment and confounders, making the average treatment effect identifiable with simple estimators such as t tests or z tests. This setup is ideal for UI changes, ranking tweaks, or online ad experiments where exposure can be controlled. 👉 Causal inference estimates effect from observational data where treatment is not randomly assigned. User choice, targeting rules, or operational constraints introduce confounding. Techniques like propensity score matching, difference in differences, regression discontinuity, and synthetic controls are used to approximate the counterfactual outcome. Example - A product team randomly assigns users to a new checkout flow and measures conversion lift. This is A/B testing because treatment assignment is independent of user behavior. - A marketing team evaluates the impact of TV ads across regions where campaigns were selectively launched. This is causal inference because exposure depends on geography and business strategy. 💡 Key distinction - A/B testing identifies causality by design. - Causal inference identifies causality by assumptions and modeling. Strong analytics comes from matching the method to the data generating process, not from applying statistical tests blindly. ➕ Follow Shyam Sundar D. for practical learning on Data Science, AI, ML, and Agentic AI 📩 Save this post for future reference ♻ Repost to help others learn and grow in AI #DataScience #ABTesting #CausalInference #CausalML #Experimentation #ProductAnalytics #Statistics

29