Latent Variable Models

Behind every complex dataset lies structure we can’t see directly. People differ in patterns, not just averages. Behaviors co-vary for reasons that aren’t obvious. Latent modeling helps uncover these hidden structures. Principal Component Analysis (PCA) takes many correlated variables and transforms them into fewer uncorrelated components that retain most of the original variance. Each component is a linear combination of the initial variables, capturing how they vary together. PCA simplifies data, reduces noise, and helps visualize multidimensional relationships. It relies on eigenvalues and eigenvectors of the correlation matrix and is data-driven; it describes structure without inferring causes. Factor Analysis (FA) goes further by assuming correlations among variables stem from hidden factors such as traits or abilities. Each observed measure reflects both common factors and unique variance. Exploratory FA searches for these latent dimensions, while Confirmatory FA tests whether a proposed model fits new data. FA accounts for measurement error and aims to reveal theoretical constructs rather than just summarize data. Estimation involves solving for factor loadings and variances through maximum likelihood or least squares and assessing how well the structure explains observed relationships. Latent Class Analysis (LCA) shifts focus from variables to people. It applies to categorical data such as survey responses or ratings and assumes the population contains unobserved subgroups defined by similar response patterns. Each person’s answers are explained by their membership in a latent class, and the model estimates both class sizes and membership probabilities. LCA reveals population heterogeneity, showing that similar averages can hide very different subgroups. Latent Profile Analysis (LPA) extends this idea to continuous data. It assumes individuals belong to profiles characterized by distinct response patterns; one group may show high scores, another moderate, another low. These profiles can be interpreted as types within a population. Like LCA, LPA is a finite mixture model estimated using algorithms such as Expectation - Maximization. Criteria like AIC, BIC, and entropy guide how many profiles best fit the data. LPA exposes structured diversity without forcing arbitrary cutoffs. Latent Dirichlet Allocation (LDA) applies the same principle to text. It models each document as a mixture of topics and each topic as a distribution of words. A document might contain several topics in varying proportions, revealing recurring themes across a corpus. LDA uses Bayesian inference through variational methods or Gibbs sampling to estimate these distributions. It supports large-scale qualitative analysis, identifying emergent ideas and linguistic patterns without manual coding. Topics are probabilistic, adapting as new data appear.

Beyond Regression: Why Structural Equation Modeling (SEM) is a Game Changer for UX Research In user experience research, the tools we choose can shape the depth and clarity of our findings. Regression analysis, a reliable workhorse for many researchers, often provides an excellent starting point for identifying relationships between variables. But when it comes to uncovering the nuanced and interconnected dynamics of user behavior, regression may not always be enough. This realization hit home during a project on optimizing visual design for memory recall using the Rule of Thirds, where structural equation modeling (SEM) proved invaluable. Initially, regression helped us establish a direct relationship between the visual alignment of elements and memory performance. It was quick and clear, showing a correlation that seemed actionable. However, the more we probed, the more evident it became that we were missing the full picture. Turning to SEM, we were able to model not just the direct effects but also the indirect relationships, like how visual attention mediated the link between alignment and memory. SEM also allowed us to explore latent variables, such as user focus, which regression couldn’t adequately address. The insights were richer, more actionable, and far better aligned with the complexity of real-world user interactions. So, what’s the real difference between regression and SEM? Regression shines when the relationships between variables are straightforward and linear. It’s efficient and excellent for testing direct effects. But UX research often deals with interconnected systems where user satisfaction, cognitive load, and task completion influence each other in intricate ways. SEM steps in here as a more advanced method that models these complexities. It allows you to include latent variables, account for indirect effects, and visualize the interplay between multiple factors, all within a single framework. One of the most valuable aspects of SEM is its ability to uncover relationships you might not even think to test using regression. For example, while regression can tell you that a particular design change improves task completion rates, SEM can show how that improvement is mediated by reduced cognitive load or increased user trust. This kind of insight is critical for designing experiences that go beyond surface-level success metrics and truly resonate with users. To be clear, this isn’t an argument to abandon regression altogether. Each method has its place in a researcher’s toolkit. Regression is great for quick analyses and when the problem is relatively simple. But when your research involves complex systems or layered relationships, SEM provides the depth and clarity needed to make sense of it all. Yes, it’s more resource-intensive and requires a steeper learning curve, but the payoff in terms of actionable insights makes it worth the effort.

What you know about Logistic regression, might only be the tip of the iceberg. There is a whole other model hidden below the surface - quite literally. Logistic regression can be interpreted through the lens of a latent variable model. The concept is elegantly straightforward. In logistic regression, we observe an outcome variable, y, which takes on the values of either 1 or 0. Now imagine there exists another unobserved variable y* that is determined by a linear model and can be any real number. This latent variable y* is the key to unlocking the observed outcomes. Specifically, we observe y=1, whenever y*>0. Otherwise, we get y=0. The intriguing twist comes into play with the distribution of error terms in this latent model for y*. If these errors follow a logistic distribution, the observed outcomes y follow a logistic model. On the other hand, if the errors are normally distributed, we step into the domain of the so-called Probit model. This perspective enriches our understanding of logistic regression and bridges our comprehension to other statistical models, illuminating the interconnectedness of seemingly disparate methods. #datascience #statistics

Microsoft Research just unveiled a groundbreaking paper that could redefine how we approach core machine learning tasks! Introducing the Latent Zoning Network (LZN), a unified principle for generative modeling, representation learning, and classification, set to appear at NeurIPS 2025. Instead of disparate solutions, LZN proposes a single, shared Gaussian latent space. Each data type – be it images, text, or labels – is equipped with an encoder to map samples to disjoint latent zones, and a decoder to translate latents back to data. This elegant framework allows various ML tasks to be expressed as compositions of these encoders and decoders, enabling unparalleled synergy. The results are truly impressive: - LZN enhances existing state-of-the-art generative models, improving image quality (e.g., boosting FID on CIFAR10) without altering their training objectives. - For unsupervised representation learning, LZN outperforms seminal methods like MoCo and SimCLR on ImageNet. - It achieves both state-of-the-art classification accuracy and improved FID when performing joint generation and classification on CIFAR10. This is a significant step towards simplifying ML pipelines and fostering positive transfer across tasks. The Microsoft team has generously open-sourced the code and pre-trained models. We encourage all researchers to explore this innovative work and consider Hugging Face as your platform for sharing new papers and models. It makes your valuable contributions more discoverable and accessible to the entire AI community! Read the paper: https://lnkd.in/e3EjrDut Find the models: https://lnkd.in/eiM-Uu8S Explore the code: https://lnkd.in/eUi55nbz Project website: https://lnkd.in/e2s6thut

Linear Factor Models (LFM) Imagine being at a cocktail party where many people are speaking at the same time. 🍸 What you hear at any single location is not an individual voice, but a superposition of many voices mixed together. Isolating one speaker from this mixture is difficult from a single recording. 🎙️ Now suppose we place multiple microphones at different locations across the room. Each microphone captures a different linear combination of the same underlying voices. The question then becomes whether we can combine these recordings to recover the voice of a single person. This is known as the cocktail party problem. ---------------------------------------------------------------------------- This challenge extends far beyond social settings. In neuroscience, for example, we aim to identify the activity of specific neuronal populations. However, EEG sensors do not measure individual neurons. Each electrode records a mixture of electrical signals generated by millions or even billions of neurons. The task is therefore to recover the activity of latent neuronal clusters from these mixed observations. ---------------------------------------------------------------------------- A natural approach is to assume that each observed signal is a linear combination of a smaller number of latent sources, corrupted by noise. By learning appropriate linear weights, we attempt to separate these hidden sources from the observations. Formally, the observations depend on latent variables, and are assumed to be conditionally independent given those latents. This assumption leads to the framework of linear factor models. ---------------------------------------------------------------------------- By introducing different assumptions at various stages, such as independence, sparsity, or Gaussian structure of the latent variables, this general framework gives rise to methods including Independent Component Analysis, sparse coding, and probabilistic Principal Component Analysis.

🎨 [Primer] Diffusion Models: http://diffusion.aman.ai - Diffusion models reverse a gradual noising process to transform pure noise into structured data, offering a stable probabilistic alternative to GANs & autoregressive Transformers. - This primer develops an end-to-end, view of diffusion modeling from theory to practice, covering discrete-time & continuous-time formulations, DDPMs, DDIMs, latent diffusion, score-based & flow-matching models, training objectives, architectures (U-Nets & DiTs), conditioning & guidance, & real-world systems for image, video, & multimodal generation. - The highlight of the primer is a dedicated section on integrating Diffusion Models with a LLM backbone to leverage the reasoning capabilities of LLMs for structured planning, compositional control, & semantic guidance of generation. 🔹 Background: Transformers vs. Diffusion Models • High-level Comparison • Training Objectives • Computational Trade-offs • Convergence: Transformer Backbones Inside Diffusion (DiT) 🔹 Advantages • High-Fidelity Sample Quality • Non-Adversarial Training 🔹 Diffusion Models: the Theory • Diffusion Models As Latent-Variable Models • Markovian Structure • Simplifying Variational Lower Bound to MSE Loss: Why Noise Prediction Works 🔹 Diffusion Models: The Math Under-the-Hood 🔹 Taxonomy of Diffusion Models • Discrete-Time   - Pixel-Space     - Denoising Diffusion Probabilistic Models (DDPMs)     - Denoising Diffusion Implicit Models (DDIMs)   - Latent-Space     - Latent Diffusion Models (LDMs) • Continuous-Time   - Stochastic Differential Equation (SDE)-Based   - Score-Based Generative Modeling (SGMs)   - Probability Flow ODE-Based   - Flow Matching Models • Comparative Analysis 🔹 Network Architecture: U-Net and Diffusion Transformer (DiT) • U-Net-Based Diffusion Models • Diffusion Transformers (DiT) • U-Net vs. DiT Architectures 🔹 Conditional Diffusion Models • Text Conditioning in Diffusion Models   - Concatenation vs. Cross-Attention Conditioning • Visual Conditioning in Diffusion Models   - Feature Map Injection via Cross-Attention (FiLM) • Multi-Modal Conditioning (Text + Image(s) + Other Modalities) 🔹 Classifier-Free Guidance • How Classifier-Free Guidance Works (Dual Training Path) 🔹 Video Diffusion Models • Architecture: Spatiotemporal U-Nets and Diffusion Transformers • Conditioning and Temporal Coherence • Video Editing via Diffusion 🔹 Evaluation Metrics • Text-to-Image • Text-to-Video  🔹 Prompting Text-to-Image & Text-to-Video Models 🔹 Integrating Diffusion Models with an LLM Backbone • LLM Backbone (Semantic Planner) • Projection Layers (LLM-Diffusion Interface) • Diffusion Model (Perceptual Decoder) 🔹 Diffusion Models in PyTorch: HuggingFace Diffusers 🔹 FAQs 🔹 Relevant Papers 🔹 Fine-tuning Diffusion Models 🔹 Diffusion Model Preference Optimization (Diffusion-DPO) by Stanford University Primer written in collaboration with Vinija Jain. #artificialintelligence #genai #llm

Humans build an intuitive “world model” largely from a continuous stream of observations: we watch what changes, infer what likely caused it, and gradually get better at predicting what will happen next, often using language to conceptualise the action taking place in between. That’s the inspiration behind our new preprint, “Self-Improving World Modelling with Latent Actions”, led by my amazing student Yifu QIU: we introduce SWIRL ꩜, a framework to help foundation models (LLMs and VLMs) learn world modelling from state-only sequences (just “before” → “after”), without requiring expensive action-labelled trajectories. The key idea is to treat the missing action as a *latent variable grounded on language* and iteratively refine two models (initialised with pre-trained weights): - Forward World Model (FWM): predicts the next state given the current state and latent action; - Inverse Dynamics Model (IDM): predicts the latent action given a state transition. SWIRL ꩜ alternates two RL phases (via GRPO), where each model in turn acts as a policy updated with the other frozen model’s reward: FWM is rewarded when IDM can reliably “explain” its generated futures (encourages identifiable dynamics). IDM is rewarded when FWM assigns a high probability to the observed transition (encourages fidelity to real data). SWIRL ꩜ was tested across visual and textual/digital environments using state-only sequences for iterative self-improvement. The environments include (1) open-world visual dynamics for VLMs (single-step next-observation prediction and multi-step rollouts), and (2) text-based simulated worlds, web/HTML interaction dynamics, and tool-use execution dynamics for LLMs. Across these settings, SWIRL ꩜ consistently improves over backbone models (i.e., Liquid as a VLM and Qwen as an LLM) after a short SFT warm-up, achieving gains of +16% (AURORA-BENCH), +28% (ByteMorph), +16% (WorldPredictionBench), +14% (StableToolBench). If you’re interested in scaling world modelling from unlabeled video/web/tool traces, I’d love to hear your thoughts! Authors: Yifu QIU, Zheng Zhao, Waylon Li, Yftah Ziser, Anna Korhonen, Shay Cohen and me. arXiv paper: https://lnkd.in/eZgYBQGs Code: https://lnkd.in/eEWf6Szv Huggingface paper: https://lnkd.in/esa5TUgc #WorldModels #LLM #VLM #ReinforcementLearning

𝐂𝐚𝐮𝐬𝐚𝐥 𝐄𝐬𝐭𝐢𝐦𝐚𝐭𝐢𝐨𝐧 𝐰𝐢𝐭𝐡 𝐋𝐚𝐭𝐞𝐧𝐭 𝐕𝐚𝐫𝐢𝐚𝐛𝐥𝐞𝐬 𝐢𝐧 𝐃𝐀𝐆𝐬 𝐔𝐬𝐢𝐧𝐠 𝐏𝐲𝐭𝐡𝐨𝐧 In observational studies, the unconfoundedness assumption is rarely satisfied in practice. However, for every challenge, a solution eventually emerges, even if it is partial or contingent on specific conditions. Several Python libraries address this issue by working directly with causal structures and latent variables. In this post, want to share some insights into how these tools handle such complexities. 🏗️ 𝐀𝐧𝐚𝐧𝐤𝐞 𝐚𝐧𝐝 𝐅𝐫𝐨𝐧𝐭-𝐝𝐨𝐨𝐫 / 𝐈𝐃 𝐀𝐥𝐠𝐨𝐫𝐢𝐭𝐡𝐦 Ananke focuses on causal graphs with latent variables (ADMGs), where hidden confounders are represented explicitly. ➡️ It implements the ID algorithm and supports front-door adjustment. ➡️ If a mediator M satisfies front-door conditions, the effect T → Y can be identified even in the presence of an unobserved U. ➡️ This is one of the few tools that answers the key question first: Is the effect identifiable at all under hidden confounding? ➡️ Only after that, estimation makes sense. 🔗 𝐏𝐫𝐨𝐱𝐢𝐦𝐚𝐥 𝐂𝐚𝐮𝐬𝐚𝐥 𝐈𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞: 𝐏𝐫𝐨𝐱𝐢𝐞𝐬 𝐈𝐧𝐬𝐭𝐞𝐚𝐝 𝐨𝐟 𝐈𝐧𝐬𝐭𝐫𝐮𝐦𝐞𝐧𝐭𝐬 Valid instrumental variables are rare in real business and behavioral data. Proximal methods replace instruments with proxy variables for latent confounders: ➡️ one proxy linked to treatment, ➡️ one proxy linked to outcome. Under completeness and bridge-function assumptions, these proxies allow consistent estimation despite hidden U. In practice, this approach is useful when: 🔸 hidden confounding is unavoidable, 🔸 good proxies exist, 🔸 IV assumptions are implausible. 👉 Currently supported via extensions in Ananke (Eff-AIPW) and CausalML (Proximal Learners). 🧩 𝐄𝐱𝐩𝐥𝐢𝐜𝐢𝐭 𝐋𝐚𝐭𝐞𝐧𝐭 𝐕𝐚𝐫𝐢𝐚𝐛𝐥𝐞 𝐌𝐨𝐝𝐞𝐥𝐢𝐧𝐠 Instead of bypassing hidden factors, some approaches model them directly. In CausalNex library, latent-variable Bayesian networks with EM estimation allow researchers to introduce unobserved concepts such as engagement, risk attitude, or market pressure. This enables counterfactual reasoning with hidden states, but requires: ➡️ strong priors, ➡️ domain constraints, ➡️ partial supervision or expert input. ❗ Without these, solutions are often unstable and hard to interpret. 🗝️ 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐓𝐚𝐤𝐞𝐚𝐰𝐚𝐲𝐬 𝐟𝐨𝐫 𝐃𝐞𝐜𝐢𝐬𝐢𝐨𝐧 𝐒𝐜𝐢𝐞𝐧𝐜𝐞 ➡️ Identification before estimation: never apply complex models to non-identifiable effects. Establishing a structural identification strategy must precede numerical optimization. ➡️ Assumptions as first-class objects: every estimate is only as credible as its underlying assumptions. Treat these as explicit, testable components of your model. ➡️ Transparent uncertainty: combining point estimates with formal sensitivity analysis provides stakeholders with explicit bounds on the risks posed by unobserved confounding. #CausalInference #DataScience #DecisionScience #DataDriven #DecisionMaking

Three generative paradigms. One goal: novel molecules. Which one should you use? 𝗩𝗔𝗘𝘀 (𝗩𝗮𝗿𝗶𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗔𝘂𝘁𝗼𝗲𝗻𝗰𝗼𝗱𝗲𝗿𝘀) How it works: Encode molecules to latent space, decode to generate ✅ Fast sampling (single forward pass) ✅ Smooth, interpretable latent space ✅ Easy to condition ✅ Well-understood theory ❌ Mode collapse issues ❌ Blurry/average outputs ❌ KL divergence trade-off ❌ Limited sample quality 𝗗𝗶𝗳𝗳𝘂𝘀𝗶𝗼𝗻 𝗠𝗼𝗱𝗲𝗹𝘀 How it works: Iteratively denoise from Gaussian noise ✅ State-of-the-art sample quality ✅ Stable training ✅ Flexible conditioning (classifier-free guidance) ✅ Captures complex distributions ❌ Slow sampling (many steps) ❌ High compute requirements ❌ Difficult to get exact likelihood 𝗙𝗹𝗼𝘄 𝗠𝗮𝘁𝗰𝗵𝗶𝗻𝗴 How it works: Learn continuous transformation from noise to data ✅ Fast sampling (few steps) ✅ Exact likelihood computation ✅ Simulation-free training ✅ Combines best of both worlds ❌ Newer, less mature ❌ Implementation complexity ❌ Still being optimized for molecules 𝗥𝗲𝗰𝗼𝗺𝗺𝗲𝗻𝗱𝗮𝘁𝗶𝗼𝗻𝘀 → Quick prototyping: VAE → Best quality, compute available: Diffusion → Production, speed matters: Flow Matching → Uncertainty quantification needed: VAE or Flow #GenerativeAI #DrugDiscovery #MachineLearning #VAE #DiffusionModels #FlowMatching #AIDD

Factor Analysis vs. Principal Component Analysis Many people often confuse factor analysis (FA) and principal component analysis (PCA). While both are dimensionality reduction techniques, they serve different purposes. Principal Component Analysis (PCA) Principal Component Analysis is a technique that transforms the original variables into a new set of uncorrelated variables called principal components. These principal components are linear combinations of the original variables, and they are ordered in such a way that the first principal component explains the maximum possible variance in the data, the second principal component explains the next highest variance, and so on. The main goals of PCA are: 1. Variance Explanation: PCA aims to explain as much of the total variance in the dataset as possible. This is achieved by finding principal components that capture the maximum variance. 2. Dimensionality Reduction: By selecting a subset of the principal components, PCA reduces the dimensionality of the data while retaining most of the variability present in the original variables. 3. Orthogonality: Principal components are orthogonal to each other, ensuring that they capture distinct aspects of the data’s variance. Factor Analysis (FA) Factor Analysis is a statistical method used to identify latent variables, or factors, that explain the observed correlations among the original variables. These latent factors are not directly observed but are inferred from the patterns of covariance among the observed variables. The primary objectives of FA are: 1. Covariance Explanation: FA focuses on explaining the covariance among the original variables. It seeks to uncover underlying factors that account for the shared variance. 2. Latent Variables: The goal is to identify a smaller number of unobserved factors that can describe the relationships among the observed variables. These factors are assumed to be the source of the observed correlations. 3. Model-Based Approach: FA is based on a specific model where the observed variables are expressed as linear combinations of the factors plus unique error terms. Key Differences 1. Purpose: PCA aims to reduce dimensionality by explaining the total variance in the data, while FA seeks to uncover latent factors that explain the covariance among variables. 2. Components vs. Factors: PCA produces principal components that are linear combinations of the original variables and aim to capture as much variance as possible. FA identifies latent factors that are inferred from the observed variables and aims to explain the covariance structure. 3. Variance vs. Covariance: PCA focuses on maximizing variance explained by the components, whereas FA focuses on modeling the covariance structure of the data. In summary, while both PCA and FA are used for reducing the dimensionality of data, they serve different purposes and are based on different conceptual frameworks. #quant #regression #pca #factor #variance