Latest Developments in Deep Learning Applications

Excited to share my analysis of the most groundbreaking DCN-V2 paper from Google, which introduces significant improvements to deep learning recommendation systems! Key technical highlights: >> Core Architecture - Starts with an embedding layer that handles both sparse categorical and dense features - Unique capability to handle variable embedding sizes from small to large vocabulary sizes - Cross network creates explicit bounded-degree feature interactions - Deep network complements with implicit feature interactions - Two combination modes: stacked and parallel architectures >> Key Technical Innovations - Enhanced cross layers with full matrix-based feature interaction learning instead of vector-based - Mixture of Low-Rank architecture with: * Multiple expert networks learning in different subspaces * Dynamic gating mechanism to adaptively combine experts * Efficient time complexity when specific conditions are met * Support for non-linear transformations in projected spaces >> Production Optimizations - Low-rank matrix approximation leveraging singular value decay patterns - Mixture-of-Experts decomposition into smaller subspaces - Efficient parameter allocation between cross and deep networks - Automatic feature interaction learning for higher-order interactions in multi-layered networks - Support for both homogeneous and heterogeneous polynomial patterns >> Real-World Impact - Successfully deployed across Google's recommendation systems - Significant gains in both offline accuracy and online metrics - Better performance-latency tradeoffs through low-rank approximations - Proven effectiveness on large-scale data with billions of training examples This represents a major leap forward in making deep learning recommendation systems more practical and efficient at scale. Thoughts? Would love to hear your experiences implementing similar architectures in production!

AI progress has long been dominated by raw scale—larger datasets, bigger models, and massive compute budgets. But recent breakthroughs suggest that efficiency in training, retrieval, and reasoning may now be more important than brute force scaling. The first shock came with DeepSeek-R1, an open-source model that demonstrated that reinforcement learning (RL) alone—without extensive supervised fine-tuning—can develop reasoning capabilities comparable to proprietary models [1]. This shift is reinforced by Qwen 2.5’s architecture optimizations and Janus-Pro’s multimodal advancements, proving that cheaper, faster, and more effective AI is possible without simply increasing parameter counts [2]. DeepSeek-R1 shows that RL can be a primary mechanism for improving LLM reasoning, not just an alignment tool [1]. Its initial version, DeepSeek-R1-Zero, trained purely via RL, displayed strong reasoning but suffered from readability issues. The refined DeepSeek-R1, incorporating minimal cold-start data and rejection sampling fine-tuning, reached OpenAI-o1-1217-level performance at a fraction of the cost. This challenges the conventional pretraining-heavy paradigm. AI architecture is also undergoing a fundamental shift. Janus-Pro, from DeepSeek-AI, introduces a decoupled approach to multimodal AI, separating image understanding from image generation [2]. Unlike previous models that forced both tasks through a shared transformer, Janus-Pro optimizes each independently, outperforming DALL-E 3 and Stable Diffusion 3 Medium in instruction-following image generation. At a more fundamental level, Bytedance’s Over-Tokenized Transformers reveal a silent inefficiency in LLM design: tokenization is a bottleneck [3]. Their research shows that expanding input vocabulary—while keeping output vocabulary manageable—drastically reduces training costs and improves performance. A 400M parameter model with an optimized tokenizer matched the efficiency of a 1B parameter baseline (!), proving that many LLMs are computationally bloated due to suboptimal tokenization strategies. Beyond efficiency, AI is also becoming more structured in reasoning and retrieval. Google DeepMind’s Mind Evolution introduces a genetic algorithm-like refinement process [4], evolving multiple solution candidates in parallel and iteratively improving them. This could lead to AI systems that autonomously refine their own answers rather than relying on static generation. Meanwhile, Microsoft’s CoRAG is redefining RAG by solving the multi-hop retrieval challenge [5]. Standard RAG models retrieve once before generating a response, failing on multi-step queries. CoRAG introduces recursive retrieval, dynamically reformulating queries at each step, leading to a 10+ point improvement on multi-hop QA benchmarks. The combined effect of these breakthroughs is a shift in how AI is trained, how it retrieves knowledge, and how it reasons in real time - everything you need to design more intelligent brains.

𝘞𝘦'𝘷𝘦 𝘳𝘦𝘭𝘦𝘯𝘵𝘭𝘦𝘴𝘴𝘭𝘺 𝘴𝘤𝘢𝘭𝘦𝘥 𝘛𝘳𝘢𝘯𝘴𝘧𝘰𝘳𝘮𝘦𝘳𝘴, 𝘣𝘶𝘵 𝘢𝘳𝘦 𝘸𝘦 𝘩𝘪𝘵𝘵𝘪𝘯𝘨 𝘢 𝘤𝘰𝘮𝘱𝘶𝘵𝘢𝘵𝘪𝘰𝘯𝘢𝘭 𝘸𝘢𝘭𝘭? 𝘛𝘩𝘦 𝘲𝘶𝘢𝘥𝘳𝘢𝘵𝘪𝘤 𝘤𝘰𝘮𝘱𝘭𝘦𝘹𝘪𝘵𝘺 (𝘖(𝘕^2)) 𝘰𝘧 𝘴𝘦𝘭𝘧-𝘢𝘵𝘵𝘦𝘯𝘵𝘪𝘰𝘯 𝘪𝘴 𝘢 𝘧𝘶𝘯𝘥𝘢𝘮𝘦𝘯𝘵𝘢𝘭 𝘣𝘰𝘵𝘵𝘭𝘦𝘯𝘦𝘤𝘬 𝘧𝘰𝘳 𝘭𝘰𝘯𝘨-𝘤𝘰𝘯𝘵𝘦𝘹𝘵 𝘈𝘐. 𝘞𝘩𝘢𝘵 𝘪𝘧 𝘵𝘩𝘦 𝘧𝘶𝘵𝘶𝘳𝘦 𝘪𝘴𝘯'𝘵 𝘫𝘶𝘴𝘵 𝘢𝘣𝘰𝘶𝘵 𝘣𝘪𝘨𝘨𝘦𝘳 𝘮𝘰𝘥𝘦𝘭𝘴, 𝘣𝘶𝘵 𝘧𝘶𝘯𝘥𝘢𝘮𝘦𝘯𝘵𝘢𝘭𝘭𝘺 𝘧𝘢𝘴𝘵𝘦𝘳 𝘢𝘳𝘤𝘩𝘪𝘵𝘦𝘤𝘵𝘶𝘳𝘦𝘴? For real-world applications in RAG, complex agents, and multimodal systems, efficiently processing vast context windows is the difference between a tech demo and a deployed product. A comprehensive new survey, "𝐒𝐩𝐞𝐞𝐝 𝐀𝐥𝐰𝐚𝐲𝐬 𝐖𝐢𝐧𝐬: 𝐀 𝐒𝐮𝐫𝐯𝐞𝐲 𝐨𝐧 𝐄𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐭 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞𝐬 𝐟𝐨𝐫 𝐋𝐚𝐫𝐠𝐞 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬," shows this exact shift. The paper dismantles the challenges of the vanilla Transformer and provides alternatives that are breaking the efficiency barrier. The survey systematically explores a new wave of architectures designed for speed and scale. The core innovation it highlights is the move towards linear-time models (O(N)). Architectures like State Space Models (e.g., Mamba), Linear RNNs (e.g., RWKV), Sparse Mixture-of-Experts (MoE), and various hybrid designs are no longer niche experiments. The key finding is that these efficient models are rapidly closing the performance gap. They are achieving competitive results by fundamentally changing the scaling laws, enabling massive context windows and significantly faster inference with a fraction of the traditional computational cost. This evolution could access to powerful AI by lowering deployment costs, enable real-time reasoning on streaming data, and ultimately force us to rethink the design of next-generation foundation models. #AI #MachineLearning #DeepLearning #LLM #AIArchitecture

🟥 Deep Learning for Rare Cell Type Detection in Cancer In cancer research, detection of rare cell populations, such as cancer stem cells, drug-resistant clones, or metastatic precursors, is critical to understanding tumor progression, treatment failure, and relapse. However, identification of these elusive cell types is challenging due to their low abundance and subtle molecular signatures. Recent advances in deep learning (DL) have enabled more accurate and scalable detection of rare cell types from complex single-cell datasets, revolutionizing our ability to analyze tumor heterogeneity. Deep learning models, especially convolutional neural networks (CNNs), variational autoencoders (VAEs), and graph neural networks (GNNs), have been trained on large-scale single-cell RNA sequencing (scRNA-seq), ATAC-seq, and spatial transcriptome datasets to classify cell types based on gene expression, chromatin accessibility, and spatial context. These models excel at recognizing nonlinear patterns and capturing subtle transcriptional features that may be missed by traditional clustering or dimensionality reduction methods. One breakthrough has been the development of unsupervised or semi-supervised deep learning models that can learn without large amounts of labeled data, making them ideal for exploratory cancer research. In addition, attention-based architectures can also assign importance scores to specific genes or pathways, providing interpretable insights into the biological basis of rare cell identities. Applications of these techniques include discovering drug-resistant clones in leukemia, detecting dormant tumor cells in breast cancer, and identifying aggressive phenotypes in glioblastoma. Deep learning has also been used to integrate single-cell multi-omics data (e.g., transcriptomics + epigenomics) to refine the definition of rare cancer subtypes. In summary, deep learning has enabled researchers to discover hidden cell subpopulations that play a key role in disease progression and treatment resistance. As models become more interpretable and generalizable, this approach has great potential to improve cancer diagnosis and monitoring, as well as design more effective and personalized therapies. References [1] Bora Uyar et al., bioRxiv 2021 (doi: https://lnkd.in/eSZZFSXZ) [2] Yichun Zhao et al., Advanced Science 2025 (https://lnkd.in/ecV6y7Ym) [3] Xiaoying Wang et al., Nature Communications 2024 (https://lnkd.in/eRMN9r5G) #DeepLearning #SingleCellAnalysis #RareCellDetection #CancerResearch #AIinBiomedicine #scRNAseq #TumorHeterogeneity #CancerStemCells #DrugResistance #MachineLearning #PrecisionOncology #MultiOmics #Bioinformatics #CellTypeIdentification #DigitalPathology #CSTAMBiotech

AI is getting closer to accessing the one thing we’ve always considered private: your thoughts. Recent advances in neuro-AI can now identify whether a person recognizes specific information using EEG signals. A 2025 study using deep-learning reached 86.7% accuracy in detecting recognition through the P300 brain wave: a response triggered before conscious awareness. Meanwhile, some jurisdictions are already experimenting with this technology. 🇮🇳 India has used brain-mapping techniques in hundreds of criminal investigations, showing just how quickly neuroscience can enter real-world decision systems. But the implications go beyond law enforcement. AI models can now (fMRI + diffusion models): Reconstruct visual experiences directly from brain activity ✔️ Models that reconstruct what you’re seeing — in near real-time — based solely on your brain activity (Think: AI generating the images your eyes are looking at.) Decode unspoken language in early experimental settings ✔️ Models that reconstruct the words you’re thinking, even if you never speak A 2023–2024 wave of studies using fMRI + LLMs demonstrated the ability to decode semantic meaning of inner speech—turning thoughts into text-like outputs. This raises critical questions for business leaders, policymakers, and innovators: How do we prepare for a world where cognitive data becomes a new category of sensitive information? What safeguards, standards, and governance frameworks will protect mental privacy as neuro-AI scales? The technology is advancing faster than the regulations around it and the organisations that understand this early will be better positioned to navigate what comes next. #AI #Neuroscience #Innovation #Leadership #Ethics #FutureOfWork Reference: Kim, S., Cheon, J., Kim, T., Kim, S. C., & Im, C.-H. (2025). Improving electroencephalogram-based deception detection in concealed information test under low stimulus heterogeneity. arXiv. https://lnkd.in/dyVqBbG3 Takagi & Nishimoto (2022). High-resolution image reconstruction with latent diffusion models from human brain activity. BioRxiv. https://lnkd.in/dfc32mS7 Tang, J., LeBel, A., Jain, S. et al. Semantic reconstruction of continuous language from non-invasive brain recordings. Nat Neurosci 26, 858–866 (2023). https://lnkd.in/dnQxcS_d

This could be a watershed moment for AI as the 'Deep Learning' era may be evolving into something new. For the last decade, the researchers and engineers have focused on enhancing AI by stacking more layers, which characterizes, the Deep Neural Networks. But a seminal new paper from Google Research for NeurIPS 2025 exposes a fundamental flaw in this approach, these models are static! Once trained, modern models are frozen in time, experiencing a form of 'anterograde amnesia' where they cannot learn from the present without forgetting the past. The paper titled 'Nested Learning: The Illusion of Deep Learning Architectures' by Ali Behrouz, Meisam Razaviyayn, Peiling Zhong, and Vahab Mirrokni proposes a paradigm shift:- Nested Learning (NL). Instead of merely stacking layers, NL reimagines models as a system of 'nested optimization problems', each operating at its own speed. Inspired by human brain waves, where high-frequency neurons manage the immediate present and low-frequency oscillations consolidate long-term memory, this approach unlocks the potential for true continual learning. Additionally, the authors introduced HOPE, a new architecture based on this paradigm. HOPE demonstrates superior performance, surpassing Transformers, RetNet, and Titans in language modeling and reasoning tasks. This could serve as the blueprint for the next generation of AI. Blog - https://lnkd.in/dQ_vermU Paper - https://lnkd.in/di8wnF7r #ArtificialIntelligence #MachineLearning #GoogleResearch #NestedLearning #ContinualLearning #AI

We just released what we think is a big jump in deep learning technology. For context: When we're using any sort of neural network (language model, vision generation model, etc.), there's a big trade-off between two things: how good the model is (performance) and how expensive it is (compute). The better the performance, the more expensive it is to run the model. This is why, for example, OpenAI or other leading labs offer multiple LLM models: they tend to pick cheap models for easy problems and expensive models for expensive queries. Having multiple trained models is quite limiting though. You need to train them separately. You also can only choose from the few you've developed. They also have different characteristics and might result in different behavior. Could you, somehow, have just one model but let it change its "size"? That is, have a single model which could become larger when you need it; or smaller when you want it to be cheaper. So, you could pick how large your model should be each time. We did exactly this. Our new-peprint presents a new architecture called Nested Subspace Networks that allows you to adjust your model size at inference (i.e. when they're being used). So, you can now have a single model, make it larger/smaller on-the-fly and at any point choose how large/small you want it to be. This turns out to be super useful. For example, we show a really nice performance-compute frontier -- we get steady decreases in performance as we make the model cheaper (and smaller). Bonus: we also show how you can convert existing big language models to have this architecture and make them adaptable.

The transformer architecture was initially celebrated as a breakthrough in NLP, but ultimately enabled breakthroughs across multiple modalities. Researchers are now addressing two of its main limitations—tokenization and quadratic scaling—opening up new multi-modal applications. At its core, the self-attention mechanism central to transformers is a simple and elegant way to extract patterns from input embeddings. The source modality of these tokens (text, images, sound) and their arrival order are irrelevant *. Self-attention enables effective comparison between all tokens in a set. This differs from architectures like CNNs or RNNs, which are tailored to specific modalities. While this makes them more data efficient (with stronger inductive biases), the remarkable scalability of transformers often compensates (see comments): we can increase dataset size until the advantage of more biased models diminishes. However, creating input embeddings remains highly modality-dependent. Text input relies on tokenization, which introduces issues like language bias and challenges in reading numbers. Additionally, the quadratic scaling of self-attention limits embedding granularity. Creating 10x more embeddings from the same input requires 100x more compute. In the last months, there was increasing focus on removing tokenization bottlenecks and reducing self-attention's quadratic cost by examining input data at different scales. This includes local attention mechanisms that combine local embeddings with global attention, and neural network-based approaches that generate embeddings dynamically (see comments). I’m really excited to soon see this enable byte-level, multi-modal models with unprecedented performance, speed and cost-effectiveness. As a bonus, 2025 might go down as the year we finally moved beyond tokenization and its quirks. #deeplearning #llms #genai * It might of course help (or even be needed) to add constraints (e.g. causal attention) or additional biases/information (e.g. positional encodings) depending on the modality to optimize. But the general idea of self-attention is really powerful irrespective of the modality, and enables us to mix modalities. * * Essentially implementing a form of fully-connected graph neural network layer.

🚀 𝘃𝟮 𝗼𝗳 𝗼𝘂𝗿 𝗽𝗮𝗽𝗲𝗿 “𝗟𝗟𝗠-𝗝𝗘𝗣𝗔” 𝗶𝘀 𝗼𝘂𝘁 𝗼𝗻 𝗮𝗿𝗫𝗶𝘃! 🔍 𝐖𝐡𝐚𝐭’𝐬 𝐧𝐞𝐰? ✅ 𝗦𝗶𝗴𝗻𝗶𝗳𝗶𝗰𝗮𝗻𝘁𝗹𝘆 𝗹𝗼𝘄𝗲𝗿 𝗰𝗼𝗺𝗽𝘂𝘁𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗼𝘃𝗲𝗿𝗵𝗲𝗮𝗱 — reduced overhead from 𝟮𝟬𝟬% → 𝟮𝟱% using a simple yet effective 𝗿𝗮𝗻𝗱𝗼𝗺 𝗝𝗘𝗣𝗔-𝗹𝗼𝘀𝘀 𝗱𝗿𝗼𝗽𝗼𝘂𝘁. ✅ 𝗕𝗿𝗼𝗮𝗱𝗲𝗿 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 — extended beyond symmetric 2-view datasets to 𝗡𝗤-𝗢𝗽𝗲𝗻 (Natural Questions for open-domain) and 𝗛𝗲𝗹𝗹𝗮𝗦𝘄𝗮𝗴 (sentence completion), and tested on reasoning models like 𝗤𝘄𝗲𝗻𝟯 and 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸-𝗥𝟭-𝗗𝗶𝘀𝘁𝗶𝗹𝗹𝗲𝗱. ✅ 𝗥𝗶𝗴𝗼𝗿𝗼𝘂𝘀 𝗮𝗯𝗹𝗮𝘁𝗶𝗼𝗻𝘀 — JEPA loss design outperforms alternatives including 𝗟𝟮, 𝗠𝗦𝗘, 𝗽𝗿𝗲𝗽𝗲𝗻𝗱 [𝗣𝗥𝗘𝗗] 𝘁𝗼𝗸𝗲𝗻𝘀, 𝗖𝗼𝗱𝗲→𝗧𝗲𝘅𝘁, and 𝗜𝗻𝗳𝗼𝗡𝗖𝗘 variants. 🧩 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐋𝐋𝐌-𝐉𝐄𝐏𝐀? If you’re seeing this for the first time: LLM-JEPA introduces the 𝗝𝗼𝗶𝗻𝘁 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴 𝗣𝗿𝗲𝗱𝗶𝗰𝘁𝗶𝘃𝗲 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 (𝗝𝗘𝗣𝗔) — a self-supervised learning paradigm proven in vision — as a 𝗿𝗲𝗴𝘂𝗹𝗮𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗹𝗼𝘀𝘀 for LLMs. Combined with next-token prediction, it enables models to: 🎯 Boost fine-tuning accuracy 🧠 Resist overfitting 🌱 Work in pretraining via 𝗽𝗮𝗿𝗮𝗽𝗵𝗿𝗮𝘀𝗲-𝗯𝗮𝘀𝗲𝗱 𝗝𝗘𝗣𝗔 🌀 Induce 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱 𝗹𝗮𝘁𝗲𝗻𝘁 𝗿𝗲𝗽𝗿𝗲𝘀𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻𝘀 unseen in either base or normally fine-tuned models 🧪 The 𝘃𝟭 𝘄𝗼𝗿𝗸𝘀𝗵𝗼𝗽 𝘃𝗲𝗿𝘀𝗶𝗼𝗻 (accepted to NeurIPS 2025 UniReps + DL4C) received valuable feedback highlighting high compute cost, limited applications, and missing ablations — all fully addressed in this release. Huge thanks to the UniReps and DL4C reviewers for their constructive and insightful comments that helped shape v2. It’s been a privilege to collaborate with Yann LeCun (NYU) and Randall Balestriero (Brown) — few experiences are more inspiring than working alongside the pioneers of modern deep and self-supervised learning. The 𝗰𝗼𝗱𝗲 𝗶𝘀 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲𝗱, and we warmly invite others to experiment with it — and help explore this emerging frontier between 𝗝𝗘𝗣𝗔 𝗮𝗻𝗱 𝗟𝗟𝗠𝘀. 💻 Code: https://lnkd.in/eUX2b8iE 📄 Paper: https://lnkd.in/ers8_yzm Together with Yann and Randall, we’re already exploring new variants and applications — and look forward to sharing more soon. Stay tuned!

Meta FAIR (yes - the one where the layoffs happened) just released research on a new architecture that could change how we think about language model generation. Standard decoder Transformers are purely autoregressive - they generate one token at a time, making all decisions about structure, content, and intent implicitly through those token choices. Think of writing a movie review: current models don't decide upfront "I'm writing a negative review," they figure it out token-by-token as they go. This works, but it's unnecessarily complicated and fragile. The Free Transformer introduces an elegant solution: let the model condition on latent random variables that are learned without supervision. Using a Variational Autoencoder approach, it can make explicit structural decisions during generation - like deciding the overall sentiment before generating words - rather than reconstructing these decisions post-hoc from the tokens themselves. The implementation is remarkably efficient, requiring only 3% additional compute. Yet on an 8B parameter model trained on 1T tokens, it delivers +11% on HumanEval+, +5% on MMLU, and notable gains across reasoning benchmarks like GSM8K and MBPP. By allowing models to learn and condition on latent structure, we're giving them the freedom to organize their generative process more naturally. Which is something that, historically speaking, has often helped to elevate Deep Learning architectures. ↓ 𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com 💡