Importance of Multimodal AI

Multimodal AI is shaping a shift in healthcare by combining different kinds of patient data to improve care across diagnostics, treatment, and monitoring. 1️⃣ It links data from imaging, wearables, clinical notes, genomics, and more to create a fuller picture of patient health. 2️⃣ Imaging, physiological signals, and clinical notes are the most commonly used data types, especially in oncology, cardiovascular, and neurological disorders. 3️⃣ Intermediate fusion is the most used integration method, combining data at the feature level for better balance between complexity and interpretability. 4️⃣ These systems enable early diagnosis, prognosis, treatment planning, and real-time monitoring, with growing applications in areas like digital twins and automated reporting. 5️⃣ Personalized medicine is a major driver, with multimodal models supporting tailored treatment decisions by analyzing combined molecular, physiological, and behavioral data. 6️⃣ Despite progress, challenges remain: data heterogeneity, privacy concerns, lack of benchmarks, and regulatory constraints slow adoption. 7️⃣ Explainability is key for clinical trust. Emerging models include attention maps, concept attribution, and human-in-the-loop feedback for better transparency. 8️⃣ Energy demands of training large models have sparked interest in "green AI", focusing on efficiency and scalability in clinical settings. 9️⃣ Future systems may rely more on self-supervised and federated learning to handle data gaps and maintain privacy across institutions. 🔟 Clinical validation and regulatory reform are needed for multimodal systems to move from labs into widespread practice. ✍🏻 Florenc Demrozi, Mina Farmanbar, Kjersti Engan. Multimodal AI for Next-Generation Healthcare: Data Domains, Algorithms, Challenges, and Future Perspectives. Current Opinion in Biomedical Engineering. 2025. DOI: 10.1016/j.cobme.2025.100632 (pre-proof)

Enterprises today are drowning in multimodal data - text, images, audio, video, time-series, and more. Large multimodal LLMs promise to make sense of this, but in practice, embeddings alone often collapse nuance and context. You get fluency without grounding, answers without reasoning, “black boxes” where transparency matters most. That’s why the new IEEE paper “Building Multimodal Knowledge Graphs: Automation for Enterprise Integration” by Ritvik G, Joey Yip, Revathy Venkataramanan, and Dr. Amit Sheth really resonates with me. Instead of forcing LLMs to carry the entire cognitive burden, their framework shows how automated Multi Modal Knowledge Graphs (MMKGs) can bring structure, semantics, and provenance into the picture. What excites me most is the way the authors combine two forces that usually live apart. On one side, bottom-up context extraction - pulling meaning directly from raw multimodal data like text, images, and audio. On the other, top-down schema refinement - bringing in structure, rules, and enterprise-specific ontologies. Together, this creates a feedback loop between emergence and design: the graph learns from the data but also stays grounded in organizational needs. And this isn’t just theoretical elegance. In their Nourich case study, the framework shows how a food image, ingredient list, and dietary guidelines can be linked into a multimodal knowledge graph that actually reasons about whether a recipe is suitable for a diabetic vegetarian diet - and then suggests structured modifications. That’s enterprise relevance in action. To me, this signals a bigger shift: LLMs alone won’t carry enterprise AI into the future. The future is neurosymbolic, multimodal, and automated. Enterprises that invest in these hybrid architectures will unlock explainability, scale, and trust in ways current “all-LLM” strategies simply cannot. Link to the paper -> https://lnkd.in/gv93znbQ #KnowledgeGraphs #MultimodalAI #NeurosymbolicAI #EnterpriseAI #KnowledgeGraphLifecycle #MMKG #AIResearch #Automation #EnterpriseIntegration

Over the past few months, I’ve noticed a pattern in our system design conversations: they increasingly orbit around audio and video, how we capture them, process them, and extract meaning from them. This isn’t just a technical curiosity. It signals a tectonic shift in interface design. For decades, our interaction models have been built on clickstreams: tapping, typing, selecting from dropdowns, navigating menus. Interfaces were essentially structured bottlenecks, forcing human intent into machine-readable clicks and keystrokes. But multimodal AI removes that bottleneck. Machines can now parse voice, gesture, gaze, or even the messy richness of a video feed. That means the “atomic unit” of interaction may be moving away from clicks and text inputs toward speech, motion, and visual context. Imagine a world where the UI is stripped to its essence: a microphone and a camera. Everything else, navigation, search, configuration, flows from natural human expression. Instead of learning the logic of software, software learns the logic of people. If this plays out, the implications are profound: UX shifts from layouts to behaviors: Designers move from arranging buttons to choreographing multimodal dialogues. Accessibility and inclusion take center stage: Voice and vision can open doors, but also risk excluding unless designed with empathy. Trust and control must be redefined: A camera-first interface is powerful, but also deeply personal. How do we make it feel safe, not invasive? We may be on the cusp of the first truly post-GUI era, where screens become less about control surfaces and more about feedback canvases, reflecting back what the system has understood from us.

Your AI is blind, deaf, and limited if it only understands text. Multimodal AI creates a universal language between images, text, audio and more. 𝗪𝗵𝗮𝘁 𝗔𝗿𝗲 𝗠𝘂𝗹𝘁𝗶𝗺𝗼𝗱𝗮𝗹 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀? Traditional embeddings convert text into vectors (mathematical representations that capture semantic meaning). Multimodal embeddings transform this capability by creating a 𝘂𝗻𝗶𝗳𝗶𝗲𝗱 𝗲𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴 𝘀𝗽𝗮𝗰𝗲 where diverse content types, images, text, audio, video exist in mathematical harmony. Think of it as a universal translator that comprehends the inherent "language" of all media types equally, positioning them in a shared mathematical universe where similar concepts naturally cluster together regardless of their original format. In this unified space: - A comprehensive research PDF on photosynthesis mechanisms - High-resolution images of plant cellular structures - Natural language queries about winter photosynthesis patterns - Time-lapse videos and audio narrations of plant growth cycles ...all converge in proximity because they share conceptual relationships, despite originating from entirely different data modalities. This shared embedding space enables 𝗮𝗻𝘆-𝘁𝗼-𝗮𝗻𝘆 𝘀𝗲𝗮𝗿𝗰𝗵. The practical implications: - Text-to-image search ("Show me electron microscopy of chloroplasts during winter dormancy") - Image-to-document search (upload a plant diagram, retrieve relevant research papers) - Audio-to-visual search (match environmental sounds to corresponding imagery) 𝘊𝘰𝘯𝘴𝘪𝘥𝘦𝘳 𝘵𝘩𝘦 𝘤𝘰𝘯𝘴𝘵𝘳𝘢𝘪𝘯𝘵𝘴 𝘰𝘧 𝘵𝘳𝘢𝘥𝘪𝘵𝘪𝘰𝘯𝘢𝘭 𝘴𝘦𝘢𝘳𝘤𝘩 - 𝘺𝘰𝘶 𝘳𝘦𝘲𝘶𝘪𝘳𝘦 𝘵𝘦𝘹𝘵 𝘵𝘰 𝘧𝘪𝘯𝘥 𝘵𝘦𝘹𝘵, 𝘪𝘮𝘢𝘨𝘦𝘴 𝘵𝘰 𝘧𝘪𝘯𝘥 𝘴𝘪𝘮𝘪𝘭𝘢𝘳 𝘪𝘮𝘢𝘨𝘦𝘴. 𝗧𝗵𝗲 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀 (𝗕𝗲𝗰𝗮𝘂𝘀𝗲 𝗡𝗼𝘁𝗵𝗶𝗻𝗴 𝗧𝗵𝗶𝘀 𝗖𝗼𝗼𝗹 𝗜𝘀 𝗘𝗮𝘀𝘆) 1. 𝗖𝗿𝗲𝗮𝘁𝗶𝗻𝗴 𝗮𝗹𝗶𝗴𝗻𝗲𝗱 𝗱𝗮𝘁𝗮𝘀𝗲𝘁𝘀 - Obtaining high-quality paired data across modalities requires substantial resources and expertise 2. 𝗛𝗮𝗻𝗱𝗹𝗶𝗻𝗴 𝗺𝗼𝗱𝗮𝗹𝗶𝘁𝘆 𝗶𝗺𝗯𝗮𝗹𝗮𝗻𝗰𝗲 - Text and image data are abundant, while quality audio, motion, and sensory data remain scarce 3. 𝗠𝗼𝗱𝗲𝗹 𝗶𝗻𝘁𝗲𝗿𝗽𝗿𝗲𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 - Explaining cross-modal relationships presents unique challenges (what connects accelerometer patterns to geopolitical discourse?) Over to you: What about 𝗠𝘂𝗹𝘁𝗶𝗺𝗼𝗱𝗮𝗹 𝗥𝗔𝗚?

We have crossed a real threshold. One model can now reason across text, vision, and audio in a single context window. That’s the shift behind multimodal models. AI can now see. Read. Listen. Speak. And reason across all of it at once. This isn’t a feature upgrade. It’s a change in how models represent the world. Until recently, models worked in silos. Text here. Images there. Audio somewhere else. Multimodal models collapse those boundaries. They don’t just process inputs. They share a unified latent representation. Adoption is accelerating. More than half of GenAI production workloads now involve multiple modalities. Over 70% of enterprise AI teams are experimenting with multimodal use cases. In media and creative pipelines, teams are seeing 30–50% reductions in production time. That matters because real work is multimodal by default. Meetings plus slides. Docs plus screenshots. Voice plus intent. Context everywhere. Humans reason across all of it naturally. AI is starting to do the same. This is why multimodality matters more than benchmark gains. It moves AI from: “Answer this prompt” to “Understand what’s happening.” And once a system understands what’s happening, new behaviors emerge. It can notice changes. Flag anomalies. Interrupt at the right moment. Suggest next steps. Eventually, it can act. The hard part isn’t capability anymore. It’s design. What should the model observe? When should it speak up? What does it have permission to do? Multimodal models won’t replace single-mode tools overnight. But expectations will shift quickly. Systems that can’t see, hear, and read together will feel limited. Systems that can will feel obvious. The next wave of AI won’t feel smarter. It’ll feel more aware.

We are moving from AI that explains things to AI that sees context, understands environments, and helps execute tasks as they happen. Watching Gemini Live recognize a car in real time and walk someone through an oil change feels like a small moment, but it points to a much bigger shift. This is a multimodal system using live camera input to understand the physical world, identify the exact vehicle, and guide real actions step by step. Cameras, sensors, models, and interfaces are collapsing into a single layer of assistance that people will start to rely on, not just experiment with. That’s where the stakes change. When AI becomes an assistant people trust in real situations, poor judgment, misuse, or missing safeguards can lead to real-world damage, not just bad answers. This is exactly why guardrails, intent detection, and safety systems have to evolve alongside capability. Reliability, trust, and clear boundaries become just as important as intelligence. One thing is clear. This category of AI is pushing the frontier. The future is assistance in the real world, and the responsibility is making sure it helps people do the right things, safely, every time. Video from @thebigbazzy

Generative AI isn’t replacing radiologists but may soon assist like a well-trained resident. 🩻 👇 A new Nature Perspective presents the frontier of Multimodal Generative AI (GenMI) in healthcare. This new class of AI models is not just interpreting medical images, it’s generating narrative reports, integrating clinical history, and even offering real-time interaction with clinicians and patients. The paper calls for a shift from single-task automation to holistic, collaborative AI assistants, or what the authors term the “AI Resident.” 👉 Key Takeaways 1. Beyond Detection: Toward Narrative Intelligence GenMI models go beyond triaging or highlighting findings,they synthesize multimodal data (e.g., imaging + clinical history) into coherent, structured reports that can rival expert drafts. 2. The “AI Resident” Paradigm Envisioned as a collaborative tool, the AI resident supports clinicians in drafting reports, enables interactive querying of findings, and can even assist in patient education and trainee feedback loops. 3. Multimodal & Multispecialty Applications While radiology is the focal domain, GenMI is expanding into pathology, dermatology, ophthalmology, and endoscopy, powered by vision-language models like GPT-4V and Google’s Gemini. 4. Challenges: Bias, Hallucination & Evaluation Gaps GenMI systems are prone to hallucinations and performance drops across underrepresented populations. Traditional NLP metrics are inadequate; new benchmarks like RadBench and RadGraph F1 are being proposed. 5. A Call for Responsible Deployment Authors advocate for gradual clinical integration, open benchmarks, diverse datasets, and human-in-the-loop calibration to ensure GenMI complements not replaces expert judgment. 🎯 GenMI represents a pivotal evolution in clinical AI from task-specific tools to interactive, multimodal assistants. If deployed with care, the AI resident could reduce burnout, democratize expertise, and reshape how medical knowledge is generated, shared, and acted upon. _________________________________________________________ #radiology #machinelearning #ai #medicine #health

Atmanity is focusing on a very interesting area in conversational AI: the subtle art of knowing when to speak versus when to stay silent. Their latest research addresses a fundamental challenge that current voice AI systems struggle with—natural turn-taking in human-computer conversations. The research reveals that effective multimodal conversation requires sophisticated understanding of contextual cues beyond just speech patterns, including visual signals, emotional states, and conversation dynamics. Traditional rule-based approaches to conversation management fall short when dealing with the nuanced timing of real human interaction. Their findings suggest that mastering these conversational protocols is critical for voice AI deployment success. Systems that can appropriately gauge when to respond, when to wait, and when to acknowledge without speaking create significantly more natural user experiences than those focused purely on speech recognition accuracy. This work highlights a fundamental gap between current voice AI capabilities and human conversational expectations - one that could determine which systems succeed in real-world applications. #ConversationalAI #VoiceAI #MultimodalAI

Can AI detect sarcasm, irony, and condescension? We explored how multimodal learning might hold the key to solving this problem. Understanding covert deception in communication - where the connotation of a message diverges from its literal meaning - is a nuanced task that even advanced AI struggles with. In our paper, “Yeah Right!” - Do LLMs Exhibit Multimodal Feature Transfer?, we investigated whether large language models (LLMs) trained on multimodal data (speech + text) or on human-to-human conversations can better navigate these complexities compared to text-only models. We evaluated state-of-the-art models like GPT-4o (speech + text) and the Llama family (both base and fine-tuned on conversations) across tasks such as detecting sarcasm, irony, and condescension. Our results show that "multimodal" models have an edge with basic prompts, but when using advanced techniques like chain-of-thought reasoning, the performance advantage shifts back to unimodal models. This highlights the potential of multimodal learning to bridge gaps in how AI interprets human intent, particularly in context-free deceptive communication. 🔗 Paper Link: https://lnkd.in/gtWV2nQa Our work opens up exciting possibilities for AI systems to better mimic human-like understanding of nuanced communication, paving the way for advancements in sentiment analysis and conversational AI. If you’re curious about how multimodal learning is shaping AI’s ability to understand human communication, we’d love for you to check out our paper and share your thoughts! This was joint work with Benjamin R. (Georgia Institute of Technology), and most of it was done during his internship period last summer at Symbl.ai. #NLP #AI #ML #LLM #Multimodal #GPT #sarcasm #sentimentanalysis #

Last Friday, we convened the conversation healthcare needs to have about AI and emerged with a roadmap. AI systems now listen to and watch patient encounters in real time: capturing audio to generate documentation, analyzing video of patient gait and facial expressions, integrating sensor data with electronic health records. This is multimodal AI, and it’s spreading faster than any innovation in modern healthcare. Two-thirds of U.S. clinicians now use AI. Investment doubled in 2025. The question: will we lead this transformation thoughtfully or repeat past failures? We brought together 40+ experts at Columbia for “Multimodal AI at the Crossroads.” Martin Michalowski, PhD, FAMIA, FIAHSI (Minnesota), Laura-Maria Peltonen (Turku), Lisiane Pruinelli, PhD, MS, RN, FAMIA, FAAN (Florida), Incheol Jeong (Hallym/Mount Sinai), Robert Klitzman (bioethics), Meghan Reading Turchioe, PhD, MPH, RN, FAHA (policy), Stephen A. Ferrara, DNP, FAAN, and @Jin Dong (business operations). Four urgent risks emerged: 1. Deploying without evidence. Ambient AI is in millions of encounters but saves only 43 seconds per visit. We’re investing billions on unverified promises. 2. Creating new burdens. Clinicians “perform for the technology” rather than connecting with patients. The “narration burden.” We did this with EHRs. 3. Can’t verify outputs. When AI integrates video, audio, sensors, and EHR data, how does a clinician independently check it? Skills atrophy when AI assistance is removed. 4. Infrastructure too slow. AI competencies identified as essential in 2019 still aren’t integrated while vendors race ahead. What gives me hope: These experts know how to address these risks. We have consensus on what “good” looks like and who owns the solutions. Next: translating insights into policy guidance and implementation roadmaps. Grateful to Zhihong Zhang and especially Pallavi Gupta, PhD who ran the even flawlessly end to end with great oversight + co-PI Jin Dong, the NAIL Collaborative, and Columbia Interdisciplinary Seed Grants. The crossroads is now. Healthcare gets to choose whether we lead AI’s advancement thoughtfully or get dragged along by it.