Vision Capabilities of AI Models

Exciting breakthrough in Vision-Language Models! Researchers from Tsinghua University and Shanghai AI Laboratory have introduced HoVLE, a groundbreaking monolithic vision-language model that revolutionizes how AI processes images and text. >> Technical Innovation HoVLE introduces a holistic embedding module that unifies visual and textual inputs into a shared space, allowing Large Language Models to interpret images as naturally as text. The model employs 8 causal Transformer layers with 2048 hidden dimensions and 16 attention heads, matching the architecture of its LLM backbone. >> Under the Hood The system processes images through dynamic high-resolution tiling at 448x448 resolution, combined with a global thumbnail for context. The training involves a sophisticated three-stage process: - Distillation stage using 500M random images and text tokens - Alignment stage with 45M multi-modal data - Instruction tuning with 5M specialized samples. >> Performance Highlights HoVLE significantly outperforms previous monolithic models, achieving ~15 points improvement on MMBench. It demonstrates competitive results with leading compositional models across 17 benchmarks while maintaining a simpler, more efficient architecture. >> Industry Impact This advancement marks a significant step toward more efficient and capable AI systems that can seamlessly understand both visual and textual information. The model's ability to maintain high performance while simplifying architecture opens new possibilities for practical applications. A remarkable achievement that pushes the boundaries of AI's multimodal understanding capabilities. The future of vision-language models looks promising!

Computer Vision (CV) algorithms are the "eyes" of AI. They allow machines to not just capture pixels, but to understand 𝐨𝐛𝐣𝐞𝐜𝐭𝐬, 𝐩𝐚𝐭𝐭𝐞𝐫𝐧𝐬, 𝐚𝐧𝐝 𝐟𝐞𝐚𝐭𝐮𝐫𝐞𝐬. From autonomous driving to medical imaging, choosing the right algorithm is a balance of 𝐬𝐩𝐞𝐞𝐝, 𝐚𝐜𝐜𝐮𝐫𝐚𝐜𝐲, 𝐚𝐧𝐝 𝐡𝐚𝐫𝐝𝐰𝐚𝐫𝐞 constraints. 𝟏. 𝐎𝐁𝐉𝐄𝐂𝐓 𝐃𝐄𝐓𝐄𝐂𝐓𝐈𝐎𝐍 (𝐑𝐞𝐚𝐥-𝐓𝐢𝐦𝐞 𝐯𝐬. 𝐇𝐢𝐠𝐡 𝐏𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧) 𝐘𝐎𝐋𝐎 (𝐘𝐨𝐮 𝐎𝐧𝐥𝐲 𝐋𝐨𝐨𝐤 𝐎𝐧𝐜𝐞): The industry standard for 𝐬𝐩𝐞𝐞𝐝. It processes the entire image in a single pass, making it ideal for real-time video feeds (e.g., security cameras, self-driving cars). 𝐑-𝐂𝐍𝐍 / 𝐅𝐚𝐬𝐭𝐞𝐫 𝐑-𝐂𝐍𝐍: Focuses on 𝐚𝐜𝐜𝐮𝐫𝐚𝐜𝐲. It uses region proposals to find objects, which is slower but much more precise for complex scenes. 𝟐. 𝐅𝐄𝐀𝐓𝐔𝐑𝐄 𝐌𝐀𝐓𝐂𝐇𝐈𝐍𝐆 & 𝐄𝐃𝐆𝐄 𝐃𝐄𝐓𝐄𝐂𝐓𝐈𝐎𝐍 Before deep learning, we relied on mathematical feature extractors. These are still vital for low-power devices: 𝐎𝐑𝐁 (𝐎𝐫𝐢𝐞𝐧𝐭𝐞𝐝 𝐅𝐀𝐒𝐓 𝐚𝐧𝐝 𝐑𝐨𝐭𝐚𝐭𝐞𝐝 𝐁𝐑𝐈𝐄𝐅): A fast, open-source alternative to SIFT/SURF. It identifies key points in an image to match them across different frames. 𝐂𝐚𝐧𝐧𝐲 𝐄𝐝𝐠𝐞 𝐃𝐞𝐭𝐞𝐜𝐭𝐨𝐫: A multi-stage algorithm used to detect a wide range of edges in images, providing the structural skeleton of an object. 𝟑. 𝐒𝐄𝐆𝐌𝐄𝐍𝐓𝐀𝐓𝐈𝐎𝐍 (𝐏𝐢𝐱𝐞𝐥-𝐋𝐞𝐯𝐞𝐥 𝐔𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠) 𝐒𝐞𝐦𝐚𝐧𝐭𝐢𝐜 𝐒𝐞𝐠𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧: Labels every pixel in an image with a category (e.g., "Road," "Sky," "Pedestrian"). 𝐈𝐧𝐬𝐭𝐚𝐧𝐜𝐞 𝐒𝐞𝐠𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 (𝐞.𝐠., 𝐌𝐚𝐬𝐤 𝐑-𝐂𝐍𝐍): Goes a step further by distinguishing between individual objects of the same class (e.g., identifying Person 1 vs. Person 2). 𝟒. 𝐓𝐇𝐄 𝐍𝐄𝐖 𝐅𝐑𝐎𝐍𝐓𝐈𝐄𝐑: 𝐕𝐈𝐒𝐈𝐎𝐍 𝐓𝐑𝐀𝐍𝐒𝐅𝐎𝐑𝐌𝐄𝐑𝐒 (𝐕𝐢𝐓) 𝐕𝐢𝐬𝐢𝐨𝐧 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐞𝐫𝐬: Unlike traditional CNNs that look at local pixel neighborhoods, ViTs split images into patches and use 𝐒𝐞𝐥𝐟-𝐀𝐭𝐭𝐞𝐧𝐭𝐢𝐨𝐧 to capture global context. 𝐔𝐬𝐞 𝐂𝐚𝐬𝐞: Handling highly complex patterns where the relationship between distant parts of an image is crucial. 💡 𝐒𝐓𝐑𝐀𝐓𝐄𝐆𝐈𝐂 𝐓𝐑𝐀𝐃𝐄-𝐎𝐅𝐅𝐒 𝐋𝐢𝐦𝐢𝐭𝐞𝐝 𝐇𝐚𝐫𝐝𝐰𝐚𝐫𝐞? → Use 𝐎𝐑𝐁 or 𝐌𝐨𝐛𝐢𝐥𝐞𝐍𝐞𝐭 (Lightweight CNN). 𝐍𝐞𝐞𝐝 𝐌𝐢𝐥𝐢-𝐬𝐞𝐜𝐨𝐧𝐝 𝐋𝐚𝐭𝐞𝐧𝐜𝐲? → Use 𝐘𝐎𝐋𝐎. 𝐃𝐞𝐞𝐩 𝐂𝐨𝐧𝐭𝐞𝐱𝐭𝐮𝐚𝐥 𝐔𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠? → Use 𝐕𝐢𝐬𝐢𝐨𝐧 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐞𝐫𝐬. 🔥 𝐓𝐇𝐄 𝐁𝐎𝐓𝐓𝐎𝐌 𝐋𝐈𝐍𝐄: A great model is nothing without great data. In 2026, the focus has shifted from just "tuning algorithms" to 𝐝𝐚𝐭𝐚-𝐜𝐞𝐧𝐭𝐫𝐢𝐜 𝐀𝐈. Experimenting with data augmentation, annotation quality, and batch composition is often more effective than simply switching architectures. #𝐂𝐨𝐦𝐩𝐮𝐭𝐞𝐫𝐕𝐢𝐬𝐢𝐨𝐧 #𝐀𝐈 #𝐌𝐚𝐜𝐡𝐢𝐧𝐞𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 #𝐘𝐎𝐋𝐎 #𝐕𝐢𝐬𝐢𝐨𝐧𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐞𝐫

📢 Breaking: Alibaba just flipped the script on “open AI” Over the weekend, Alibaba released Qwen3-VL-30B-A3B-Instruct and Qwen3-VL-30B-A3B-Thinking — two compact multimodal models built on a 30B-parameter backbone, yet only 3B active parameters per inference. They match or surpass GPT-5 and Claude 4 Sonnet on math, vision + language, reasoning, video, and even agentic tasks. Here’s what really makes this launch epic and what’s really new: 1️⃣ Frontier performance is now open Until now, open-source models (like LLaMA or Mistral) were powerful but behind proprietary ones. Qwen3-VL models deliver the same performance level as GPT-5 and Claude 4 Sonnet, but are free to use, adapt, and commercialize under an Apache 2.0 license. 2️⃣ Efficiency becomes the new scale Each model contains 30B parameters, but only 3B are used per query, thanks to its Mixture-of-Experts (MoE) design. This means it behaves like a large model in intelligence but like a smaller one in cost and speed. 3️⃣ Reliability is built in Alibaba released two versions: Instruct (for speed and clarity) & Thinking (for step-by-step reasoning). The model switches between quick responses and deep reasoning, making reliability and factual accuracy designed-in. 4️⃣ Multimodality finally reaches full capability Qwen3-VL can processes text, images, and videos, and can turn visuals into working outputs like diagrams (Draw.io) or web code (HTML, CSS, JS). 5️⃣ Memory expands from short-term to continuous With a 256K token context window (expandable to 1M), Qwen3-VL can analyze full books, multi-hour meetings, or entire videos without splitting them into chunks. 6️⃣ AI can use software, not just talk about it Qwen3-VL can interpret a computer screen and act on it: clicking, typing, navigating, executing. It doesn’t need an API to perform a task; it can visually operate digital interfaces like a human, opening the door to “visual agents”. 7️⃣ New AI economics of AI A free, efficient model that performs like a paid, proprietary one changes the cost baseline. Enterprises can now self-host GPT-5-level intelligence without usage fees or vendor lock-in. 8️⃣ Openness becomes China’s credibility strategy By releasing these models openly, Alibaba is exporting transparency and trust rather than control. This marks a strategic shift in how China positions itself in the global AI ecosystem. ✳️ Alibaba didn’t just release two new models. It redrew the boundaries of what’s considered “frontier,” who can access it, and how much it costs to use. AI capability is becoming a global public good, reshaping margins, strategies, and alliances. #AI #GenAI #OpenSource #ArtificialIntelligence #Innovation

How can AI models learn to perceive the richness of human appearance and behavior without relying on massive, privacy-sensitive datasets of real people? Meet Fatemeh Saleh, a research scientist at Microsoft in Cambridge, whose work explores how to build perceptual AI systems without relying on real‑world human data. Much of Fatemeh’s research focuses on replacing traditional training pipelines with synthetic ones. In recent work, she and her collaborators showed that vision models trained entirely on synthetic data can achieve state‑of‑the‑art results using significantly less data and compute than standard approaches (https://lnkd.in/gJVT-aY6). Earlier work reflects the same goal of capturing faces, bodies, and hands without markers, calibration, or specialized hardware (https://lnkd.in/g7WcVMsH). She’s also shown that synthetic training signals can support fine‑grained facial modeling, including eyelid folds and gaze dynamics (https://lnkd.in/gvKMUxEE). Across these projects is a consistent technical objective: doing more with less supervision. As a PhD student, Fatemeh developed approaches for semantic segmentation that required only image‑level labels, inferring object locations from the network’s own internal features (https://lnkd.in/gZxYBbwN). She later extended these ideas to video, enabling models to separate objects in motion with minimal manual annotation (https://lnkd.in/gTsYGdxi). These methods now inform user‑facing systems such as AI Avatars in Microsoft Teams, where models trained on synthetic data help create digital representations without requiring large datasets of real people (https://lnkd.in/gW_hhDmB). If you're not yet following Fatemeh’s research, I highly recommend checking it out! #AIInnovators #AppliedResearch #SyntheticData #LeadingLikeAScientist

🛰️ Imagine an AI that can read the Earth's story from space – pixel by pixel, month by month. Remote sensing AI models vary widely in their capabilities. Gabriel Tseng et al. developed Galileo, a multimodal model designed to more comprehensively analyze Earth observation data. The model integrates multiple data sources: - Multispectral imagery from Sentinel-2 - Synthetic aperture radar (SAR) from Sentinel-1 - Elevation and land cover maps - Time-varying weather data - Static geospatial coordinates Key technical features: - Adapted Vision Transformer (ViT) architecture - Processes 24 monthly time steps - Analyzes 96 × 96 pixel images at 10m resolution - Uses self-supervised learning to capture global and local features In validation across multiple datasets, Galileo's three model variants performed consistently well. Ablation studies provided insights into the model's most critical characteristics. The approach offers a more comprehensive method of analyzing satellite and geospatial data. https://lnkd.in/eqKj5_Dw #RemoteSensing #EarthObservation #AI #Geospatial __________________ Enjoyed this post? Like 👍, comment 💬, or re-post 🔄 to share with others. Click "View my newsletter" under my name ⬆️ to join 1500+ readers.

Researchers at Caltech just introduced Conversational Image Segmentation. It's a vision-language model that grounds intent and physical reasoning into pixel-accurate masks, not just object labels. Ask a standard segmentation model "where can I safely store the knife?" or "which suitcase won't destabilize the stack?" and it's lost. These questions require reasoning about physics, context, and intent, not object recognition. Aadarsh Sahoo and Georgia Gkioxari's benchmark spans five reasoning types: 👉entities 👉spatial relations 👉events 👉affordances 👉physical safety Their model, CONVERSEG-NET, outperforms much larger competitors on all of them. The 3B model beats LISA-13B by +2.8% and Seg-Zero 7B by +1.6% on the CONVERSEG benchmark. To scale training data, they built an automated pipeline that generates and verifies 106K prompt-mask pairs using VLMs. But human annotators still had the final say on benchmark quality 🗣️ Humans reviewed and accepted or rejected each AI-generated example. Scale through automation, quality through human judgment. Most annotation pipelines are still built around object labels. This research suggests that's no longer enough because real-world queries are about intent and consequence, not categories. If your models need to understand intent, not just objects, how far does your current training data actually get you? #MLResearch #ImageSegmentation #VisionLanguageModels

Last week Microsoft's Azure AI team dropped the paper for Florence-2: the new version of the foundation computer vision model. This is significant advancement in computer vision and is a significant step up from the original Florence model. 📥 Dataset: Florence-2 has the ability to interpret and understand images comprehensively. Where the original Florence excelled in specific tasks, Florence-2 is adept at multitasking. It's been trained on an extensive FLD-5B dataset encompassing a total of 5.4B comprehensive annotations across 126M images, enhancing its ability to handle a diverse range of visual task such as object detection, image captioning, and semantic segmentation with increased depth and versatility. 📊 Multi-Task Capability: Florence-2's multitasking efficiency is powered by a unified, prompt-based representation. This means it can perform various vision tasks using simple text prompts, a shift from the original Florence model's more task-specific approach. 🤖 Vision and Language Integration: Similar to GPT-4's Vision model, Florence-2 integrates vision and language processing. This integration is facilitated by its sequence-to-sequence architecture, similar to models used in natural language processing but adapted for visual content. 👁️ Practical Applications: Florence-2's capabilities can enhance autonomous vehicle systems' environmental understanding, aid in medical imaging for more accurate diagnoses, surveillance, etc. Its ability to process and understand visual data on a granular level opens up new avenues in AI-driven analysis and automation. Florence-2 offers a glimpse into the future of visual data processing. Its approach to handling diverse visual tasks and the integration of large-scale data sets for training sets it apart as a significant development in computer vision. Paper: https://lnkd.in/deUQf9NG Researchers: Ce Liu, Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Lu Yuan #Microsoft #AzureAI #Florence #computervision #foundationmodels

🔍 Vision-language models' reasoning over scientific charts maybe over estimated. Even state-of-the-art models like GPT-4V, 4o, and the recent O3 still show signs of taking shortcuts. The shortcuts aren’t about simple tasks like “is this a dog?”—those are largely solved. But for deeper scientific reasoning—questions like “how much taller is the blue bar than the red one?”—models often rely on text labels, not the visual elements themselves. 🧪 When we remove the text, performance drops. These models aren’t really seeing the chart—they’re reading it. (O3's drop is the most minor though) In our recent manuscript, we highlight this issue and propose a new approach. Instead of using flat image formats like PNGs, we move to structured representations like SVGs—where the model can reason over chart objects and relationships, not just pixels. SVGs remove the shortcuts and better reflect how scientists interpret data visually. 📄 Check out our paper here: https://lnkd.in/gsj9nkuG Excited to hear your thoughts and connect with others working at the intersection of AI, vision-language reasoning, and scientific understanding. #AI #VisionLanguageModels #ScientificReasoning #LLMs #DataVisualization #SVG #MachineLearning #DeepLearning #ChartUnderstanding https://lnkd.in/gsj9nkuG

MoAI Mixture of All Intelligence for Large Language and Vision Models The rise of large language models (LLMs) and instruction tuning has led to the current trend of instruction-tuned large language and vision models (LLVMs). This trend involves either meticulously curating numerous instruction tuning datasets tailored to specific objectives or enlarging LLVMs to manage vast amounts of vision language (VL) data. However, current LLVMs have disregarded the detailed and comprehensive real-world scene understanding available from specialized computer vision (CV) models in visual perception tasks such as segmentation, detection, scene graph generation (SGG), and optical character recognition (OCR). Instead, the existing LLVMs rely mainly on the large capacity and emergent capabilities of their LLM backbones. Therefore, we present a new LLVM, Mixture of All Intelligence (MoAI), which leverages auxiliary visual information obtained from the outputs of external segmentation, detection, SGG, and OCR models. MoAI operates through two newly introduced modules: MoAI-Compressor and MoAI-Mixer. After verbalizing the outputs of the external CV models, the MoAI-Compressor aligns and condenses them to efficiently use relevant auxiliary visual information for VL tasks. MoAI-Mixer then blends three types of intelligence (1) visual features, (2) auxiliary features from the external CV models, and (3) language features by utilizing the concept of Mixture of Experts. Through this integration, MoAI significantly outperforms both open-source and closed-source LLVMs in numerous zero-shot VL tasks, particularly those related to real-world scene understanding such as object existence, positions, relations, and OCR without enlarging the model size or curating extra visual instruction tuning datasets.

Happy Friday! This week in #learnwithmz, let’s talk about how AI “sees” the world through Vision Language Models (VLMs). We often treat AI as text-only, but modern models like Gemini, DeepSeek-VL and GPT-4o, etc. blend vision and language, allowing them to describe, reason about, and even “imagine” what they see. An excellent article by Frederik Vom Lehn mapped out how information flows inside a VLM, from raw pixels all the way to text predictions. What’s going on inside a VLM? - Early layers detect colors and simple patterns. - Middle layers respond to shapes, edges, and structures. - Later layers align visual regions with linguistic concepts: like “dog,” “street,” or “sky.” - Vision tokens have large L2 norms, which makes them less sensitive to spatial order (a “bag-of-visual-features” effect). - The attention mechanism favors text tokens, suggesting that language often dominates reasoning. - You can even use softmax probabilities to segment images or detect hallucinations in multimodal outputs. Why it Matters? Understanding how VLMs allocate attention helps explain why they sometimes hallucinate objects or struggle with spatial reasoning. PMs & Builders If you’re working with multimodal AI, think copilots, chat with images, or agentic vision, invest time in visual explainability. It’s understanding how AI perceives. Read the full visualization breakdown here: https://lnkd.in/gc2pZnt2 #AI #VisionLanguageModels #LLMs #ProductManagement #learnwithmz #DeepLearning #MultimodalAI