Word Embedding Models

I've been diving into the latest research from Google's Gemini team, and their new Gemini Embedding model is truly groundbreaking. This state-of-the-art embedding model leverages the power of Google's most capable LLM to produce highly generalizable text representations across numerous languages and textual modalities. What makes Gemini Embedding special is its ability to create dense vector representations that can be precomputed and applied to a variety of downstream tasks including classification, similarity matching, clustering, ranking, and retrieval. The team has achieved remarkable results on the Massive Multilingual Text Embedding Benchmark (MMTEB), substantially outperforming prior state-of-the-art models across multilingual, English, and code benchmarks. >> Technical Deep Dive: The architecture is fascinating - Gemini Embedding is initialized from Gemini LLM parameters and further refined through a two-stage training pipeline: 1. Pre-finetuning: The model is first adapted on a large corpus of potentially noisy query-target pairs using a contrastive learning objective with large batch sizes to stabilize gradients. 2. Finetuning: The model is then fine-tuned on task-specific datasets containing query-target-hard negative triples with smaller batch sizes limited to single datasets. The team employed several innovative techniques: - Mean pooling of token embeddings followed by a linear projection to the target dimension - Noise-contrastive estimation loss with in-batch negatives and masking for classification tasks - Multi-resolution learning to support different embedding dimensions (768, 1536, and 3072) - Model Soup parameter averaging from different fine-tuning runs for enhanced generalization What's particularly impressive is how they used Gemini itself to improve training data quality through synthetic data generation, data filtering, and hard negative mining. Their ablation studies show that task diversity matters more than language diversity for fine-tuning, and the model demonstrates exceptional cross-lingual capabilities even when trained only on English data. The results speak for themselves - Gemini Embedding achieves a task mean score of 68.32 on MTEB(Multilingual), a +5.09 improvement over the second-best model, and shows remarkable performance on cross-lingual retrieval tasks like XTREME-UP with a 64.33 MRR@10 score. Kudos to the Gemini Embedding team led by Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, and Madhuri Shanbhogue for this significant advancement in representation learning!

Think all embeddings work the same way? Think again. Here are 𝘀𝗶𝘅 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝘁𝘆𝗽𝗲𝘀 of embeddings you can use, each with their own strengths and trade-offs: 𝗦𝗽𝗮𝗿𝘀𝗲 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 Think keyword-based representations where most values are zero. Great for exact matching but limited for semantic understanding. 𝗗𝗲𝗻𝘀𝗲 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 The most common type - every dimension has a value. These capture semantic meaning really well, and come in many different lengths. 𝗤𝘂𝗮𝗻𝘁𝗶𝘇𝗲𝗱 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 Compressed versions of dense embeddings that reduce memory usage by using fewer bits per dimension. Perfect when you need to save storage space. 𝗕𝗶𝗻𝗮𝗿𝘆 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 Ultra-compressed embeddings using only 0s and 1s. Super fast for similarity calculations but with reduced accuracy. 𝗩𝗮𝗿𝗶𝗮𝗯𝗹𝗲 𝗗𝗶𝗺𝗲𝗻𝘀𝗶𝗼𝗻𝘀 (𝗠𝗮𝘁𝗿𝘆𝗼𝘀𝗵𝗸𝗮) These embeddings let you use just the first 8, 16, 32, etc. dimensions while still retaining most of the information. This ability comes during model training: earlier dimensions capture more information than later ones. You can truncate a 3072-dimension vector to 512 dimensions and still get great performance. 𝗠𝘂𝗹𝘁𝗶-𝗩𝗲𝗰𝘁𝗼𝗿 (𝗖𝗼𝗹𝗕𝗘𝗥𝗧) Instead of one vector per object, you get many vectors that represent different parts of your object (like tokens for text, patches for images). This enables "late interaction" - comparing individual parts of texts rather than whole documents. Way more nuanced than single-vector approaches. 𝗦𝗼 𝘄𝗵𝗶𝗰𝗵 𝘀𝗵𝗼𝘂𝗹𝗱 𝘆𝗼𝘂 𝗰𝗵𝗼𝗼𝘀𝗲? • Dense for general semantic search. • Matryoshka when you need flexible performance/cost trade-offs. • Multi-vector for precise text matching. • Quantized/Binary when storage and speed matter most. Modern vector databases (like Weaviate 😄) support all of these approaches, so you can experiment and find what works best for your use case. Want code examples or deep dives for any of these? Drop a comment on which one and I’ll send it over 🫡

Embeddings are the backbone of modern AI. And this is the simplest explanation you'll get in less than 60 seconds Every RAG system. Every semantic search. Every recommendation engine. They all start here. But most engineers either: → Oversimplify ("vectors that capture meaning") → Dive straight into linear algebra Here's what you actually need to know: 𝗪𝗵𝗮𝘁 𝗲𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 𝗮𝗿𝗲 ↳ They turn text into numbers. ↳ Similar meanings = similar numbers. ↳ "cat" and "kitten" → close in vector space ↳ "cat" and "refrigerator" → far apart ↳ This is how machines find "related" without exact keyword matching. 𝗛𝗼𝘄 𝘁𝗵𝗲𝘆 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝘄𝗼𝗿𝗸 ↳ Each embedding = a point in high-dimensional space (e.g., 768, 1536 dimensions) ↳ Distance between points = semantic similarity ↳ The embedding model learns these positions from massive text datasets. ↳ Same sentence → same embedding (deterministic) ↳ Different embedding models → different embeddings (incompatible) 𝗪𝗵𝗮𝘁 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 𝗶𝗻 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 ↳ Model size ≠ always better. 384-dim can beat 1536-dim for your domain. ↳ Training data determines strength. General-purpose vs specialized (code, legal, medical). ↳ Speed vs accuracy tradeoff. Local models (cheap) vs API (better, costs $). ↳ Dimensionality = storage + speed. More dimensions = more storage, slower search. ↳ You can't mix models. OpenAI embeddings ≠ Voyage embeddings. Different vector spaces. 𝗛𝗼𝘄 𝘁𝗼 𝗰𝗵𝗼𝗼𝘀𝗲 𝗮𝗻 𝗲𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴 𝗺𝗼𝗱𝗲𝗹 ↳ Start with general-purpose (OpenAI, Voyage, Cohere) ↳ Test on YOUR data (benchmarks lie) ↳ Consider cost: API vs local deployment ↳ Don't over-optimize early; most modern models are "good enough" ↳ Upgrade when: clear retrieval failures, domain mismatch, or cost becomes issue 𝗪𝗵𝗮𝘁 𝗺𝗼𝘀𝘁 𝗽𝗲𝗼𝗽𝗹𝗲 𝗴𝗲𝘁 𝘄𝗿𝗼𝗻𝗴 ❌ "Bigger embeddings = always better" ❌ "Fine-tuning is necessary" ❌ "All embedding models are interchangeable" ❌ "Embeddings capture ALL meaning" Embeddings are tools. Pick one, test it, iterate. Save this. You'll reference it. ♻️ Repost to help your network understand the backbone of AI --- P.S. This is one piece of the RAG puzzle. I'm building a hands-on cohort covering all of it: embeddings, chunking, retrieval, evaluation, and deployment. Details dropping soon. Follow + 🔔

🏎️ Google just launched EmbeddingGemma: an efficient, multilingual 308M embedding model that's ready for semantic search & more on just about any hardware, CPU included. Details: - 308M parameters, 2K token context window, 768-dimensional embeddings - Matryoshka-style dimensionality reduction (512/256/128) - Supports 100+ languages, trained on 320B token multilingual corpus - Quantized model <200MB of RAM, perfect for on-device use - Compatible with Sentence-Transformers, LangChain, LlamaIndex, Haystack, txtai, Transformers.js, ONNX Runtime, and Text-Embeddings-Inference - Gemma3 architecture but bidirectional attention, mean pooling and linear projections - Outperforms any <500M embedding model on Multilingual & English MTEB We're so excited about this model that we wrote all about it, including full inference snippets for 7 frameworks, and show you how to finetune it for your domain for even stronger performance. Read our blogpost here: https://lnkd.in/egpuyTJb I really think this can be a strong step forwards for open-weight multilingual information retrieval, at a size that's actually feasible: I can process 100+ sentences/second locally with just my CPU, and 3500+ on my desktop's GPU.

Embedding models are silently revolutionizing search. I researched for weeks to write a full ebook on choosing and working with embedding models, which I'm releasing for free: https://lnkd.in/gVQY-KNq But if you don't have time to read the whole thing, here's the alpha: Smaller often wins. Especially at scale. Lightweight models can outperform larger ones, reducing latency and resource use. You can often get the same effect by quantizing your larger embedding models. But at 50M+ vectors, you often don't have a choice: go small or go home. Domain-specific embeddings (e.g., Voyage AI's finance & law models) offer huge advantages in specialized fields. Top teams are moving away from closed-source APIs. Open-source models outperform OpenAI's best on MTEB, are much cheaper, and are up to 500x faster at inference time if you deploy them yourself with Text Embedding Inference (TEI). If you are going to choose a closed source model, only choose something you can deploy on a dedicated instance: (some closed source embedding models can be found on Azure/AWS marketplaces) Reranking is transformative. Lightweight embeddings + cross-encoders boost accuracy while maintaining speed, especially if you deploy the cross-encoder/and embedding models yourself using TEI: https://lnkd.in/gzt37Y3M Multimodal models (CLIP, ImageBind) enable unified search across text, images, and audio. Evaluation is crucial. Complex retrieval pipelines can improve performance, but can be fragile if you don't have evaluations on your own data. Vector databases (e.g., KDB.AI) with flat indices accelerate model evaluation, especially for large datasets.

We tested 11 open-source embedding models on 490k documents.   The most downloaded model on HuggingFace came second-to-last. all-MiniLM-L6-v2 has 200M+ downloads. It retrieved the correct document only 28% of the time. Meanwhile, e5-small (5× smaller than some competitors) hit 100% Top-5 accuracy. Why is there such a difference? We didn't measure "semantic similarity."   We measured whether the model actually found the right answer. 3 takeaways if you're building RAG systems: → Popular ≠ good. Benchmark on your own data.   → Bigger ≠ better. e5-small outperformed 500M+ param models.   → Similarity scores lie. A 0.92 similarity score means nothing if the model found the wrong document. For the full research and methodology, search for: Open-source embedding models AIMultiple

#TuesdayPaperThoughts Edition 53: Embedding Theoretical Limits This week's #TuesdayPaperThoughts explores "On the Theoretical Limitations of Embedding-Based Retrieval" from Google DeepMind and The Johns Hopkins University. The paper provides mathematical proof that single-vector embeddings have fundamental representational constraints tied to their dimensionality. Simple BM25 outperforms SOTA embedding models. Key Takeaways: 1️⃣ Mathematical Dimensional Bounds: The authors prove embedding dimension d fundamentally limits top-k document combinations through sign-rank theory. Their free embedding optimization reveals critical corpus sizes: 500k docs (512 dim), 1.7m (768 dim), 4m (1024 dim). No amount of training can overcome this mathematical constraint when relevance matrices exceed the embedding's representational capacity. 2️⃣ LIMIT Dataset Reality Check: Despite trivially simple queries like "who likes apples?", SOTA embedding models score under 20% recall@100 on their constructed dataset. Meanwhile, BM25 achieves near-perfect scores due to its higher effective dimensionality. 3️⃣ Practical Architecture Trade-offs: Multi-vector models like GTE-ModernColBERT significantly outperform single-vector approaches, while cross-encoders like Gemini-2.5-Pro solve the task completely (100%) but remain expensive for first-stage retrieval. Since lexical search is cheaper and faster, running it parallel with vector search improves recall without sacrificing the benefits of embeddings for long-tail queries. Embeddings power modern AI from RAG systems to agentic memory, but this work shows we shouldn't rely solely on single-vector approaches when representational complexity exceeds dimensional limits. Research Credits: Orion Weller, Michael Boratko, Iftekhar Naim, Jinhyuk Lee Paper Link: In comments

🔍 Choosing the Right Embedding Model for Your AI Application Embeddings power AI search, recommendations, and Retrieval-Augmented Generation (RAG). But with so many models, how do you pick the right one? - Need High Accuracy: OpenAI, BERT, RoBERTa are solid, but E5 & Sentence-BERT may perform even better. - Domain-Specific: BioBERT, LegalBERT, and CodeBERT are great—but check if they’re tuned for semantic similarity. - Looking for Speed: MiniLM & TinyBERT are fast, but all-MiniLM-L6-v2 is best for sentence-level tasks. - Need Multilingual Support: LaBSE, mBERT, and paraphrase-multilingual-mpnet work well. - Prefer Self-Hosting: Check out E5, MPNet, GTE, or deploy models via Hugging Face. - Going Cloud: OpenAI and Cohere are great, but Google Vertex AI & AWS Bedrock offer alternatives. Before choosing, balance accuracy, speed, and cost. You can find a preliminary decision tree on which model to choose in my GitHub repo (Link in comments). Open for suggestions. Which model do you prefer?

🌟 Day 28 of My 90-Day AI Learning Journey 🌟 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝗪𝗼𝗿𝗱 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 (𝗪𝗼𝗿𝗱𝟮𝗩𝗲𝗰, 𝗚𝗹𝗼𝗩𝗲) At the heart of modern NLP lies a powerful idea of representing words as numbers that capture meaning. Traditional models saw words as discrete tokens. Word embeddings like 𝗪𝗼𝗿𝗱𝟮𝗩𝗲𝗰 and 𝗚𝗹𝗼𝗩𝗲 map words into continuous vector spaces where 𝘀𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗿𝗲𝗹𝗮𝘁𝗶𝗼𝗻𝘀𝗵𝗶𝗽𝘀 emerge naturally. These models learn from large text corpora like Wikipedia, books, or news articles, capturing context by observing which words co-occur. 𝟭. 𝗪𝗼𝗿𝗱𝟮𝗩𝗲𝗰: 𝗣𝗿𝗲𝗱𝗶𝗰𝘁𝗶𝘃𝗲 𝗮𝗽𝗽𝗿𝗼𝗮𝗰𝗵 Uses two training styles:  • 𝗖𝗕𝗢𝗪 (𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗕𝗮𝗴 𝗼𝗳 𝗪𝗼𝗿𝗱𝘀): predicts a target word from its context words.  → Example: “I ___ coffee every morning.” → predict “drink.”  • 𝗦𝗸𝗶𝗽-𝗚𝗿𝗮𝗺: Predicts context words given a target word.  → Example: given “coffee”, predict “morning”, “drink”, “cup”, etc. This way, the model learns relationships between words based on how they co-occur. 𝟮. 𝗚𝗹𝗼𝗩𝗲 (𝗚𝗹𝗼𝗯𝗮𝗹 𝗩𝗲𝗰𝘁𝗼𝗿𝘀): 𝗖𝗼𝘂𝗻𝘁𝗶𝗻𝗴 𝗮𝗽𝗽𝗿𝗼𝗮𝗰𝗵 It doesn’t predict words. It counts how often words appear together across the entire dataset. It builds a co-occurrence matrix (how many times each word appears near another) and learns word vectors that best represent these global relationships. Word2Vec focuses on local context, GloVe captures global statistics.  → 𝗧𝗵𝗲 𝗩𝗲𝗰𝘁𝗼𝗿 𝗦𝗽𝗮𝗰𝗲 After training, each word becomes a vector - a list of numbers that represent its meaning. In this “vector space,” similar words are close to each other:  • “king”, “queen”, “prince”, “princess” cluster together.  • “car”, “bus”, “train”, “truck” form another cluster. You can even do arithmetic with meaning: vector("king") - vector("man") + vector("woman") ≈ vector("queen")  → 𝗦𝘁𝗮𝘁𝗶𝗰 𝘃𝘀. 𝗖𝗼𝗻𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 The main limitation is that they’re 𝘀𝘁𝗮𝘁𝗶𝗰 - every word has one fixed vector. Example: “bank” in “river bank” → same vector as “bank” in “savings bank.” That’s where 𝗰𝗼𝗻𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝗲𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 (like BERT and GPT) changed the game. They assign different vectors for the same word depending on surrounding words. 𝟭. 𝗕𝗘𝗥𝗧 (𝗕𝗶𝗱𝗶𝗿𝗲𝗰𝘁𝗶𝗼𝗻𝗮𝗹 𝗘𝗻𝗰𝗼𝗱𝗲𝗿 𝗥𝗲𝗽𝗿𝗲𝘀𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻𝘀 𝗳𝗿𝗼𝗺 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿𝘀) Reads texts in both directions during training, using masked language modeling. This allows it to deeply understand context, relationships making it ideal for understanding tasks like classification, question answering, and entity recognition. 𝟮. 𝗚𝗣𝗧 (𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗣𝗿𝗲𝘁𝗿𝗮𝗶𝗻𝗲𝗱 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿) Reads text left to right and learns to predict the next word in a sequence. This is exceptional at generating coherent text powering chatbots, content creation, and code generation. #NLP #MachineLearning #ArtificialIntelligence #DataScience #GenerativeAI #OpenToWork

𝗧𝗵𝗶𝘀 𝗶𝘀 𝗵𝗮𝗻𝗱𝘀-𝗱𝗼𝘄𝗻 𝗼𝗻𝗲 𝗼𝗳 𝘁𝗵𝗲 𝗰𝗹𝗲𝗮𝗻𝗲𝘀𝘁 𝘃𝗶𝘀𝘂𝗮𝗹 𝗲𝘅𝗽𝗹𝗮𝗻𝗮𝘁𝗶𝗼𝗻𝘀 𝗼𝗳 𝗲𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 𝗜’𝘃𝗲 𝘀𝗲𝗲𝗻! ⬇️ This 60-second clip explains a concept so fundamental, it powers almost everything in GenAI today: 𝗪𝗼𝗿𝗱 𝗲𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 𝗟𝗲𝘁'𝘀 𝗯𝗿𝗲𝗮𝗸 𝗶𝘁 𝗱𝗼𝘄𝗻: ⬇️ ➜ AI doesn’t “read” words like we do. Every input — whether it’s a sentence, a word, or a name — is first broken down into smaller pieces called tokens. ➜ Each of these tokens is then mapped to a set of numbers. This numeric representation is called an embedding. ➜ You can think of an embedding as a position in a high-dimensional space — not just a point on a 2D map, but a vector in a space with hundreds of dimensions. In that space, similar meanings end up closer together. ➜ That’s how analogies become math: If the vector for “Hitler” is near “Germany”, and “Mussolini” is close to “Italy”, the model can compute: Hitler + Italy – Germany ≈ Mussolini It’s not because the model knows history — it’s because the geometry of these word vectors reflects the structure of language, culture, and context learned from billions of examples. This is what enables AI to reason with language — not just store it. → “Hitler + Italy – Germany = Mussolini” → “King – Man + Woman = Queen” → “Paris – France + Italy = Rome” 𝗧𝗵𝗶𝘀 𝗶𝘀𝗻’𝘁 𝗺𝗮𝗴𝗶𝗰. 𝗜𝘁’𝘀 𝗺𝗮𝘁𝗵. 𝗔𝗻𝗱 𝗶𝘁’𝘀 𝗲𝘅𝗮𝗰𝘁𝗹𝘆 𝗵𝗼𝘄 𝗟𝗟𝗠𝘀 𝗹𝗶𝗸𝗲 𝗚𝗣𝗧-4, 𝗖𝗹𝗮𝘂𝗱𝗲, 𝗚𝗲𝗺𝗶𝗻𝗶 𝗼𝗿 𝗠𝗶𝘀𝘁𝗿𝗮𝗹 𝗹𝗲𝗮𝗿𝗻 𝗿𝗲𝗹𝗮𝘁𝗶𝗼𝗻𝘀𝗵𝗶𝗽𝘀, 𝗮𝗻𝗮𝗹𝗼𝗴𝗶𝗲𝘀, 𝗮𝗻𝗱 𝗻𝘂𝗮𝗻𝗰𝗲 — 𝗮𝘁 𝘀𝗰𝗮𝗹𝗲. 𝗜𝘁’𝘀 𝘁𝗵𝗲 𝗰𝗼𝗿𝗲 𝗶𝗱𝗲𝗮 𝘁𝗵𝗮𝘁 𝗺𝗮𝗱𝗲 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗺𝗼𝗱𝗲𝗹𝘀 𝘄𝗼𝗿𝗸. 𝗔𝗻𝗱 𝗶𝘁’𝘀 𝘁𝗵𝗲 𝗿𝗲𝗮𝘀𝗼𝗻 𝗔𝗜 𝘁𝗼𝗱𝗮𝘆 𝗱𝗼𝗲𝘀𝗻’𝘁 𝗷𝘂𝘀𝘁 𝗰𝗼𝗽𝘆 𝘁𝗲𝘅𝘁 — 𝗶𝘁 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝘀 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲. This snippet is part of 3Blue1Brown video on transformers: https://lnkd.in/dM7B9FMY (I highly recommend to watch the full video and follow their channel!)