Syncing Embeddings in PostgreSQL with pgedge-vectorizer

Hey LinkedIn fam today we're diving into Vibhor Kumar's brilliant piece on the *real* shift in data platforms. Spoiler alert: It's not just AI jazz—it's all about doing less (seams that is). Data world we’re in a pickle. We’ve been running around with a patchwork quilt of systems it’s like we’re living in a database version of Frankenstein. AI might be stealing the spotlight but the real magic trick? Making things simpler. Kumar lays it all out: fewer pipelines less data shuffling and none of that "guess where the truth lives" game. PostgreSQL enthusiasts this is your time to shine. Your trusted pal Postgres is more than just an OLTP champ. It’s a platform hero ready to streamline data chaos into something even your mom could brag about over coffee. So my fellow database wranglers let's simplify Let's make our architectures seam-aware not just AI-ready. Credit to Vibhor Kumar for this wisdom and... reminding us all we had the power to be 'seamless' all along. Keep it nerdy folks! #Databases #PostgreSQL #SeamlessAndFlawless #AIshenanigans

You probably don't need a separate vector database. If you're already running #PostgreSQL, you can add AI-powered semantic search with one extension: pgvector. Here's how it works in 5 steps: - Enable the extension → CREATE EXTENSION vector; - Add a VECTOR column to your existing tables - Store embeddings from any ML model (OpenAI, Hugging Face, etc.) - Query by meaning using similarity operators in plain SQL - Add an HNSW index for fast nearest-neighbor search at scale No new infrastructure. No sync jobs. No new vendor. Just SQL and vectors. The best part? You can combine vector search with everything PostgreSQL already does joins, filters, transactions, full-text search in a single query. Use cases teams are building right now: → Smart product search that understands intent, not just keywords → FAQ chatbots that match questions by meaning → Content recommendation engines Vibhor Kumar and Marc Linster wrote a great step-by-step walkthrough covering all of this from setup to production use cases. Full article here - https://lnkd.in/gBKF2APe Follow our Substack page for more such how to tutorials straight to your inbox - Data Engineering Byte

Really excited to finally share something we’ve been working on - our new book is officially out now 📘 ✨   “AI-Ready PostgreSQL 18: Building Intelligent Data Systems with Transactions, Analytics, and Vectors.” by Vibhor Kumar and Marc Linster   If you’ve been building data-heavy or AI-enabled applications lately, you’ve probably felt this shift; everything is converging. Transactions, analytics, AI… they’re no longer separate concerns anymore 🔄   That’s exactly what this book explores.   It’s a practical guide to using PostgreSQL 18 as a unified platform not just for storing data, but for building intelligent systems that can handle real-time decisions, analytics, and even AI workloads like vector search and LLM integrations 🤖 📊   Inside, we’ve focused on: Real examples you can actually apply 🛠️ Clear guidance on architecture and trade-offs 🧠 Ways to get the most out of PostgreSQL 18’s latest features ⚡ When to stick with Postgres and when not to 🎯   Whether you’re a developer, data engineer, or architect, the goal is simple: help you build smarter systems without unnecessary complexity 🚀   📣 If this sounds relevant to your work, check it out and grab your copy here: https://packt.link/gQKTu   Curious to hear your take 👉 Are you already using PostgreSQL for AI use cases? 👉 Do you think one system can realistically handle OLTP, OLAP, and AI together?   Would love to hear what you’re seeing in your own work 👇 💬   #postgresql #aiengineering #dataarchitecture #realtimeanalytics #vectorsearch #techleadership #dataengineering #dataplatforms  

“𝗪𝗵𝗮𝘁 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗵𝗮𝗽𝗽𝗲𝗻𝘀 𝘄𝗵𝗲𝗻 𝘆𝗼𝘂 𝗶𝗻𝗱𝗲𝘅 𝗮 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁 𝗶𝗻 𝗘𝗹𝗮𝘀𝘁𝗶𝗰𝘀𝗲𝗮𝗿𝗰𝗵?” You send a simple JSON. Elasticsearch doesn’t store it. It transforms it completely. What we think “I send a document… Elasticsearch stores it and makes it searchable.” Simple pipeline, right? 𝗪𝗵𝗮𝘁 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗵𝗮𝗽𝗽𝗲𝗻𝘀 Under the hood, Elasticsearch passes your data through an analysis pipeline powered by Apache Lucene. That pipeline has three critical steps: 𝟭. 𝗧𝗲𝘅𝘁 𝗶𝘀 𝗯𝗿𝗼𝗸𝗲𝗻 𝗶𝗻𝘁𝗼 𝘁𝗼𝗸𝗲𝗻𝘀 {  "𝘵𝘪𝘵𝘭𝘦": "𝘓𝘦𝘢𝘳𝘯𝘪𝘯𝘨 𝘌𝘭𝘢𝘴𝘵𝘪𝘤𝘴𝘦𝘢𝘳𝘤𝘩 𝘐𝘯𝘵𝘦𝘳𝘯𝘢𝘭𝘴" } Becomes: ["𝘭𝘦𝘢𝘳𝘯𝘪𝘯𝘨", "𝘦𝘭𝘢𝘴𝘵𝘪𝘤𝘴𝘦𝘢𝘳𝘤𝘩", "𝘪𝘯𝘵𝘦𝘳𝘯𝘢𝘭𝘴"] This is done by a tokenizer. 𝟮. 𝗧𝗼𝗸𝗲𝗻𝘀 𝗮𝗿𝗲 𝗻𝗼𝗿𝗺𝗮𝗹𝗶𝘇𝗲𝗱 Depending on configuration: • Lowercased • Stop words removed ("the", "is") • Stemmed ("learning" → "learn") So now: ["learn", "elasticsearch", "internal"] 𝟯. 𝗧𝗼𝗸𝗲𝗻𝘀 𝗮𝗿𝗲 𝗶𝗻𝗱𝗲𝘅𝗲𝗱 (𝗻𝗼𝘁 𝘁𝗵𝗲 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁) Instead of storing your JSON directly for search… Elasticsearch builds an inverted index: "𝘭𝘦𝘢𝘳𝘯" → [𝘥𝘰𝘤1] "𝘦𝘭𝘢𝘴𝘵𝘪𝘤𝘴𝘦𝘢𝘳𝘤𝘩" → [𝘥𝘰𝘤1] "𝘪𝘯𝘵𝘦𝘳𝘯𝘢𝘭" → [𝘥𝘰𝘤1] 𝗬𝗼𝘂𝗿 𝗼𝗿𝗶𝗴𝗶𝗻𝗮𝗹 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁? Still stored—but not used for search. Here’s the twist Search doesn’t happen on your data. It happens on the processed version of your data. Which means: Searching "learn" matches "learning" Searching "run" might match "running" Searching "The Elasticsearch" ignores "The" Why this matters in real systems This pipeline quietly decides: ▸ What results users see ▸ What results are missed ▸ Why “exact match” sometimes fails And once data is indexed… Changing this behavior means reindexing everything. 𝗧𝗵𝗲 𝗱𝗲𝘀𝗶𝗴𝗻 𝗶𝗻𝘀𝗶𝗴𝗵𝘁 Elasticsearch is not just storing data. It’s rewriting your data to optimize retrieval. And the way it rewrites it… defines your search quality. “In Elasticsearch, you don’t search what you store. You search what it understood.” #Elasticsearch #SearchEngineering #SystemDesign #DistributedSystems #BackendEngineering #SoftwareArchitecture #ScalableSystems #FullTextSearch #TechDeepDive #EngineeringInsights #JavaDevelopers #Codefarm

"SQL doesn't scale" is usually a confession that you don't know how to write an index. We reach for NoSQL far too early. It’s often used as a shortcut to fix slow queries, but usually, it just masks poor architecture while destroying ACID compliance. Stop sharding your database and start looking at your execution plan. The Reality: - Indexing Power: A properly indexed SQL database handles millions of rows with sub-millisecond lookups. If it’s crawling, you likely have a full table scan, not a scaling limit. - Vertical Scaling: With modern NVMe drives and massive RAM, vertical scaling goes further than ever. Most "performance issues" are actually just inefficient schemas. - The NoSQL Tax: Switching to NoSQL moves complexity to the application. You trade a simple JOIN for manual data stitching and eventual consistency nightmares. My Take: Don’t abandon battle-tested technology because you’re too lazy to run EXPLAIN ANALYZE. High-performance engineering is about mastering the tools you already have. P.S. Are you actually hitting physical limits, or is your schema just a mess? #SoftwareEngineering #Database #SQL #Scalability #Lampstand

Database performance bottlenecks often hide in plain sight, draining resources and slowing down applications. This video explores how MongoDB Query Insights provides the visibility needed to identify, analyze, and resolve execution patterns that impact performance. Watch it now -> https://lnkd.in/g2aXgPNu #mongodb #nosql #sql #genai #ai #database #developer #llm #RAG #architect #vector #search  #mongodbatlas

Most AI data agents don’t fail because of models. They fail because they can’t handle real data. That’s the gap we tried to break. We built Oracle Forge — Oracle Forge is a multi-database AI agent that answers natural language questions across PostgreSQL, MongoDB, SQLite, and DuckDB simultaneously. The hard part isn't query generation. It's context. Enterprise data is fragmented, inconsistently keyed, and full of unstructured fields. Most agents fail silently on exactly that. Our approach: → 3-layer context injection before every query (schema, domain knowledge, self-learned corrections) → LangGraph conductor that decomposes questions into per-database sub-tasks and merges results → Typed failure routing — JoinKeyMismatch, ContractViolation, DialectError — each with its own recovery protocol, not a generic retry → Evaluation harness that traces every tool call and scores against ground truth The biggest lesson: we went from 1.85% → 66.7% on the yelp dataset with one KB update and a model ID fix. No architecture change. No prompt rewrite. Just better context. That's the gap this kind of engineering closes. Built by: Dereje Getie , Eyoel Nebiyu , Chalie Lijalem , Liul Teshome , Nuhamin Alemayehu and Rafia Kedir https://lnkd.in/eBKKD6yM #DataEngineering #AIAgents #ContextEngineering #LLM #BuildingInPublic

AI-Ready PostgreSQL 18 Is Out: Why AI Applications Win or Lose at the Seams Most AI projects do not fail because the model is weak. They fail because data truth, governance, and production reality break at the seams. This post explains why AI-ready systems keep vectors close to SQL truth and includes a minimal demo using PostgreSQL and pgvector....