Adaptive Search Filters

Weaviate just released a new filtered search method based on the popular ACORN paper that makes certain queries up to 10x faster 🤯 𝗧𝗵𝗲 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲: Picture searching for a "gold ring" with a price filter under $10. Seems simple, right? But there's a technical challenge here: • The query ("gold ring") and filter (price < $10) have 𝘭𝘰𝘸 𝘤𝘰𝘳𝘳𝘦𝘭𝘢𝘵𝘪𝘰𝘯 • Gold rings are typically expensive, so most similar items won't pass the filter • This creates inefficiencies in traditional search approaches 𝗣𝗿𝗲-𝗔𝗖𝗢𝗥𝗡 𝗦𝗼𝗹𝘂𝘁𝗶𝗼𝗻𝘀 𝗛𝗮𝗱 𝗧𝗿𝗮𝗱𝗲-𝗼𝗳𝗳𝘀: 1️⃣ Pre-filtering: Apply filters first, then search • Great for small result sets • Becomes slow with larger filtered datasets, because you need to brute force 2️⃣ Post-filtering: Search first, then filter • Challenging to determine how many results to fetch • Can waste resources or miss relevant results 𝗘𝗻𝘁𝗲𝗿 𝗔𝗖𝗢𝗥𝗡! 🌳 Original paper: https://lnkd.in/eJhPkeTK Acorn uses a two-hop based expansion of the neighborhood, evaluating nodes that are two hops away rather than one. This speeds up graph traversal and helps maintain connectivity even when filtered nodes are sparse. Weaviate took the concepts from the paper’s implementation, but made a few key changes: • Only uses two-hop expansion when needed • Keep regular HNSW graph, no reindexing • Additional entry points to better handles cases where filters and queries have low correlation Practical Benefits: • No reindexing required for existing users • Up to 10x performance improvement in challenging scenarios • Maintains good performance across different use cases 𝗧𝗵𝗲 𝗥𝗲𝘀𝘂𝗹𝘁𝘀? 📊 • With low correlation queries: ACORN significantly outperforms traditional methods • With high selectivity filters: Automatically switches back to optimize performance • Especially shines when filters are restrictive and have low correlation with queries Learn more in this blog post: https://lnkd.in/ejUFU3uv Or try it out with Weaviate 1.27: https://lnkd.in/e2EHCEty

The hardest problem in vector search isn't finding similar documents - it's finding similar documents 𝘁𝗵𝗮𝘁 𝗺𝗮𝘁𝗰𝗵 𝘆𝗼𝘂𝗿 𝗳𝗶𝗹𝘁𝗲𝗿𝘀. Imagine you're building a legal document search system. A lawyer queries for "contract breach cases" but also needs results filtered by jurisdiction, date range, and case outcome. Or consider a financial analyst searching for "market volatility analysis" filtered by specific sectors, time periods, and regulatory frameworks. This is where things get tricky. The challenge isn't just finding semantically similar documents, it's finding similar documents 𝘁𝗵𝗮𝘁 𝗮𝗹𝘀𝗼 𝗺𝗮𝘁𝗰𝗵 𝘀𝗽𝗲𝗰𝗶𝗳𝗶𝗰 𝗰𝗿𝗶𝘁𝗲𝗿𝗶𝗮. And the technical complexity here is real. 𝗧𝗵𝗲 𝗰𝗼𝗿𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺: Do you filter first, then search? Or search first, then filter? • Filter-first (pre-filtering): Fast when filters are restrictive, but can become slow as the filtered dataset grows • Search-first (post-filtering): Uses the HNSW index efficiently, but struggles to predict how many results to fetch before filtering Here's where it gets interesting: 𝗰𝗼𝗿𝗿𝗲𝗹𝗮𝘁𝗶𝗼𝗻 𝗺𝗮𝘁𝘁𝗲𝗿𝘀. In that legal search example, if you're looking for recent contract cases in California, the filter (California + recent) likely correlates well with your query. Most results will pass the filter. But what if you're searching for "corporate merger precedents" filtered by cases from 1950-1960? Low correlation. The search might explore thousands of modern cases before finding relevant historical ones. Performance tanks. 𝗪𝗵𝘆 𝘄𝗲 𝗯𝘂𝗶𝗹𝘁 𝗔𝗖𝗢𝗥𝗡 𝗶𝗻𝘁𝗼 𝗪𝗲𝗮𝘃𝗶𝗮𝘁𝗲 ACORN (ANN Constraint-Optimized Retrieval Network) solves this elegantly using two-hop graph expansion. Instead of traversing the HNSW graph one node at a time, it can "look ahead" two hops when needed, maintaining connectivity even when filters are restrictive. The results speak for themselves: • Up to 𝟭𝟬𝘅 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗶𝗺𝗽𝗿𝗼𝘃𝗲𝗺𝗲𝗻𝘁 in low-correlation scenarios • No reindexing required - works with existing data • Adaptively switches between normal HNSW and two-hop expansion based on filter density In our tests with 20% selectivity filters (where only 20% of objects pass), ACORN maintained strong throughput while the previous approach slowed to near brute-force speeds. 𝗪𝗵𝗮𝘁 𝘁𝗵𝗶𝘀 𝗺𝗲𝗮𝗻𝘀 𝗳𝗼𝗿 𝘆𝗼𝘂: Whether you're building legal research tools, financial analysis platforms, or any system requiring filtered vector search - you don't need to worry about filter correlation anymore. ACORN handles it automatically. By default in Weaviate 1.35+, Available since Weaviate 1.27+, and it just works.

Today we’re releasing our biggest product yet… The first people search API that lets you nest and combine filters in virtually unlimited ways, so you can actually search the way you think. Here are some examples of what you could search for with the new API: - Find founders of fintech companies that were founded in or after 2024 who were software engineers at Brex, Ramp, PayPal, Revolut, or Stripe for at least two years. - Identify senior professionals with more than 10 years of experience who left Google, Meta, Amazon, Microsoft, or Apple and joined growth-stage companies (50-100 employees) in or after 2023. You could have done all of this with our previous APIs (or any other b2b data API) but you’d have to run multiple calls with different filters and use a combination of company and people enrichment APIs to get accurate results. That works if you’re happy exporting long lists and cleaning them manually. But if you’re building real workflows and products, your needs are probably different: - You want your first query to be the only query you run. - You need one clean list, not multiple lists and the headache of filtering out what you actually need. - You want your search logic to match the way you think, not the way the API limits you. - You want confidence that the search actually found everyone you care about. So we spent the last few months listening to customer pains and building a search that finds only the people you want. Here’s what’s different: 1. Increased the number of search filters 3x from 20 to 60: More filters including specific job titles at present or past companies, skills, specific start and end dates in present or past companies and more. 2. Real logic with nesting: Added AND/OR logic and nesting, so you can combine filters in any way to zero in on exactly the person you’re looking for. 3. Stronger operators: Use in / not_in to include or exclude sets, and greater than, less than or equal to operators on numeric & date fields (experience, headcount, start/end dates, etc). 4. Lower latency: The response time is down to less than 10 seconds, which is 90% less than our current Search API. 5. ~30× cheaper: Runs on our own database which is updated monthly making it significantly cheaper than our current Search API. This is one of the launches I’m most excited about because it is the first truly precise person search API for B2B in the market. If you want access to the API, drop a comment or DM me.