Is Vector Database RAG Outdated? Exploring Vectorless RAG with PageIndex

Most modern RAG (Retrieval-Augmented Generation) systems depend heavily on embeddings and vector databases. If you’ve built anything in the AI space recently, chances are you’ve used chunking + embeddings + similarity search.

But recently I came across something interesting: vectorless RAG.

And it made me question something important:

Do we really need embeddings for every retrieval system?

Let’s break this down properly.

How Traditional RAG Works (And Why It Became Popular)

A typical RAG pipeline looks like this:

1. Split documents into chunks (500–1000 tokens)
2. Generate embeddings for each chunk
3. Store them in a vector database like Pinecone
4. When a query comes in:

• Convert query into embedding
• Run similarity search
• Retrieve top K chunks
• Send them to the LLM

It works well. It’s scalable. It’s fast.

But it’s not perfect.

Where Embedding-Based RAG Starts Struggling

After working with document-based AI systems, I noticed some real issues:

• Chunking often breaks context.
• Similarity ≠ relevance.
• Long PDFs (100+ pages) become messy.
• Explainability is weak — you just get “top K similar chunks”.

If you’re working with legal documents, financial reports, or research papers, this becomes very noticeable.

Sometimes the most “similar” chunk isn’t logically connected to the question.

That’s where things get interesting.

Enter Vectorless RAG

I recently explored PageIndex, an open-source framework that takes a completely different approach.

Instead of embeddings and similarity search, it builds a hierarchical tree structure of the document.

Think of it like this:

• Document → Chapters
• Chapters → Sections
• Sections → Subsections

Now instead of asking:

“Which chunk is most similar to my query?”

It asks:

“Where in this structured document would a human logically search for the answer?”

That’s a big shift.

How PageIndex Actually Works

Here’s the simplified version:

1. The document is parsed into a tree-like structure.
2. Each node represents a logical section.
3. When a query comes in:

• The LLM evaluates top-level nodes.
• It selects the most relevant branch.
• Then it drills down recursively.

4. Finally, it extracts the relevant content.

It feels less like searching and more like reasoning.

That difference matters.

Vector DB vs PageIndex — Practical Comparison

When Vector Databases Are Still the Best Choice

If you’re building:

• A chatbot over thousands of small documents
• A support bot over FAQs
• A product search system
• A multi-tenant SaaS knowledge base

Embedding-based RAG is still incredibly practical.

It scales beautifully.

When Vectorless RAG Makes More Sense

If you’re working with:

• 200-page contracts
• Financial reports
• Academic papers
• Highly structured PDFs

Then, structure-aware reasoning can outperform similarity search.

Especially when logical relationships matter more than surface similarity.

My Personal Take

As someone building AI agents and experimenting with different RAG architectures, I don’t see PageIndex as a replacement for vector databases.

I see it as an alternative strategy.

In fact, the most powerful systems might combine both:

• Vector DB for broad retrieval
• Structured reasoning for deep document understanding

Hybrid retrieval might be the real future.

Final Thoughts

Embeddings changed how we build AI systems.

But we shouldn’t assume they’re the only way.

Vectorless RAG challenges the default approach and forces us to rethink retrieval.

And honestly, that’s healthy for the ecosystem.

AI infrastructure is still evolving.

And experiments like this are what push it forward.

If you’re building RAG systems in production, I’d recommend at least testing both approaches.

You might be surprised.