Multi data-store architecture

Gone are the days when a typical search application talks to single type of database with a master-slave or some other form of replication as a fail over mechanism.

With the changing landscape as part of Web 3.0, user expectations from search applications are changing drastically. Now, user expects search to be much more smarter and intelligent. Generally, if he doesn't find what he is looking for on the first page of search results, it doesn't go well with him.

This is where concept of multi data-store architecture comes in handy.

Using this concept, multiple search types like following can be combined in backend to give a intelligent search to user.

1. Free Text search

2. Context Based search

3. Hierarchical search

4. Semantic/ Synonym based search

Features like faceting, auto suggest, auto correct,sorting search results aid user in finding acute result set.

There are a lot of different ways in which an application can take an search input from user.

1. Default is google way of accepting search term in free text form wherein application doesn't know in what context user is looking for search term. This is pure free text search.

2. One way is to derive implicit context from user search term using ontologies or taxonomies and then give back the results based on the context identified.

3. Another way is provide different contexts available as auto suggest recommendation to user and let user select the context from dropdown.

4. Old way of solving context problem is by using Advanced Search or Query builder wherein user can build a dynamic query by selecting contexts from dropdowns and combining them with binary operators.

Generally, user research gives an indication as to which kind of input methodology or UX an application should use depending on the domain users and existing competing products if any. However, future trend is to build artificial intelligence in search algorithm itself using natural language processing and let the user enter the search term in free form.

To deliver such smart search, one needs to use different search technologies and use the philosophy of a fit for purpose database.

One such smart search example we all use in everyday life is google search wherein google uses the IP address/City/Country details and then maps search term to these localized contexts to give very specific result set to user based on current trend in that locality.

One of the examples of this multi data-store architecture could be to build a search application wherein user requirements are to deliver search results matching with search term or any of its synonyms or in its hierarchy at top combined with free text result set at bottom.

A possible solution could be to use two NoSQL databases, a document based database like Marklogic and secondly graph database. 

Marklogic could be used for its strengths like customized free text search, boosting specific documents based on business rules, out of box faceting, sorting and dynamic snippet.

Graph Database could be used for its own strong points like deriving implicit context from search terms using ontologies or taxonomies along with synonym , hierarchy based search and relationships.

Output of graph could possibly be used as an input to Marklogic to combine context based search with free text search. A lower boost for free text search query combined with higher boost to context based queries could be used. If required, custom Lucene indexes or analyzers could be used with graph database or Elastic Search to build much more acute search queries.

Importantly, one needs to take into consideration performance of overall search response while combining different databases. 

To summarize, always try to use fit for purpose database for different search requirements and combine them rather than trying to fit all search requirements into a single database.