Materialized Views and Microservices
It might be time to create a materialized view.
What is a materialized view?
A materialized view is a read-only representation of the source data in a format that best serves one or queries and queries are served from the view. A materialized view can be useful when the format in which data is optimally stored is not the optimal format in which it will be queried.
Eg: say CustomerService has all information about customers, including the location/region/state, OrderService has details of all orders, the items in the order and has a reference to the customerId. Following best practices CustomerService and OrderService have separate databases. Now if your UI had a dashboard view that needs to show the top 5 items ordered in the last week by customers in each state/region, pulling this information from two different services and stitching it together is not easy; unless you have a materialized view that has the customer, region, order , items data in a single datastore from which this query can be served.
Or say you have a use case to stitch together information related to the configuration of some IoT devices stored in a document store and the hourly average of their performance metrics available in a time series database and serve these on a UI that will support filtering and sorting by configuration fields or the performance related fields. With a materialized view, that has the configuration information and the averaged performance info available in the same datastore , this use case is far easier to achieve than without it.
Persistence of materialized view
The data in the view is only a read-only copy and can be rebuilt from the source data, if lost. So the materialized view can be held only in a cache with no persistence. That said, it often makes sense to persist the materialized view since significant resources may be needed to rebuild the view from the source.
Recommended by LinkedIn
The view can be persisted in a datastore completely different from the source data.
The view needs to be updated whenever the source data changes. This often involves subscribing to change notifications from the source and reacting to those change notifications by updating the view data. Note that the data in the view may not always be consistent with the source data since it relies on notification handling and notifications could get delayed.
It is useful to create indexes on the materialized view where the datastore supports it.
Challenges with Materialized Views
Keeping the materialized view in sync with the source data requires additional processing; additional event notifications. This could be a good fit in event driven architectures where change notifications and pub/sub patterns are already in use.
More than one materialized view may be needed for the same source data if there are multiple use cases that would result in complex queries without materialized view. The additional overhead of keeping the views consistent will increase in this case.
When to use materialized views?
Materialized views can solve some unique challenges related to stitching together information from disparate services or datastores for the UI. Use this pattern to avoid some very complex queries and in-memory joins of data.