Microservices, Event driven and clean architecture - containerisation, distributed caching; CQRS, Backend For Frontend, Mediator and Command pattern

Technology used

Backend

.Net 6, RabbitMq, MsSQL, MediatR, Swagger, Docker, Redis Cache

Update 26 August 2022 - Upgrade from .Net 5 to .Net 6

Frontend

React, Material UI, React-Redux, Redux, Redux-Thunk

Future

Kubernetes, Azure AKS, EKS, Azure SQL Server, AWS MS SQL Server

This article is not going to cover the "cherry on top" i.e. Kubernetes, Azure AKS, EKS , AWS MS SQL Server, GitLab. This will only be show cased in my next and final free micro service article. The frontend React and Redux application is also not going to be discussed in this article. This article does not discuss low level implementation detail of the Socca solution. However the source code is available in the Socca solution on the provided Github link.

Update 02 August 2022 - The application layers for the microservices was updated to match the domain centric approach / clean architecture. The application layer was removed. The services, events, commands, event handlers and command handlers were moved to the domain/core layer.

The source code can be found here: https://github.com/mandavamunya/Socca

NB. This application was only tested on a Linux environment (Macbook).

Introduction

Fictional Football League Management System

The ABC football league stakeholders would want to be able to manage football teams and football players. The managers should be able to fetch the player details. They should be able to link teams to a stadium. They should be able to transfer players from one team to another.

Assumptions

• The football clubs do not have a permanent home stadiums and can be reassigned to a new stadium before the beginning of each season.
• The football players do not have a permanent football team and can be reassigned to a new football club before the beginning of each season.
• Multiple events will be generated and possibly the event processors / consumers might take long to process hence why the use of an event bus.

Requirements

There are seven services in this project: FootballClub, Players, FootballClubStadium, PlayerTransfers, Stadium, DistributedCache (CQRS) and Web (Backend For Frontend).

The domain model of our applications contains five entities: FootballClub, Player, Stadium, FootballClubStadium and PlayerTransfer.

Event Driven Architecture

Update 17 May 2022 - NOTE: Create another figure as the one above. When using rabbitmq broker's queue one event is processed by only one consumer. Each event will have only one consumer. An event store is going to be used to log all the events; a MessageLogEvent queue will be used to carry each event as a json object. The distributed cache will also use a CacheDataEvent queue to save all the data.

The Event Driven Architecture is usually more suited to applications such as a Health Care Software System. An Event or message represents fact - is created when a change of an observed value happens, such as an action made by a user.

Characteristics of EDA Systems

Multicast Communication - When an event is published/produced the system is able to notify more than one subscriber/consumer

Real Time - Events are published as they occur or immediately after they are caught

Asynchronous Communication - Events keep coming and do not wait for already sent Events to get processed

Fine-grained Communication - means that all types of Events are sent even trivial simple ones, especially trivial and significant ones

Event Filtering - filters events and classifies them based on a set of parameters

Only Meta - Event messages inform the system and the interested parties about what happened. not how it happened or what to do about it

Implementation Approaches

Mediator - used for completed large Event Driven systems that need central orchestration.

A mediator is used for events that have multiple steps to be performed.

In this case the mediator divides the Initial Event into multiple Processing Events each corresponding to its individual step and after that it send them to the appropriate Event Processors via Event Channels.

Broker - is used when the system is relatedly simple and does not require central orchestration The event broker contains Event Channels that are used to transfer Events to and from Event Processors.

In this case we are going to use a broker together with other patterns for example Producer-Consumer pattern also known as the Publisher-Subscriber pattern.

Producers/Publishers/Event Generators create events and Consumers/Subscribers/Event Processors consume or subscribe or listen to the events.

Producer Services:

FootballClub 

• Produces a FootballClubAddedEvent event when a FootballClub is added to the FootballClub table.
• Produces a LinkToStadiumCreatedEvent event when a football club is assigned to a stadium.

Player

• Produces a PlayerAddedEvent event when a player is added to the Players table.
• Produces a PlayerTransferCreatedEvent event when a player is transferred.

Stadium 

• Produces a StadiumAddedEvent event when a stadium is added to the Stadiums table.

Consumer Services:

FootballClubStadium

• Subscribes or consumes the LinkToStadiumCreatedEvent events and saves the values to the FootballClubStadiums table.

PlayerTransfer

• Subscribes or consumes the PlayerTransferCreatedEvent events and saves the values to the PlayerTransfer table.

DistributedCache

• The Event Sourcing pattern was applied in this project. When implementing micro services we have a bounded context and each service will manipulate its own domain data. However this does not mean the data in the different services is not related. An issue arises when implementing joint queries across multiple databases. To solve this issue the update and read operations were separated from the database by implementing the Command and Query Responsibility Segregation (CQRS). Redis cache is used to store the events from all the services. The distributed cache service is a consumer/subscriber and it consists mainly of event handlers listening to domain events produced by the producers/publishers/event sources that own the data. 
• The DistributedCache service listens to all events (PlayerAddedEvent, FootballClubAddedEvent, LinkToStadiumCreatedEvent, PlayerTransferCreatedEvent, StadiumAddedEvent) and updates the Redis cache database.

Backend For Frontend:

Web

• Is a Backend For Frontend (BFF), i.e. a public API that is a bridge between the micro services and frontend application. There are cons and pros that are tied to the implementation of the BFF pattern[6].

Commands

The Command pattern is used to encapsulate all information needed to perform an action within each service and then trigger an event which is published to other services. The Mediator pattern is also used to make it easy to implement Event Handlers and Command Handlers.

Message Broker

The broker component does not need central orchestration.

In such simple instance Events flow through Event Processors using the Message Broker as a medium of communication between event processors

The Broker component contains Event Channels. Thats what we use for the Event flow

A broker is used when the system is relatedly simple and does not require central orchestration. The event broker contains Event Channels / Event Queues that are used to transfer Events from Event Generators to Event Processors

• RabbitMq a light weight broker is used as an Event Queue that channels events to Event Processors.

Advantages of Event Driven Architecture

• Improved Performance - asynchronous - parallel operations.
• Scalability - the best - extreme loose coupling between components.
• Deployability - scalability means ease of deployment - also thanks to the loosely coupled autonomous nature of components.
• Versatility - important - ability to adapt or be adapted to many different functions or activities.
• Adaptability - important - event based nature of the system. suitable to survive and thrive in unpredictable non linear environments and circumstances.
• Evolutionary Architecture - the reactive nature of system makes it possible for new processors to respond to new events as the business model changes also real time analytics events are seen learning behaviours of the REAL WORLD IN REALTIME etc are all great gains as well and of course there are limitations.

Disadvantages / Limitations of Event Driven Architecture

• Upfront Investment - and planning are required to inspire confidence.
• Increased complexity - Due to the highly distributed nature of the system it is nearly impossible to track a chain of events triggered by components if the system is large.
• Very Limited Testing - Testing is near impossible, this is why logging is very important, the asynchronous nature of the system makes it very difficult to test the system as a whole. And sometimes event unit tests in certain types of events is difficult. This actually is why monitoring and logging are extremely important.

Request-Response Approach (Command based)

A request-response approach is also used in this application especially when retrieving information from the database.

Frontend

Teams

Players

Redis Cache and Kubernetes

Redis Cache

Redis is an in-memory data structure store, used as a distributed, in-memory key–value database, cache and message broker, with optional durability[4]. Redis supports different kinds of abstract data structures, such as strings, lists, maps, sets, sorted sets, HyperLogLogs, bitmaps, streams, and spatial indices[4].

Kubernetes

Kubernetes is an open-source container-orchestration system for automating computer application deployment, scaling, and management[5]. It was originally designed by Google and is now maintained by the Cloud Native Computing Foundation[5].

Considerations should be made when load balancing a web app

The way states are managed in a web application (session persistence) has an impact on the load balancing configuration and scalability.

Non sticky persistence: in the case that the session state is saved in a distributed cache or database (e.g. Redis Cache). The load balancer can route requests to any web server that has access to that distributed cache.

Sticky persistence: in the case that the session state is saved in an in-memory cache which is an in-process memory state in the web server. The load balancer must route the requests or traffic from the client to a specific web server that saved its session state.

Event Sourcing

A distributed cache service was added to keep track of each application client's state. Only the current state of the client is saved in Redis cache.

What's next

• Upgrade application to .Net 6
• The PlayerTransferCreatedEvent and LinkToStadiumCreatedEvent implementation was added as an example of event sourcing. The PlayerAddedEvent, StadiumAddedEvent and FootballClubAddedEvent events implementation is still outstanding.
• Configure frontend client application and bff to communicate via https. They are currently being served via the http protocol:

  "ServiceProviders": 
    "DistributedCacheBase": "http://host.docker.internal:14417",
    "FootballClubBase": "http://host.docker.internal:62245",
    "FootballClubStadiumBase": "http://host.docker.internal:63784",
    "PlayerBase": "http://host.docker.internal:19578",
    "PlayerTransferBase": "http://host.docker.internal:16956",
    "StadiumBase": "http://host.docker.internal:3094"
  }        

Also take not the use of host.docker.internal instead of localhost. Use of kubernetes will make the code much more structured.

CORS was also configured to allow the Client to access the BFF via the following ports:

  "Settings": {
    "AllowedOrigins": "http://localhost:57347, https://localhost:44358, https://localhost:5001, http://localhost:5000"
  }        

• Add a security micro service
• Add Kubernetes support
• Add centralised logging
• Add self-healing using health checks
• Add metrics with Prometheus and Grafana
• Create pipelines on GitLab.
• Make use of Azure SQL Server, AWS MS SQL Server
• Deploy to Azure AKS and AWS EKS
• Add more unit tests
• Add functional tests and integration tests

Outstanding work

The entities FootballClubStadium and PlayerTransfer are actually event logs or history data and are not meant to be deleted. Each event must have a date occurred or CreatedDate property.

A property IsCurrent will also be added to each event and therefore another update old event implementation is needed to set IsCurrent to false before adding a new event.

    public class FootballClubStadium
    {
        ...
        public DateTime DateCreated { get; set;}
        public bool IsCurrent { get; set; }
    }


        

    public class PlayerTransfer
    {
        ...
        public DateTime DateCreated { get; set;}
        public bool IsCurrent { get; set; }
    
    }
            

Dockerizing

docker tag soccafootballclubstadium mandavadev/soccafootballclubstadium:1.0.0        

How to run

• Simply click the run button in Visual Studio 2019. Please see button encircled in an orange oval in the image below.

Dockerized microservices

Accessing dockerized APIs

The image above shows the BFF API

The image above shows the FootballClub API

Accessing all your microservices in the browser

FootballClub: https://localhost:44301/swagger/index.html

Players: https://localhost:44328/swagger/index.html

FootballClubStadium: https://localhost:44350/swagger/index.html

PlayerTransfers: https://localhost:44370/swagger/index.html

Stadium: https://localhost:44309/swagger/index.html

Distributed Cache: https://localhost:44305/swagger/index.html

Conclusion

The purpose of the articles is to educate the reader on the following topics; microservices architecture, event driven architecture, CQRS, event sourcing, clean architecture, containerisation, distributed caching and cache-aside pattern. The high level details were explained and an example of an application was given and can be found on Github; https://github.com/mandavamunya/Socca.

References

1. Introducing CQRS, The Microsoft Press Store by Pearson: https://www.microsoftpressstore.com/articles/article.aspx?p=2248809&seqNum=3
2. https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-docker-container-deployment?view=sql-server-ver15&pivots=cs1-bash
3. https://www.nginx.com/resources/glossary/load-balancing/
4. https://en.wikipedia.org/wiki/Redis
5. https://en.wikipedia.org/wiki/Kubernetes
6. https://akfpartners.com/growth-blog/backend-for-frontend
7. https://jlasoc.medium.com/what-is-domain-centric-architecture-e030e609c401

ARTICLE UPDATE

22-01-2022 Correction on the Event Driven Architecture diagram (added distributed cache service) and updated the related Distributed Cache section in the article.