Applying reactive principles to a monolithic application

The reactive-systems paradigm and reactive-programming is getting more and more attention these days. Even though the design of an application is ideally done very early on in the development, it can be very rewarding to learn about different architectures (such as reactive-systems and microservices) in the context of existing (monolithic) applications. This article explains an example of how 'reactive' principles can be applied to old (non-reactive) applications. The article is rather technical and assumes some knowledge on software-design, java and java-EE.

problem setting

Imagine you are maintaining a legacy application and are looking into improving the performance of it. Depending on the nature and the design of the application, the complexity of such a challenge can range from trivial to extremely difficult.

When optimizing an application, it will always be the most rewarding when you take on the slowest part of the system first: after all, that's where there's most to gain.    

One popular way to optimize things, is to try and parallelize tasks that can be ran in parallel (asynchronous, in a non-blocking way). Let's say you have 4 tasks to execute. In order to run task-4, you need to know the result of task-1, task-2 and and task-3. If those 3 tasks don't depend on each other, (or on anything else for that matter), they can be executed in parallel. Then, when all 3 of them are done, task-4 can be started based on the result of the previous 3 tasks. 

That's simple enough in theory, but putting it in practice in a real-live application can be more complex that you would expect.

Now imagine your application has a critical resource (such as a database, a file, a message-bus, an external service, ...), which doesn't deal very well with multi-threaded access. Dealing with such resources is often done by means of some sort of blocking operation: database transaction, file-IO, network-IO, http request/response, ...  

Such resource is a point of contention in the application and becomes the main bottleneck which results in poor performance. In such a case, working on that bottleneck seems the most logical thing to do. But the 'bottleneck' nature of the resource makes it hard to simply 'parallelize' the use of it.

example

Let's take a concrete example to illustrate. Let's assume the resource is a database: A certain action triggers the following effects in the application:

1. a record needs to be updated in the database (implemented by storeRecord())
2. an external system needs to be consulted: request information based on the result of storeRecord() (implemented by consultServiceX())
3. then another external system needs to be consulted based on the same result of storeRecord() (consultServiceY())
4. the results of consultServiceX() and consultServiceY() need to be stored in the database too (processResults())

The traditional, straightforward way of implementing this (in java), could look like this:

public String applyChanges(String newValue){    
    String result1 = storeRecord(newValue);
    String result2 = consultServiceX(result1);
    String result3 = consultServiceY(result1);
    return processResults(Arrays.asList(result2,result3));
}

 In java-EE, transaction-management is typically taken care off by the application-server, and not by the application itself. (https://docs.oracle.com/javaee/7/tutorial/transactions003.htm)

When this code is running on a java-EEapplication server, it would run under one-and-the-same transaction:

• storeRecord(), when writing to the database, will start a transaction.
• This transaction stays open during consultServiceX() and consultServiceY().
• When finally processResults() completes successfully, the transaction will be committed (the changes are stored in the database).

The 'caller', which calls applyChanges() is blocked until the 4 steps are completed. During that time, because of the open transaction, the resource (database) is (partially) locked: other parties that want to read/write from/to it have to wait until the lock is freed. In this example this could be for a (relative) long time because consultServiceX() and consultServiceY() in the example are contacting an external service

Since consultServiceX() and consultServiceY() are mixed in the logic of the 'applyChanges()' method, they make that we end up with a long-running transaction (even though those 2 methods don't use the database). This is a problem: the database is locked for long periods which results in a slow and unresponsive system.

This is off course a synthetic and simplified example. Preventing this from happening in a new project shouldn't be a problem. But in case of a large and old code-base, it's a different story. Typically, the parts that make up the complete scheme, is more complex than this 4-line example too.

So far the problem-setting. So how to improve this?

 solution 1

A naive improvement on this could be to split it in 2 parts: running the middle and last steps asynchronous from the first one, chaining the different steps the one after the other:

public CompletableFuture<String> applyChanges(String newValue){
	String result1 = taskOne(newValue);
	return CompletableFuture.supplyAsync(() -> consultServiceX(result1))
	        .thenApply(result2 -> consultServiceY(result2))
	        .thenApply(result3 -> processResults(result3));
}

In this case the 'caller' receives a 'future' which will eventually contain the result. This looks better, but in reality, this could make things worse than before. Remember that processResults() is using a 'bottleneck' resource. This implementation will result in multiple instances of processResults() using the 'bottleneck' at the same time, possibly dead-locking each other and/or other tasks that are using the resource. So this doesn't work.

solution 2

To prevent overwhelming the bottleneck with multithreaded access, it is in this case a good idea to embrace the fact that some resources are just better off in a single-threaded environment. A simple approach to accomplish this in a multi-threaded application is by using a single-threaded threadpool dedicated for tasks that use this 'bottleneck' resource. This gives a number of advantages:

• no explicit blocking anymore when using this resource. Usage of the resource is always non-blocking by means of scheduling a task on that special threadpool.
• because there are no concurrent tasks locking it, the usage of the resource is optimal: only 1 (write) task can run at a time because the threadpool only has a single thread.

Think of it as as (extremely simplified) actor-like approach (https://en.wikipedia.org/wiki/Actor_model).

The CompletableFuture-API from java allows fine-grained control over which task runs on which threadpool. This allows to update the example to this:

ExecutorService bottleNeckThreadpool = Executors.newSingleThreadExecutor();
...
public CompletableFuture<String> applyChanges(String newValue){
    String result1 = taskOne(newValue);
    return CompletableFuture.supplyAsync(() -> consultServiceX(result1))
            .thenApply(result2 -> consultServiceY(result2))
            .thenApplyAsync(result2And3 -> processResults(result2And3), bottleNeckThreadpool);
}

The result is that consultServiceX() and consultServiceY() are executed asynchronously , as in solution 1. The difference however, is that the processing of the results of consultServiceX() and consultServiceY() is scheduled to be executed on the 'bottleNeckThreadpool'. This makes that, even though there can be multiple instances of consultServiceX() and consultServiceY() in parallel, all instances of processResults() will be executed sequentially, without interfering with each other. The resource still gets locked, but never for a long time. It makes that it stays available for reading which results in a better responsiveness of the system. All this without changing the implementation of the actual business-logic (implemented in the 4 methods).

the saga-pattern

When dealing with an realistic (more complex) application, it can be appropriate to create a dedicated class that contains the execution-flow of your logic (not necessarily the logic itself). For instance to keep track of intermediate values and to be able to do the right thing in case of errors.

When you do this, I suggest to read-up about the 'saga-pattern', which is a mechanism originating from the world of microservices: it's a way of implementing transactions without blocking. This simplified example was actually inspired by this pattern.

wrap up

The example here is obviously not complete, it is only meant to illustrate a concept. I didn't cover exception-handling and there are a few other things to consider too: for example what happens when consultServiceX() never completes? Additionally there a a few gotchas when using a single-threaded threadpool in combination with CompletableFuture which can result in a few surprises. Maybe this could be the topic of another post later.

In my concrete case, applying this trick to an existing application resulted in a substantial gain in performance, without having to completely overhaul the architecture. Overhauling the architecture of an existing application is often not realistically possible.

The key to get it to work was embracing the fact that some parts of the application are just not well equipped to deal with multiple concurrent threads. Introducing a single-threaded threadpool allowed to use this particular resource in a non-blocking fashion, regardless of that limitation.

In the end, it's still a monolithic java-EE application, but by carefully applying some 'reactive' principles, significant improvements are possible.

references and background:

• https://en.wikipedia.org/wiki/Actor_model 
• https://akka.io/  
• https://www.reactivemanifesto.org/
• https://dzone.com/articles/20-examples-of-using-javas-completablefuture
• https://microservices.io/patterns/data/saga.html