Resilience with Java : The rate limiter

The database admin : Please, reduce the number of incoming queries, because the CPU load of the database is too high.

Me : Ok, how many queries can it handle properly ?

The database admin : about 350 a second, less or more

Me: Ok, then we will throttle the queries flux with a rate limiter.

Introduction

In distributed architectures, or just between an application and a database, sometimes we want to limit the rate of calls to not exhaust a resource. A slower response it better than no response and lost data.

Resiliency in distributed systems is the ability of a system to continue to function even when unexpected events occur.

An exemple in the Java world, with Resilience4j

Resilience4j is a lightweight fault tolerance Java library designed for functional programming.

It has 6 Core Modules :

• CircuitBreaker
• Bulkhead
• RateLimiter
• Retry
• TimeLimiter
• Cache

Here we will use the RateLimiter.

Lets start with a simple use-case : an `executorService` is given a threadpool of 1000 and is submitted 1000 times this task :

private static void someBlockingIOs() {
    try {
        sleep(550); // Some blocking IOs
        System.out.println("Hello world");
    } catch (InterruptedException ignored) {
    }
}        

We run and measure the rate of "Hello world"s with this code :

int numberOfTasks = 1_000;
var start = Instant.now();
try (ExecutorService executor = Executors.newFixedThreadPool(1000)) {
    range(0, numberOfTasks)
            .mapToObj(i -> (Runnable) RateLimiterPocApplication::someBlockingIOs)
            .map(runnable -> RateLimiter.decorateRunnable(customRateLimiter, runnable))
            .forEach(executor::submit);
    executor.shutdown();
    System.out.println("All tasks have completed successfully : " + executor.awaitTermination(1, MINUTES));
}
var end = Instant.now();
long durationInMillis = end.toEpochMilli() - start.toEpochMilli();
System.out.println("durationInMillis : " + durationInMillis);
System.out.println("rate : " + (double) numberOfTasks / durationInMillis * 1000 + " tasks/second");        

And we get this result :

[...]
Hello world
Hello world
All tasks have completed successfully : true
durationInMillis : 615
rate : 1626.0162601626016 tasks/second        

Ok, now let's create a rate-limiter from Resilience4j, with a limit of 350 per period of 1000 ms :

RateLimiterConfig config = RateLimiterConfig.custom(
                .limitRefreshPeriod(Duration.ofMillis(1000))
                .limitForPeriod(350)
                .timeoutDuration(Duration.ofMillis(1000))
                .build();
        RateLimiterRegistry rateLimiterRegistry = RateLimiterRegistry.of(config);
        RateLimiter customRateLimiter = rateLimiterRegistry.rateLimiter("ratelimiter");)        

Then, we can decorate each runnable with the rate-limiter :

[...]
.map(i -> (Runnable) () -> {
                        try {
                            sleep(550); // Some blocking IOs
                            System.out.println(i);
                        } catch (InterruptedException ignored) {
                        }
                    })
                    .map(runnable -> RateLimiter.decorateRunnable(customRateLimiter, runnable))
                    .forEach(executor::submit);
[...]        

And when we run it again we observe the right rate of tasks per second (~350) :

[...]
Hello world
Hello world
All tasks have completed successfully : true
durationInMillis : 2570
rate : 389.10505836575874 tasks/second        

Conclusion

The library Resilience4j is really easy to set up into existing code thanks of its use of the decorator pattern around each types of the Java Function API. It effectively reduced the rate of parallel execution of tasks with blocking calls inside. We didn't showed it, but Resilience4J also have compatibility with many libraries and frameworks : rxJava3, Micronaut, Spring-Cloud, ...

Don't forget it in a distributed architecture where deployment entities all have different scaling capacities. Then, you will finally be allowed to use the word "microservices".

Sources can be found here :

Thanks for reading !