Virtual Threads, a new solution to improve concurrent API calls

Java applications traditionally rely on platform threads for concurrency, which can become a bottleneck when handling massive concurrent API calls that involve waiting for I/O or external resources. This limitation arises because each platform thread is tied to an operating system thread, which are expensive to create and manage. When numerous platform threads block waiting for external responses, they consume system resources and hinder overall application performance. This can lead to slow response times and reduced scalability as the number of concurrent calls increases.

What is the Virtual Thread?

A virtual thread is a thread that is not tied to a specific OS thread. A virtual thread still runs code on an OS thread, but when the code calls a blocking I/O operation, the Java runtime suspends the virtual thread until it can be resumed. The OS thread associated with the suspended virtual thread is now free to perform operations for other virtual threads.

Virtual threads are implemented in a similar way to virtual memory. To simulate a lot of memory, an operating system maps a large virtual address space to a limited amount of RAM. Similarly, to simulate a lot of threads, the Java runtime maps a large number of virtual threads to a small number of OS threads.

Unlike platform threads, virtual threads typically have a shallow call stack, performing as few as a single HTTP client call or a single JDBC query. They support thread-local variables and inheritable thread-local variables, but you should carefully consider using them because a single JVM might support millions of virtual threads. Virtual threads are suitable for running tasks that spend most of the time blocked, often waiting for I/O operations to complete. However, they are not intended for long-running CPU-intensive operations.

Platform Threads vs Virtual Threads

When to Use Virtual Threads?

Use virtual threads in high-throughput concurrent applications, especially those that consist of a great number of concurrent tasks that spend much of their time waiting. Server applications are examples of high-throughput applications because they typically handle many client requests that perform blocking I/O operations such as fetching resources.

Virtual threads do not execute code faster than platform threads. Their goal is to provide scale (greater throughput) rather than speed (lower latency).

Here are some key scenarios where virtual threads can help in Java:

1. High-Throughput Concurrent Applications:

• If your application deals with a large volume of concurrent tasks that spend most of their time waiting (e.g., waiting for network I/O, database calls, or external service responses), virtual threads are a great choice.
• They excel at handling numerous, short-lived tasks efficiently due to their lightweight nature and ability to avoid blocking the underlying carrier threads.

2. I/O-Bound Operations:

• Virtual threads are particularly well-suited for applications that perform a lot of I/O operations, such as web servers handling many client requests.
• Since they can efficiently context switch when waiting for I/O, other virtual threads can make progress, improving overall throughput.

3. Simplifying Asynchronous Programming:

• If your concurrent tasks involve waiting for external resources, virtual threads can often simplify your code.
• You might not need complex asynchronous programming approaches like callbacks or reactive streams, as virtual threads handle context switching internally.

Summary

In summary, virtual threads are a game-changer for handling massive concurrent API calls in multicast scenarios. Unlike traditional threads that block carrier threads during I/O waits, virtual threads relinquish control, allowing the carrier threads to handle other requests. This is ideal for high-concurrency situations where numerous API calls wait for external responses (like network calls or remote service invocations). This frees up carrier threads to service other requests simultaneously, dramatically improving scalability and responsiveness in multicast environments.