What are Java Virtual Threads ?
Introduction to Java Virtual Threads
Introduction
Java 21 was recently released. The latest release includes exciting and innovative features such as Virtual Threads, unnamed classes, Generational ZGC, etc. Java 21 is expected to revolutionize efficiency, performance, and scalability.
One of the features that caught my attention was Virtual Threads. They were first introduced as a part of Project Loom. This feature was in preview in the older releases i.e Java 19, 20. Now, it is a part of the language core feature.
Virtual threads are lightweight versions of the original Java platform threads. They are specially designed for developing high-throughput applications.
In this article, we will explore how Java managed concurrency before the introduction of virtual threads. Later, we will see what a virtual thread is and how it works. The article will also present illustrations and code snippets demonstrating working of Java virtual threads. With that, let’s embark on our journey and demystify Virtual threads.
Java Threads
Before understanding what a Virtual thread is, let’s understand the fundamentals of multi-threading in Java. A thread is a fundamental building block of concurrency. It represents a path of execution. A Java process is made up of different threads where each thread represents a computational procedure.
Modern computers have multiple processors. By executing independent threads on different processors, applications allow users to do multiple things at a time. For eg:- A server application can handle a user request and perform a database call at the same time.
Here is an example of how you can create and launch a thread in java :-
Internally, Java threads are designed to be wrappers over Operating system threads. In other words, whenever you create a thread, the java application will request Operating system to create a thread. Java threads are also known as Platform threads.As illustrated below, each Java thread is mapped to an OS thread.
Creating threads is an expensive operation. So, applications create a pool of threads, that are managed by an executor. New tasks are submitted to the executor to run. A free thread picks up the task from the queue and executes it.
One Thread per Request
Let’s assume that you have a simple server application. The application receives a request, fetches data from the database, and sends a response.
Here is the breakdown of the time taken for the above three operations :-
In this process, the most expensive operation is fetching the data from the database. Reading the data from database involves a network I/O and becomes a bottleneck. To find what is the CPU utilization, we divide the CPU time by the total time taken for the request.
In the above case, the CPU utilization is 0.001% (0.1 us+0.1 us)/(20ms + 0.1us+0.1us). This implies our CPU is heavily underutilised.
What if the server receives lot of requests concurrently ? How do we handle this ?
One of the techniques to handle multiple concurrent requests is to launch a thread for each new request. This is also known as Thread per Request model. As we increase the number of threads, the CPU utilisation would also increase. Here is a sneak-peak of how the CPU utilisation in proportion to the number of threads
As seen from the above, to achieve a 100% CPU utilisation, we need 100k threads. It sounds simple but does it work ? No. As discussed in the previous section, every Java thread maps to an OS thread. There is an upper limit on the maximum threads JVM can create.
Following are the drawbacks of Thread per Request model :-
Memory limitation - Assuming that each platform thread consumes 1MB, the total memory required for 100k threads is 100 GB (100K * 1MB). This would result in Out Of Memory error in any java application.
Context switching - If there are more threads than available CPUs, the OS spends a lot of time in context switching which degrades the performance.
Asynchronous programming
As seen in the previous section, there is an upper limit on the number of threads that a Java application can create. Hence, we can’t use the request per thread model to handle large number of requests concurrently.
Java introduced a new way to solve this problem by introducing Asynchronous programming. According to this programming model, developers had to break down a task into multiple sub-tasks. Each sub-task would be run in a separate lambda function.
The idea behind using asynchronous programming was to not block the main thread while performing a blocking I/O call. The main thread would be free to execute another task and it would resume the original task once it’s blocking I/O call completed.
This resulted in efficient resource utilization. And we were able to develop applications with fixed number of threads. Let’s now look at how to write asynchronous code.
Java provided libraries and APIs such as CompletableFutures, supplyAsync, runAsAsync, etc. The below code snippet shows asynchronous code written in java :-
In the above example, we call two different services, combine the results and then finally convert it into upper case. Fetching the data from the services is an I/O operation and is handled efficiently by the framework.
However, our code has become complicated. Following are the downsides of writing asynchronous code in java :-
Complexity - It can be seen from the above example that our simple logic has become complex and spread across more lines. Our actual code consists of only four lines. However, by using the asynchronous semantics, we need more than 10 lines.
Learning curve - It can have a steep learning curve for junior programmers who are acquainted with procedural programming. Personally, it took me more than 3 months to master the APIs and concepts back in 2018.
Debugging - It is difficult to debug asynchronous code as compared to sequential code. In sequential code, the stack trace gives you enough information of where the issue lies. The same doesn’t apply to asynchronous code.
The Thread per request and Asynchronous programming have certain downsides and limitations. To overcome these limitations, a new concurrency concept known as Virtual Threads was introduced. Let’s now understand Virtual threads in detail.
What are Virtual threads ?
Virtual threads are lightweight threads managed by the JVM (Java Virtual Machine) rather than the operating system. They were introduced as a part of the project Loom to simplify & improve concurrency in Java applications.
As seen in the previous section, there are several downsides of using Thread per Request or Asynchronous programming. Virtual threads overcomes these downsides by allowing the user to create light-weight threads on-demand.
Unlike platform threads, we don’t need to pool virtual threads. Additionally, since they are 1000x lighter than platform threads, you can create a million of them on your laptop.
To create a virtual thread, we use the builder method Thread.OfVirtual() & initiate the execution by passing the task to the start() method. Following example illustrates the process of creating virtual threads :-
The output of the above program is as follows :-
We can also use the ExecutorService, which manages the scheduling and execution of threads. Below example illustrates the process of launching virtual threads using the executor service.
In the above example, we launched 10 threads and each thread waited for 1 second and then printed a line on the console. Instead of 10, we can launch 1 million threads and the program wouldn’t crash.
So far, we have understood what Virtual threads are and how to create them. We will now understand how they actually work under the hood.
How do Virtual threads work ?
Virtual threads are user-mode threads. They differ from platform threads and don’t have 1:1 mapping with the OS threads. JVM manages a pool of OS threads. It then schedules virtual threads on the OS threads.
As seen above, the virtual threads T1, T2, and T3 are scheduled and executed on platform threads P1 and P2.
Blocking I/O is one of the primary bottlenecks which reduces the throughput of the system. In case a network call or a database call is done in a thread, the thread goes in blocked state. While it waits for the response, it can’t execute any other task. This reduces the overall efficiency.
Every virtual thread runs on a platform thread. Virtual thread has the ability to detect when a blocking call is made. As soon, as it detects a blocking call, it is moved out of the platform thread onto JVM’s heap. The constructs used are out of the scope of this article and I will write a separate article covering the internal constructs used.
Once the blocking call completes, the OS then delegates JVM to handle the response. JVM then copies the context of Virtual thread on to Platform thread. The Virtual thread then resumes the execution.
The below diagram illustrates the process of how Virtual threads are executed on the underlying platform threads.
When the Virtual thread is blocked, JVM can accept other tasks and execute it on the platform thread. Platform threads are only busy while performing CPU bound operations. They don’t go in a blocked state during the I/O operation. This results in efficient usage of computing resources.
Virtual threads have an overhead of copying the virtual thread context onto the platform thread. However, this is far less expensive than blocking the whole platform thread.
Let’s now look at few pros/cons of using Virtual threads. These pros & cons help us asses whether switching to Virtual threads would result in significant performance improvements.
Advantages
Overhead - Virtual threads are light-weight and have far less overhead as compared to OS threads. Applications can launch them at the run-time. Also, there is no upper limit on the number of virtual threads that can be launched. This results in efficient resource utilization.
Scalability - An application can process large number of requests concurrently by executing them on different virtual threads. This improves the overall application throughput.
Simplicity - As compared to asynchronous programming, virtual threads are easier to understand. Developers can write code that looks like sequential code. It is easier to debug issues and maintain code.
Asynchronous I/O - When a virtual thread encounters a blocking call, it is moved out of the platform thread. The platform thread is not blocked and can be used to run a different task. This results in efficient I/O and the application doesn’t get blocked.
Existing API integration - Virtual threads work seamlessly with existing Java concurrency APIs and libraries. Developers can harness the benefits of Virtual threads by using the same tools and libraries.
Disadvantages
CPU-bound task - If your application has CPU-bound tasks such as running video games, using Virtual threads might not result in performance improvements. Virtual threads are helpful only for I/O bound tasks.
JVM overhead - Inspite of being lightweight, JVM still needs to manage the virtual threads. It needs to schedule the virtual thread on the OS thread and copy the context. This becomes a concern when JVM handles large number of virtual threads.
Compatibility - Virtual threads are not available in the older release of Java. Developers may need to upgrade the JVM to take advantage of Virtual threads.
Adoption rate - The dependent frameworks and libraries such as Spring Boot, Apache TomCat, etc are building support for virtual threads. This adoption could take time and consumers of these frameworks will need to wait until these frameworks support virtual threads.
Summary
Virtual threads are light-weight threads introduced as a part of Project Loom. The feature was in preview in Java-19 and Java-20. It has now become a part of JDK-21.
Virtual threads address the challenges and limitations of Thread per request model and Asynchronous programming. These threads are managed by the JVM and are user threads. Internally, they are executed on OS threads.
Applications can now create millions of virtual threads and improve their throughput. They are simpler to understand and easier to debug.
Although, there are multiple advantages of using Virtual threads, they can’t be used if you are running CPU bound tasks. They are only helpful for running I/O heavy workloads. Moreover, there is an additional overhead for JVM to manage the virtual thread.
Virtual threads is one of the remarkable features introduced in JDK-21. In my opinion, it is going to be a revolutionary feature and the adoption will witness an increase in the coming years.
Thanks for reading the article! Before you go: