Analyzing JVM Thread issues

1. Issue reported

Users reported saying application is extremely slow; loading of the web pages takes more than 5 minutes or sometimes even unresponsive.

2. Audience for this article

• Java developers, Students,
• Any developers who's application runs on JVM
• Java application support engineers
• Architects

3. Validating the issue

Before jumping into actual troubleshooting steps, I thought of validating the issue that is reported. To do so, just tried logging into that application with administrator credentials (which has complete privilege) and found the poor response of the application while connecting to http console/portal. So issue reported is proven and needs addressing.

4. About the issue application

In my case, it was a 3rd party proprietary software which had the issues. However I want to keep this article generic for all Java platform users, so I am not adding or discussing about my application specifically.

5. Basic troubleshooting steps

• VM Health - The first step would be observing the behavior of the Virtual Machine, this is essential because your java application runs on top the VM and we need to ensure VM is healthy. I observed VM (in my case VM was assigned with 8 Cores & 16 GB RAM) was in peak usage; average CPU usage was >80+ and memory usage was around 60 to 70%.
• Process - Second step would be to identify the process which is consuming the cpu and memory, in my case it was my java application which was consuming most resources.

Please note, in some cases utility software's like anti-virus software, custom scripts, etc., that are running on the VM may consume the most VM resources which may affect your java application performance. In that case, issue needs to be reported to Infra team or necessary stakeholders to take action.

Tips - In linux machines, use 'top' command to find the VM resource usages and in windows, use 'Task Manager'.

6. Generating the thread dumps

Now we have narrowed the issue and arrived to the core area, that is identifying the application issue. To do so,

1. Generate 3 thread dumps with gap of 30 seconds each - This is required to understand the thread state of the process/application. I have collected 3 dumps in my case.
2. Generate the heap dump - This is required to collect the memory snapshot of a java process/application. For my case, it is clear that CPU usage was peak rather than memory and so focused only CPU/thread issue.

7. Thread dumps finding

On analyzing the dumps, I identified there are '105 Blocked threads' currently which is causing the java process to consuming more CPU.

A thread state. A thread can be in one of the following states:

• NEW - A thread that has not yet started is in this state.
• RUNNABLE - A thread executing in the Java virtual machine is in this state.
• BLOCKED - A thread that is blocked waiting for a monitor lock is in this state.
• WAITING - A thread that is waiting indefinitely for another thread to perform a particular action is in this state.
• TIMED_WAITING - A thread that is waiting for another thread to perform an action for up to a specified waiting time is in this state.
• TERMINATED - A thread that has exited is in this state.

8. Work around solution

As a work around or a quick resolution to this problem would be "release all the blocked threads".

As many of you know, easiest and simplest way to release the blocked threads is by 'restarting the application'. Yes, I also did the same to restored the application quickly as issue was in production, however we haven't a applied a permanent fix yet which is next and final step.

Please note, after restarting the application I have taken the thread dumps again to ensure and prove there are no blocked threads.

9. Permanent fix

In my case, I am supporting an application developed by a 3rd party software vendor and no access to source code. So I have logged a case with them to dig further on this by providing all the necessary data, and requesting a patch/fix to address the issue at the core.

Hope this article is helpful to detect the performance issue (related to CPU and thread) for java application.