Jstack and Jmap stopped working
Hi Team,
I have a long pending issue with me and not able to get anything close to RC. Need your help in diagnosing this error.
Error :- Jstack and Jmap suddenly started failing only for jboss process with error "Unable to open socket file: target process not responding or HotSpot VM not loaded"
At the same time, Jstack and Jmap is working on a standalone java application in the same environment. Below more details.
The issue is predominantly seen in RHEL7.3 system.
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)
$jps
2576 jboss-modules.jar
4218 TftpServer
26300 Jps
$jstack 2576
2576: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
$jmap 2576
Attaching to process ID 2576, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file
But jstack of other process works 4218, only 2576 fails.
$ jstack 4218
2021-02-18 01:10:23
Full thread dump OpenJDK 64-Bit Server VM (25.275-b01 mixed mode):
I have already checked the thread for the same error -> https://access.redhat.com/solutions/317143, but somehow the scenario is different here. The commands are ran using the user owning the process and have all necessary permission in /tmp folder.
The issue occurs only after the application is running for a month or so. If we restart the application, then jstack starts working on the new process that is generated and subsequently fails after some operational days.
Kindly help me in understanding the RC of
1. Why suddenly the jstack and jmap starts failing, while it was working before.
2. What logs and traces needs to taken to check this issue.
3. Why jstack is working for the other process (4218) and failing only for 2576. One difference between the process i see is the memory used, will that be factor? Also 2576 is a jboss process, while 4218 is a standalone java.
The process "2576 "is using around 20G
top -n 1 | grep 2576
2576 user 20 0 20.007g 0.014t 10808 S 766.7 60.3 481518:00 java
While 4218 is using only few mb.
top -n 100 | grep 4218
4218 user 20 0 3902324 129696 10600 S 0.3 0.5 43:02.42 java
- As i said issue currently seen only in RHEL7.3, any known bug in 7.3?
Any help in this is much appreciated.
BR
Pradhun
