Infected with Bitcoin Mining pool zombie??

Latest response

Update:
I was able to read the suspicous binary file /tmp/dovecat and it was actually mining bitcoin and send to pool.minexmr.com:443. I reported the website the user is illegally using our server resource.

My question is, how do I identify how this script has been planted to the server because this is the 2nd time the script with a different name has been generated in /tmp. Is there any way to dig down and find the main script which generates this mining script??

[root@localhost lib64]# strings /tmp/dovecat | grep 443 -B 10 -A 10
[1;37m nvidia
[45;1m
[1;37m opencl
    "autosave": true,
    "donate-level": 0,
    "cpu": true,
    "opencl": false,
    "cuda": false,
    "pools": [
        {
            "url": "pool.minexmr.com:443",
            "user": "47F6dAwURi1fKZbDiSyN6y4bKXuP6kjixFBMft9RtNRMJXUm8AHMJ3mQmQYJWZi7T2igLwkqfPxGdPhwyMkPkpecSUQP6Ne",
            "rig-id": "w1",
            "keepalive": true,
            "tls": true
        }
    ]
[0;33m"%s" was changed, reloading configuration
[0;31mreloading failed
/1/config
/2/config

Original post

First of all, I'm pretty new to Red hat and very little knowledge of tomcat.
Server: CentOS7, tomcat running.

I have a few strange processes, and they connect to cloud computers in France, Germany, and China and consuming CPU and sending data . The CPU resource is almost being used by 50%.

And did losf to find what files are opened by the suspicious process, there are some foundings as below. Has anyone experience this ? If then, how can I locate the trojan and remove. It seems it keeps planting a binary script under /tmp

  1. There is a main binary script under /tmp/ as dovecat or init and it connects to the mining pool computer outside.
  2. There are other two processes and also connect via 145 and sending data.

First I found this process because the client with the infected server gets complaints from users that the server is slow. Then, I found /tmp/init process was suspicious, I killed them and changed permission to read only. Then, today, another binary script is embedded dovecat and consumes 50% CPU.

The process lists tomcat as one of opened files, I asked the application developer to change all host manager passwords, and but it doesn't solve, it created another process dovecat today and does the same thing.

[root@localhost ~]# lsof -Pni
COMMAND     PID   USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
systemd       1   root  738u  IPv4      36402      0t0  TCP *:111 (LISTEN)
systemd       1   root  739u  IPv4      36403      0t0  UDP *:111
systemd       1   root  740u  IPv6      36404      0t0  TCP *:111 (LISTEN)
systemd       1   root  741u  IPv6      36405      0t0  UDP *:111
os         3136   tcuser    4u  IPv4 1382319426      0t0  TCP 10.204.63.12:39560->45.159.179.132:145 (ESTABLISHED)
avahi-dae  8002  avahi   12u  IPv4      13190      0t0  UDP *:5353
avahi-dae  8002  avahi   13u  IPv4      13191      0t0  UDP *:33585
chronyd    8059 chrony    1u  IPv4      46396      0t0  UDP 127.0.0.1:323
chronyd    8059 chrony    2u  IPv6      46397      0t0  UDP [::1]:323
sshd       8641   root    3u  IPv4      60673      0t0  TCP *:22 (LISTEN)
sshd       8641   root    4u  IPv6      60675      0t0  TCP *:22 (LISTEN)
cupsd      8646   root   11u  IPv6      49305      0t0  TCP [::1]:631 (LISTEN)
cupsd      8646   root   12u  IPv4      49306      0t0  TCP 127.0.0.1:631 (LISTEN)
httpd      8701   root    4u  IPv6      53733      0t0  TCP *:80 (LISTEN)
httpd      8706 daemon    4u  IPv6      53733      0t0  TCP *:80 (LISTEN)
httpd      8711 daemon    4u  IPv6      53733      0t0  TCP *:80 (LISTEN)
httpd      8720 daemon    4u  IPv6      53733      0t0  TCP *:80 (LISTEN)
dnsmasq    9337 nobody    3u  IPv4      40665      0t0  UDP *:67
dnsmasq    9337 nobody    5u  IPv4      40668      0t0  UDP 192.168.122.1:53
dnsmasq    9337 nobody    6u  IPv4      40669      0t0  TCP 192.168.122.1:53 (LISTEN)
sendmail   9419   root    4u  IPv4      53855      0t0  TCP *:25 (LISTEN)
httpd     11362 daemon    4u  IPv6      53733      0t0  TCP *:80 (LISTEN)
xrdp-sesm 14904   root    7u  IPv4  696142283      0t0  TCP 127.0.0.1:3350 (LISTEN)
xrdp      14905   root   11u  IPv4  696224861      0t0  TCP *:9012 (LISTEN)
sshd      16470   root    3u  IPv4 1389679980      0t0  TCP 10.204.63.12:22->136.166.44.14:59416 (ESTABLISHED)
Xvnc      19213   root    7u  IPv4  696298543      0t0  TCP 127.0.0.1:5911 (LISTEN)
Xvnc      19213   root    8u  IPv6  696298544      0t0  TCP [::1]:5911 (LISTEN)
dovecat   22567   tcuser   10u  IPv4 1386745097      0t0  TCP 10.204.63.12:40456->88.99.242.92:443 (ESTABLISHED)
java      22580   tcuser   40u  IPv6 1370165684      0t0  TCP *:8080 (LISTEN)
java      22580   tcuser   41u  IPv6 1370165685      0t0  TCP *:8009 (LISTEN)
java      22580   tcuser   43u  IPv6 1388425085      0t0  TCP 10.204.63.12:8080->10.204.63.254:19341 (CLOSE_WAIT)
java      22580   tcuser   44u  IPv6 1383635903      0t0  TCP 10.204.63.12:49124->10.204.63.13:1521 (ESTABLISHED)
java      22580   tcuser   45u  IPv6 1388783564      0t0  TCP 10.204.63.12:8080->10.204.63.254:35594 (CLOSE_WAIT)
java      22580   tcuser   46u  IPv6 1380820509      0t0  TCP 10.204.63.12:48618->10.204.63.13:1521 (ESTABLISHED)
java      22580   tcuser   47u  IPv6 1382641721      0t0  TCP 10.204.63.12:48902->10.204.63.13:1521 (ESTABLISHED)
java      22580   tcuser   50u  IPv6 1388711795      0t0  TCP 10.204.63.12:8080->10.204.63.254:38309 (CLOSE_WAIT)
java      22580   tcuser   51u  IPv6 1385689103      0t0  TCP 10.204.63.12:49430->10.204.63.13:1521 (ESTABLISHED)
java      22580   tcuser   52u  IPv6 1384431089      0t0  TCP 10.204.63.12:49240->10.204.63.13:1521 (ESTABLISHED)
java      22580   tcuser   64u  IPv6 1380489059      0t0  TCP 10.204.63.12:48566->10.204.63.13:1521 (ESTABLISHED)
java      22580   tcuser   67u  IPv6 1380476430      0t0  TCP 10.204.63.12:48568->10.204.63.13:1521 (ESTABLISHED)
java      22580   tcuser   68u  IPv6 1380476431      0t0  TCP 10.204.63.12:48570->10.204.63.13:1521 (ESTABLISHED)
java      22580   tcuser   70u  IPv6 1380525335      0t0  TCP 10.204.63.12:48572->10.204.63.13:1521 (ESTABLISHED)
java      22580   tcuser   71u  IPv6 1370165700      0t0  TCP 127.0.0.1:8005 (LISTEN)
java      22580   tcuser   77u  IPv6 1385310324      0t0  TCP 10.204.63.12:49368->10.204.63.13:1521 (ESTABLISHED)
java      22580   tcuser   78u  IPv6 1370150837      0t0  TCP 10.204.63.12:47054->10.204.63.13:1521 (ESTABLISHED)
java      22580   tcuser   79u  IPv6 1370159764      0t0  TCP 10.204.63.12:47056->10.204.63.13:1521 (ESTABLISHED)
java      22580   tcuser   80u  IPv6 1370169784      0t0  TCP 10.204.63.12:47058->10.204.63.13:1521 (ESTABLISHED)
java      22580   tcuser   83u  IPv6 1385407410      0t0  TCP 10.204.63.12:49386->10.204.63.13:1521 (ESTABLISHED)
.netnsd   23232   tcuser    0u  IPv4 1389702370      0t0  TCP 10.204.63.12:42472->8.8.8.8:6003 (SYN_SENT)
zabbix_ag 25102 zabbix    4u  IPv4 1044738462      0t0  TCP *:10050 (LISTEN)
zabbix_ag 25102 zabbix    5u  IPv6 1044738463      0t0  TCP *:10050 (LISTEN)
zabbix_ag 25103 zabbix    4u  IPv4 1044738462      0t0  TCP *:10050 (LISTEN)
zabbix_ag 25103 zabbix    5u  IPv6 1044738463      0t0  TCP *:10050 (LISTEN)
zabbix_ag 25105 zabbix    4u  IPv4 1044738462      0t0  TCP *:10050 (LISTEN)
zabbix_ag 25105 zabbix    5u  IPv6 1044738463      0t0  TCP *:10050 (LISTEN)
zabbix_ag 25106 zabbix    4u  IPv4 1044738462      0t0  TCP *:10050 (LISTEN)
zabbix_ag 25106 zabbix    5u  IPv6 1044738463      0t0  TCP *:10050 (LISTEN)
zabbix_ag 25107 zabbix    4u  IPv4 1044738462      0t0  TCP *:10050 (LISTEN)
zabbix_ag 25107 zabbix    5u  IPv6 1044738463      0t0  TCP *:10050 (LISTEN)
zabbix_ag 25108 zabbix    4u  IPv4 1044738462      0t0  TCP *:10050 (LISTEN)
zabbix_ag 25108 zabbix    5u  IPv6 1044738463      0t0  TCP *:10050 (LISTEN)
Xvnc      27832   root    7u  IPv4  263579313      0t0  TCP 127.0.0.1:5910 (LISTEN)
Xvnc      27832   root    8u  IPv6  263579314      0t0  TCP [::1]:5910 (LISTEN)

[root@localhost ~]# ps aux | grep 22567
tcuser     22567 1399  3.7 3652052 2430768 ?     Sl   08:59 1611:15 /tmp/dovecat
root     28148  0.0  0.0 116876  1040 pts/0    S+   10:54   0:00 grep --color=auto 22567
[root@localhost ~]# lsof -p 3136
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
COMMAND  PID USER   FD   TYPE     DEVICE SIZE/OFF       NODE NAME
os      3136 tcuser  cwd    DIR      253,2     4096 4026534465 /home/tcuser/apache-tomcat-7.0.42/bin
os      3136 tcuser  rtd    DIR      253,0      270        512 /
os      3136 tcuser  txt    REG      253,2  1135000 4026534490 /home/tcuser/apache-tomcat-7.0.42/bin/os (deleted)
os      3136 tcuser    0u   CHR        1,3      0t0       1028 /dev/null
os      3136 tcuser    1u   CHR        1,3      0t0       1028 /dev/null
os      3136 tcuser    2u   CHR        1,3      0t0       1028 /dev/null
os      3136 tcuser    3uW  REG      253,2        4 4026541620 /home/tcuser/apache-tomcat-7.0.42/bin/vga.conf
os      3136 tcuser    4u  IPv4 1382319426      0t0        TCP localhost.localdomain:39560->45.159.179.132:uaac (ESTABLISHED)
[root@localhost ~]# ps aux | grep 3136
root       303  0.0  0.0 116872  1040 pts/0    S+   10:56   0:00 grep --color=auto 3136
tcuser      3136  0.7  0.0 107948   824 ?        Ssl  04:10   2:53 ./os

[root@localhost ~]# lsof -p 22567
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
COMMAND   PID USER   FD      TYPE     DEVICE SIZE/OFF       NODE NAME
dovecat 22567 tcuser  cwd       DIR      253,2     4096 4026534465 /home/tcuser/apache-tomcat-7.0.42/bin
dovecat 22567 tcuser  rtd       DIR      253,0      270        512 /
dovecat 22567 tcuser  txt       REG      253,0  7312872  117496376 /tmp/dovecat
dovecat 22567 tcuser  mem       REG      253,0   105824   41945874 /usr/lib64/libresolv-2.17.so
dovecat 22567 tcuser  mem       REG      253,0    31408   41945804 /usr/lib64/libnss_dns-2.17.so
dovecat 22567 tcuser  mem       REG      253,0   163400   41945781 /usr/lib64/ld-2.17.so
dovecat 22567 tcuser  mem       REG      253,0  2151672   41945788 /usr/lib64/libc-2.17.so
dovecat 22567 tcuser  mem       REG      253,0    61624   41945806 /usr/lib64/libnss_files-2.17.so
dovecat 22567 tcuser    0r      CHR        1,3      0t0       1028 /dev/null
dovecat 22567 tcuser    1w     FIFO        0,9      0t0 1386730382 pipe
dovecat 22567 tcuser    2w     FIFO        0,9      0t0 1386730382 pipe
dovecat 22567 tcuser    3u  a_inode       0,10        0       6625 [eventpoll]
dovecat 22567 tcuser    4r     FIFO        0,9      0t0 1386745094 pipe
dovecat 22567 tcuser    5w     FIFO        0,9      0t0 1386745094 pipe
dovecat 22567 tcuser    6r     FIFO        0,9      0t0 1386745095 pipe
dovecat 22567 tcuser    7w     FIFO        0,9      0t0 1386745095 pipe
dovecat 22567 tcuser    8u  a_inode       0,10        0       6625 [eventfd]
dovecat 22567 tcuser    9r      CHR        1,3      0t0       1028 /dev/null
dovecat 22567 tcuser   10u     IPv4 1386745097      0t0        TCP localhost.localdomain:40456->static.92.242.99.88.clients.your-server.de:https (ESTABLISHED)
[root@localhost ~]# ps aux | grep 22567
root     14793  0.0  0.0 116872  1040 pts/0    S+   10:48   0:00 grep --color=auto 22567
tcuser     22567 1399  3.7 3652052 2430768 ?     Sl   08:59 1529:27 /tmp/dovecat
[root@localhost ~]#

[root@localhost ~]# lsof -p 23232
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
COMMAND   PID USER   FD   TYPE     DEVICE SIZE/OFF     NODE NAME
.netnsd 23232 tcuser  cwd    DIR      253,0      270      512 /
.netnsd 23232 tcuser  rtd    DIR      253,0      270      512 /
.netnsd 23232 tcuser  txt    REG      253,2  1292104    36401 /home/tcuser/.netnsd
.netnsd 23232 tcuser  mem    REG      253,0   163400 41945781 /usr/lib64/ld-2.17.so
.netnsd 23232 tcuser  mem    REG      253,0  2151672 41945788 /usr/lib64/libc-2.17.so
.netnsd 23232 tcuser  mem    REG      253,0    61624 41945806 /usr/lib64/libnss_files-2.17.so
.netnsd 23232 tcuser    0u  IPv4 1390210057      0t0      TCP localhost.localdomain:42550->dns.google:6003 (SYN_SENT)
[root@localhost ~]# ps aux | grep 23232
tcuser     23232  0.0  0.0  10484   332 ?        Ss   01:00   0:00 /home/tcuser/.netnsd -daemon
root     26091  0.0  0.0 116872  1040 pts/0    S+   10:53   0:00 grep --color=auto 23232
[root@localhost ~]#

[root@localhost ~]# lsof -p 22567
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
COMMAND   PID USER   FD      TYPE     DEVICE SIZE/OFF       NODE NAME
dovecat 22567 tcuser  cwd       DIR      253,2     4096 4026534465 /home/tcuser/apache-tomcat-7.0.42/bin
dovecat 22567 tcuser  rtd       DIR      253,0      270        512 /
dovecat 22567 tcuser  txt       REG      253,0  7312872  117496376 /tmp/dovecat
dovecat 22567 tcuser  mem       REG      253,0   105824   41945874 /usr/lib64/libresolv-2.17.so
dovecat 22567 tcuser  mem       REG      253,0    31408   41945804 /usr/lib64/libnss_dns-2.17.so
dovecat 22567 tcuser  mem       REG      253,0   163400   41945781 /usr/lib64/ld-2.17.so
dovecat 22567 tcuser  mem       REG      253,0  2151672   41945788 /usr/lib64/libc-2.17.so
dovecat 22567 tcuser  mem       REG      253,0    61624   41945806 /usr/lib64/libnss_files-2.17.so
dovecat 22567 tcuser    0r      CHR        1,3      0t0       1028 /dev/null
dovecat 22567 tcuser    1w     FIFO        0,9      0t0 1386730382 pipe
dovecat 22567 tcuser    2w     FIFO        0,9      0t0 1386730382 pipe
dovecat 22567 tcuser    3u  a_inode       0,10        0       6625 [eventpoll]
dovecat 22567 tcuser    4r     FIFO        0,9      0t0 1386745094 pipe
dovecat 22567 tcuser    5w     FIFO        0,9      0t0 1386745094 pipe
dovecat 22567 tcuser    6r     FIFO        0,9      0t0 1386745095 pipe
dovecat 22567 tcuser    7w     FIFO        0,9      0t0 1386745095 pipe
dovecat 22567 tcuser    8u  a_inode       0,10        0       6625 [eventfd]
dovecat 22567 tcuser    9r      CHR        1,3      0t0       1028 /dev/null
dovecat 22567 tcuser   10u     IPv4 1386745097      0t0        TCP localhost.localdomain:40456->static.92.242.99.88.clients.your-server.de:https (ESTABLISHED)
[root@localhost ~]# ps aux | grep 22567
tcuser     22567 1399  3.7 3652052 2430768 ?     Sl   08:59 1611:15 /tmp/dovecat
root     28148  0.0  0.0 116876  1040 pts/0    S+   10:54   0:00 grep --color=auto 22567
[root@localhost ~]#

Responses