Infected with Bitcoin Mining pool zombie??
Update:
I was able to read the suspicous binary file /tmp/dovecat
and it was actually mining bitcoin and send to pool.minexmr.com:443
. I reported the website the user is illegally using our server resource.
My question is, how do I identify how this script has been planted to the server because this is the 2nd time the script with a different name has been generated in /tmp
. Is there any way to dig down and find the main script which generates this mining script??
[root@localhost lib64]# strings /tmp/dovecat | grep 443 -B 10 -A 10
[1;37m nvidia
[45;1m
[1;37m opencl
"autosave": true,
"donate-level": 0,
"cpu": true,
"opencl": false,
"cuda": false,
"pools": [
{
"url": "pool.minexmr.com:443",
"user": "47F6dAwURi1fKZbDiSyN6y4bKXuP6kjixFBMft9RtNRMJXUm8AHMJ3mQmQYJWZi7T2igLwkqfPxGdPhwyMkPkpecSUQP6Ne",
"rig-id": "w1",
"keepalive": true,
"tls": true
}
]
[0;33m"%s" was changed, reloading configuration
[0;31mreloading failed
/1/config
/2/config
Original post
First of all, I'm pretty new to Red hat and very little knowledge of tomcat.
Server: CentOS7, tomcat running.
I have a few strange processes, and they connect to cloud computers in France, Germany, and China and consuming CPU and sending data . The CPU resource is almost being used by 50%.
And did losf
to find what files are opened by the suspicious process, there are some foundings as below. Has anyone experience this ? If then, how can I locate the trojan and remove. It seems it keeps planting a binary script under /tmp
- There is a main binary script under
/tmp/
asdovecat
orinit
and it connects to the mining pool computer outside. - There are other two processes and also connect via 145 and sending data.
First I found this process because the client with the infected server gets complaints from users that the server is slow. Then, I found /tmp/init
process was suspicious, I killed them and changed permission to read only. Then, today, another binary script is embedded dovecat
and consumes 50% CPU.
The process lists tomcat as one of opened files, I asked the application developer to change all host manager passwords, and but it doesn't solve, it created another process dovecat
today and does the same thing.
[root@localhost ~]# lsof -Pni
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 738u IPv4 36402 0t0 TCP *:111 (LISTEN)
systemd 1 root 739u IPv4 36403 0t0 UDP *:111
systemd 1 root 740u IPv6 36404 0t0 TCP *:111 (LISTEN)
systemd 1 root 741u IPv6 36405 0t0 UDP *:111
os 3136 tcuser 4u IPv4 1382319426 0t0 TCP 10.204.63.12:39560->45.159.179.132:145 (ESTABLISHED)
avahi-dae 8002 avahi 12u IPv4 13190 0t0 UDP *:5353
avahi-dae 8002 avahi 13u IPv4 13191 0t0 UDP *:33585
chronyd 8059 chrony 1u IPv4 46396 0t0 UDP 127.0.0.1:323
chronyd 8059 chrony 2u IPv6 46397 0t0 UDP [::1]:323
sshd 8641 root 3u IPv4 60673 0t0 TCP *:22 (LISTEN)
sshd 8641 root 4u IPv6 60675 0t0 TCP *:22 (LISTEN)
cupsd 8646 root 11u IPv6 49305 0t0 TCP [::1]:631 (LISTEN)
cupsd 8646 root 12u IPv4 49306 0t0 TCP 127.0.0.1:631 (LISTEN)
httpd 8701 root 4u IPv6 53733 0t0 TCP *:80 (LISTEN)
httpd 8706 daemon 4u IPv6 53733 0t0 TCP *:80 (LISTEN)
httpd 8711 daemon 4u IPv6 53733 0t0 TCP *:80 (LISTEN)
httpd 8720 daemon 4u IPv6 53733 0t0 TCP *:80 (LISTEN)
dnsmasq 9337 nobody 3u IPv4 40665 0t0 UDP *:67
dnsmasq 9337 nobody 5u IPv4 40668 0t0 UDP 192.168.122.1:53
dnsmasq 9337 nobody 6u IPv4 40669 0t0 TCP 192.168.122.1:53 (LISTEN)
sendmail 9419 root 4u IPv4 53855 0t0 TCP *:25 (LISTEN)
httpd 11362 daemon 4u IPv6 53733 0t0 TCP *:80 (LISTEN)
xrdp-sesm 14904 root 7u IPv4 696142283 0t0 TCP 127.0.0.1:3350 (LISTEN)
xrdp 14905 root 11u IPv4 696224861 0t0 TCP *:9012 (LISTEN)
sshd 16470 root 3u IPv4 1389679980 0t0 TCP 10.204.63.12:22->136.166.44.14:59416 (ESTABLISHED)
Xvnc 19213 root 7u IPv4 696298543 0t0 TCP 127.0.0.1:5911 (LISTEN)
Xvnc 19213 root 8u IPv6 696298544 0t0 TCP [::1]:5911 (LISTEN)
dovecat 22567 tcuser 10u IPv4 1386745097 0t0 TCP 10.204.63.12:40456->88.99.242.92:443 (ESTABLISHED)
java 22580 tcuser 40u IPv6 1370165684 0t0 TCP *:8080 (LISTEN)
java 22580 tcuser 41u IPv6 1370165685 0t0 TCP *:8009 (LISTEN)
java 22580 tcuser 43u IPv6 1388425085 0t0 TCP 10.204.63.12:8080->10.204.63.254:19341 (CLOSE_WAIT)
java 22580 tcuser 44u IPv6 1383635903 0t0 TCP 10.204.63.12:49124->10.204.63.13:1521 (ESTABLISHED)
java 22580 tcuser 45u IPv6 1388783564 0t0 TCP 10.204.63.12:8080->10.204.63.254:35594 (CLOSE_WAIT)
java 22580 tcuser 46u IPv6 1380820509 0t0 TCP 10.204.63.12:48618->10.204.63.13:1521 (ESTABLISHED)
java 22580 tcuser 47u IPv6 1382641721 0t0 TCP 10.204.63.12:48902->10.204.63.13:1521 (ESTABLISHED)
java 22580 tcuser 50u IPv6 1388711795 0t0 TCP 10.204.63.12:8080->10.204.63.254:38309 (CLOSE_WAIT)
java 22580 tcuser 51u IPv6 1385689103 0t0 TCP 10.204.63.12:49430->10.204.63.13:1521 (ESTABLISHED)
java 22580 tcuser 52u IPv6 1384431089 0t0 TCP 10.204.63.12:49240->10.204.63.13:1521 (ESTABLISHED)
java 22580 tcuser 64u IPv6 1380489059 0t0 TCP 10.204.63.12:48566->10.204.63.13:1521 (ESTABLISHED)
java 22580 tcuser 67u IPv6 1380476430 0t0 TCP 10.204.63.12:48568->10.204.63.13:1521 (ESTABLISHED)
java 22580 tcuser 68u IPv6 1380476431 0t0 TCP 10.204.63.12:48570->10.204.63.13:1521 (ESTABLISHED)
java 22580 tcuser 70u IPv6 1380525335 0t0 TCP 10.204.63.12:48572->10.204.63.13:1521 (ESTABLISHED)
java 22580 tcuser 71u IPv6 1370165700 0t0 TCP 127.0.0.1:8005 (LISTEN)
java 22580 tcuser 77u IPv6 1385310324 0t0 TCP 10.204.63.12:49368->10.204.63.13:1521 (ESTABLISHED)
java 22580 tcuser 78u IPv6 1370150837 0t0 TCP 10.204.63.12:47054->10.204.63.13:1521 (ESTABLISHED)
java 22580 tcuser 79u IPv6 1370159764 0t0 TCP 10.204.63.12:47056->10.204.63.13:1521 (ESTABLISHED)
java 22580 tcuser 80u IPv6 1370169784 0t0 TCP 10.204.63.12:47058->10.204.63.13:1521 (ESTABLISHED)
java 22580 tcuser 83u IPv6 1385407410 0t0 TCP 10.204.63.12:49386->10.204.63.13:1521 (ESTABLISHED)
.netnsd 23232 tcuser 0u IPv4 1389702370 0t0 TCP 10.204.63.12:42472->8.8.8.8:6003 (SYN_SENT)
zabbix_ag 25102 zabbix 4u IPv4 1044738462 0t0 TCP *:10050 (LISTEN)
zabbix_ag 25102 zabbix 5u IPv6 1044738463 0t0 TCP *:10050 (LISTEN)
zabbix_ag 25103 zabbix 4u IPv4 1044738462 0t0 TCP *:10050 (LISTEN)
zabbix_ag 25103 zabbix 5u IPv6 1044738463 0t0 TCP *:10050 (LISTEN)
zabbix_ag 25105 zabbix 4u IPv4 1044738462 0t0 TCP *:10050 (LISTEN)
zabbix_ag 25105 zabbix 5u IPv6 1044738463 0t0 TCP *:10050 (LISTEN)
zabbix_ag 25106 zabbix 4u IPv4 1044738462 0t0 TCP *:10050 (LISTEN)
zabbix_ag 25106 zabbix 5u IPv6 1044738463 0t0 TCP *:10050 (LISTEN)
zabbix_ag 25107 zabbix 4u IPv4 1044738462 0t0 TCP *:10050 (LISTEN)
zabbix_ag 25107 zabbix 5u IPv6 1044738463 0t0 TCP *:10050 (LISTEN)
zabbix_ag 25108 zabbix 4u IPv4 1044738462 0t0 TCP *:10050 (LISTEN)
zabbix_ag 25108 zabbix 5u IPv6 1044738463 0t0 TCP *:10050 (LISTEN)
Xvnc 27832 root 7u IPv4 263579313 0t0 TCP 127.0.0.1:5910 (LISTEN)
Xvnc 27832 root 8u IPv6 263579314 0t0 TCP [::1]:5910 (LISTEN)
[root@localhost ~]# ps aux | grep 22567
tcuser 22567 1399 3.7 3652052 2430768 ? Sl 08:59 1611:15 /tmp/dovecat
root 28148 0.0 0.0 116876 1040 pts/0 S+ 10:54 0:00 grep --color=auto 22567
[root@localhost ~]# lsof -p 3136
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
os 3136 tcuser cwd DIR 253,2 4096 4026534465 /home/tcuser/apache-tomcat-7.0.42/bin
os 3136 tcuser rtd DIR 253,0 270 512 /
os 3136 tcuser txt REG 253,2 1135000 4026534490 /home/tcuser/apache-tomcat-7.0.42/bin/os (deleted)
os 3136 tcuser 0u CHR 1,3 0t0 1028 /dev/null
os 3136 tcuser 1u CHR 1,3 0t0 1028 /dev/null
os 3136 tcuser 2u CHR 1,3 0t0 1028 /dev/null
os 3136 tcuser 3uW REG 253,2 4 4026541620 /home/tcuser/apache-tomcat-7.0.42/bin/vga.conf
os 3136 tcuser 4u IPv4 1382319426 0t0 TCP localhost.localdomain:39560->45.159.179.132:uaac (ESTABLISHED)
[root@localhost ~]# ps aux | grep 3136
root 303 0.0 0.0 116872 1040 pts/0 S+ 10:56 0:00 grep --color=auto 3136
tcuser 3136 0.7 0.0 107948 824 ? Ssl 04:10 2:53 ./os
[root@localhost ~]# lsof -p 22567
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dovecat 22567 tcuser cwd DIR 253,2 4096 4026534465 /home/tcuser/apache-tomcat-7.0.42/bin
dovecat 22567 tcuser rtd DIR 253,0 270 512 /
dovecat 22567 tcuser txt REG 253,0 7312872 117496376 /tmp/dovecat
dovecat 22567 tcuser mem REG 253,0 105824 41945874 /usr/lib64/libresolv-2.17.so
dovecat 22567 tcuser mem REG 253,0 31408 41945804 /usr/lib64/libnss_dns-2.17.so
dovecat 22567 tcuser mem REG 253,0 163400 41945781 /usr/lib64/ld-2.17.so
dovecat 22567 tcuser mem REG 253,0 2151672 41945788 /usr/lib64/libc-2.17.so
dovecat 22567 tcuser mem REG 253,0 61624 41945806 /usr/lib64/libnss_files-2.17.so
dovecat 22567 tcuser 0r CHR 1,3 0t0 1028 /dev/null
dovecat 22567 tcuser 1w FIFO 0,9 0t0 1386730382 pipe
dovecat 22567 tcuser 2w FIFO 0,9 0t0 1386730382 pipe
dovecat 22567 tcuser 3u a_inode 0,10 0 6625 [eventpoll]
dovecat 22567 tcuser 4r FIFO 0,9 0t0 1386745094 pipe
dovecat 22567 tcuser 5w FIFO 0,9 0t0 1386745094 pipe
dovecat 22567 tcuser 6r FIFO 0,9 0t0 1386745095 pipe
dovecat 22567 tcuser 7w FIFO 0,9 0t0 1386745095 pipe
dovecat 22567 tcuser 8u a_inode 0,10 0 6625 [eventfd]
dovecat 22567 tcuser 9r CHR 1,3 0t0 1028 /dev/null
dovecat 22567 tcuser 10u IPv4 1386745097 0t0 TCP localhost.localdomain:40456->static.92.242.99.88.clients.your-server.de:https (ESTABLISHED)
[root@localhost ~]# ps aux | grep 22567
root 14793 0.0 0.0 116872 1040 pts/0 S+ 10:48 0:00 grep --color=auto 22567
tcuser 22567 1399 3.7 3652052 2430768 ? Sl 08:59 1529:27 /tmp/dovecat
[root@localhost ~]#
[root@localhost ~]# lsof -p 23232
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
.netnsd 23232 tcuser cwd DIR 253,0 270 512 /
.netnsd 23232 tcuser rtd DIR 253,0 270 512 /
.netnsd 23232 tcuser txt REG 253,2 1292104 36401 /home/tcuser/.netnsd
.netnsd 23232 tcuser mem REG 253,0 163400 41945781 /usr/lib64/ld-2.17.so
.netnsd 23232 tcuser mem REG 253,0 2151672 41945788 /usr/lib64/libc-2.17.so
.netnsd 23232 tcuser mem REG 253,0 61624 41945806 /usr/lib64/libnss_files-2.17.so
.netnsd 23232 tcuser 0u IPv4 1390210057 0t0 TCP localhost.localdomain:42550->dns.google:6003 (SYN_SENT)
[root@localhost ~]# ps aux | grep 23232
tcuser 23232 0.0 0.0 10484 332 ? Ss 01:00 0:00 /home/tcuser/.netnsd -daemon
root 26091 0.0 0.0 116872 1040 pts/0 S+ 10:53 0:00 grep --color=auto 23232
[root@localhost ~]#
[root@localhost ~]# lsof -p 22567
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dovecat 22567 tcuser cwd DIR 253,2 4096 4026534465 /home/tcuser/apache-tomcat-7.0.42/bin
dovecat 22567 tcuser rtd DIR 253,0 270 512 /
dovecat 22567 tcuser txt REG 253,0 7312872 117496376 /tmp/dovecat
dovecat 22567 tcuser mem REG 253,0 105824 41945874 /usr/lib64/libresolv-2.17.so
dovecat 22567 tcuser mem REG 253,0 31408 41945804 /usr/lib64/libnss_dns-2.17.so
dovecat 22567 tcuser mem REG 253,0 163400 41945781 /usr/lib64/ld-2.17.so
dovecat 22567 tcuser mem REG 253,0 2151672 41945788 /usr/lib64/libc-2.17.so
dovecat 22567 tcuser mem REG 253,0 61624 41945806 /usr/lib64/libnss_files-2.17.so
dovecat 22567 tcuser 0r CHR 1,3 0t0 1028 /dev/null
dovecat 22567 tcuser 1w FIFO 0,9 0t0 1386730382 pipe
dovecat 22567 tcuser 2w FIFO 0,9 0t0 1386730382 pipe
dovecat 22567 tcuser 3u a_inode 0,10 0 6625 [eventpoll]
dovecat 22567 tcuser 4r FIFO 0,9 0t0 1386745094 pipe
dovecat 22567 tcuser 5w FIFO 0,9 0t0 1386745094 pipe
dovecat 22567 tcuser 6r FIFO 0,9 0t0 1386745095 pipe
dovecat 22567 tcuser 7w FIFO 0,9 0t0 1386745095 pipe
dovecat 22567 tcuser 8u a_inode 0,10 0 6625 [eventfd]
dovecat 22567 tcuser 9r CHR 1,3 0t0 1028 /dev/null
dovecat 22567 tcuser 10u IPv4 1386745097 0t0 TCP localhost.localdomain:40456->static.92.242.99.88.clients.your-server.de:https (ESTABLISHED)
[root@localhost ~]# ps aux | grep 22567
tcuser 22567 1399 3.7 3652052 2430768 ? Sl 08:59 1611:15 /tmp/dovecat
root 28148 0.0 0.0 116876 1040 pts/0 S+ 10:54 0:00 grep --color=auto 22567
[root@localhost ~]#