My VPS is under attack

You know I can't live without my VPS. I am sharing a recent security incident I fixed in less than an hour and hope you can learn some lesson from me.

Problem

Recently, I got an notification email from Azure:

Action Required

I didn't realize there is any brute force traffic from VPS. Is it a false alarm? Is my VPS exploited? Let me find it out!

Investigation

First, I checked the networking metric in azure portal:

Azure portal metrics

The outbound connection rate has increased dramatically to near 500 per sec since 2 days ago.

Then, I checked the active connections using netstat

> netstat -p
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 test2.internal.cl:47518 206.189.240.115:ssh     TIME_WAIT   -
tcp        0     68 test2.internal.cl:40230 milcao.net:ssh          ESTABLISHED 22810/httpd
tcp        0      0 test2.internal.cl:38426 45.77.161.164.vultr:ssh TIME_WAIT   -
tcp        0      0 test2.internal.cl:38662 95.179.158.9.vultru:ssh TIME_WAIT   -
tcp        0      0 test2.internal.cl:56592 211.103.135.97:ssh      TIME_WAIT   -
tcp        0      0 test2.internal.cl:41496 kvm3rdn9.websouls.n:ssh TIME_WAIT   -
tcp        0      0 test2.internal.cl:37192 192-46-237-139.ip.l:ssh TIME_WAIT   -
tcp        0      1 test2.internal.cl:53804 static.151.182.161.:ssh SYN_SENT    22810/httpd
tcp        0      0 test2.internal.cl:50576 204.16.243.93:ssh       TIME_WAIT   -
tcp        0      1 test2.internal.cl:44016 www.lummerstorfer.f:ssh SYN_SENT    22810/httpd
tcp        0      0 test2.internal.cl:59716 114.117.249.27:ssh      TIME_WAIT   -
^C

I noticed that process 22810/httpd is trying to ssh to a number of machines periodically. My VPS is definitely exploited to do this brute force attack.

Next step is to figure out who is running this malicious program?

> ps -ef|grep 22810
test     22810 22805 16 09:09 ?        00:02:18 /usr/sbin/httpd .rsync/c/blitz64 -t 515 -f 1 -s 12 -S 8 -p 0 -d 1 p ip

test is a user I once created for test purpose and it has a very simple password: 123456. Usually, sshd has some sort of protection against brute force attack. For instance, if there were many successive failed login from a specific IP, further login from this IP would be forbidden for a while. However, my password is so simple that the attacker can easily obtain it using brute force attack without triggering the default defense mechanism. Fortunately, there is no data loss because this account has no access to important data or config.

ps command shows the attacker is launching a bunch of processes:

> ps -fu test
UID        PID  PPID  C STIME TTY          TIME CMD
test      3483     1  0  2022 ?        00:00:19 ./bin/tor -f etctor/tor/torrc1 --RunAsDaemon 1
test     22642     1  0 Jan01 ?        00:00:00 /bin/bash ./go
test     22804 22642  0 09:09 ?        00:00:00 timeout 6h ./blitz -t 515 -f 1 -s 12 -S 8 -p 0 -d 1 p ip
test     22805 22804  0 09:09 ?        00:00:00 /bin/bash ./blitz -t 515 -f 1 -s 12 -S 8 -p 0 -d 1 p ip
test     22810 22805 18 09:09 ?        00:04:16 /usr/sbin/httpd .rsync/c/blitz64 -t 515 -f 1 -s 12 -S 8 -p 0 -d 1 p
test     24143     1 99 Jan01 ?        1-08:17:30 ./kswapd0
test     30037     1  0 Jan01 ?        00:00:00 rsync

top command shows one of them is eating up my CPU power.

> top -U test
top - 09:35:51 up 687 days, 23:06,  2 users,  load average: 1.75, 1.45, 1.35
Tasks: 173 total,   3 running, 102 sleeping,   4 stopped,   0 zombie
%Cpu(s): 59.8 us,  1.5 sy,  0.0 ni, 36.0 id,  0.0 wa,  0.0 hi,  2.7 si,  0.0 st
KiB Mem :  8153496 total,   178612 free,  4117724 used,  3857160 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  3367876 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
24143 test      20   0 2433936   2.3g      4 S 100.0 29.4   1940:11 kswapd0
22810 test      20   0  104704  28620      4 S  25.9  0.4   4:55.28 blitz64
 3483 test      20   0   26220  17328   4052 R   0.0  0.2   0:19.70 tor
22642 test      20   0  113300   3096   2848 S   0.0  0.0   0:00.13 go
22804 test      20   0  112336    776    684 S   0.0  0.0   0:00.00 timeout
22805 test      20   0  113296   2912   2724 S   0.0  0.0   0:00.00 blitz
30037 test      20   0  136632   5908   2880 S   0.0  0.1   0:00.28 rsync

Solution

I am going to delete this account but I have to kill all processes launched by this user first.

pkill -9 -u test
userdel test

Without -9, pkill failed to kill process 30037 probably because it has intercepted the TERM signal.

Directory /home/test won't be deleted.

Conclusion

The lesson from this incident is that never use a weak password for a common account name.