Story of a Linux server malware infection

On May 2016 I was asked to check a Linux Server that seemed to be functioning slowly and because of an alert from the communication services of the University that warned that the server could be serving malware and/or SPAM. This a report of what I found.

FORENSIC ANALYSIS REPORT

• Date: 27th of May 2016
• System name: SXXXXXXXa
• Operative System: Ubuntu 14.04 LTS
• Installation date: 12th of April, 2016
• Usage of the System: Server used at the XXXXXXXXX department of the University of XXXXX for their daily operation. It's connected to the internal network of the University and at the same time it's directly accessible from internet through university's subdomain. XXXX.XXXXX.es (IP=193.XXX.XXX.X9). The system has, at the moment of the incident a SSH server accesible from internet. It does not offer any web site nor other available services.
• Event:The department was informed from the communication services of the University that this server has been detected as the origin of malicious traffic and that probably it has a trojan running. To avoid further problem, the traffic was cut off from and to internet for the server.

Timeline of the problem

After accessing physically to the server the following facts were detected:

1. The command 'ps' did not show any strange process running. The command netstat did not show any strange connection.
2. In /root directory of the system, I found a file named XXXX.nano modified on April 21st, 2016 approximately at 12:00 AM in which I could see the line “PermitRootLogin yes”. This is a line for the SSH config file that allows SSH server to be accessed with root user. Checking the SSH config file, I can see it has the same date and time than the .nano file and therefore with that date, the system was allowed to be accessed by root. From my point of view this was not actually part of the attack but a very bad practice used by administrator some days after the system installation and days before the sttack started.
3. That same day, and approximately same hour, it appears an SSH connection with a normal user (I'll use the fake username 'chris' to refer to it), and 5 minutes later an SSH connection with root user.
4. Four days later we find that on April 25th, 09:03, an SSH connection with user 'chris' is started, but this time the connection comes from IP 183.3.202.187 (located in China).
5. In command history of root user we find a total of 13 commands (From 71 to 83 in next image), where we can see that a file is downloaded from a server in China (IP 117.18.4.133), and it's give execution permissions and it's eventually executed... Finally on line 83 the root password is changed.

Unfortunately we cannot know the exact date and time of these commands, but we know that they happened before an SSH connection found on history line 86 (not shown) that happened on April 25th at 12:00 AM. So it seems the attack occurred on April 25th 9:03 and the server was owned a few minutes later.

Post Infection

The system was installed on April 12th, from the day after installation (April 13th, 2016) we can see SSH root Access attempts (which is normal as it had its own public IP address). Actually until the real attack, 42893 SSH connection attempts were found and in the whole life of the server more tan 131000 SSH Access attempts were found:

From IP 183.3.202.187 we can see five failed SSH connection attempts on April 25th at 09:03. The fifth attempt was the access with user 'chris' reported previously on point 3. From that moment on hundreds of connections from that IP were done (As seen in previous image). When asked user 'chris', he/she told me that the password was identical to the username. So the attacker simply tried a number of users with the same passwords and finally he could Access the system. Apparently user chris had sudo permissions and the attacked could easily Access the system as root.

After being owned the server was accessed successfully 61 times from 7 different IP addresses. This is a summary of the connections:

• April 25th, 09:03. With user root from IP 183.3.202.187. This same day another 16 SSH connections were done with user root from IPs: 183.3.202.187, 185.103.252.14, 110.10.129.201 and 89.248.167.131.
• The next day, April 26th a total of 15 connections are found from IPs 183.3.202.187 and 110.10.129.201, in these connections the history shows that a package named FakeRoot is installed.
• The next day, April 27th, another 16 root connections are done from IPs 183.3.202.187, 222.186.21.100 and 110.10.129.201.
• The next day, April 28th, another 12 root connections from IPs: 183.3.202.187 and 100.43.129.140.

From that day no root connections are done. Actually no new connections are performed from strange IPs.

Location of these IPs

• 183.3.202.187 ==> Guangzhou, Guangdong, China.
• 185.103.252.14 ==> Moscu, Russia. .
• 110.10.129.201 ==> Seoul, South Korea.
• 89.248.167.131 ==> ¿?, The netherlands.
• 222.186.21.100 ==> Nanjing, Jiangsu Sheng, China.
• 100.43.129.140 ==> Orange, California, USA.

All but the last one had been previously reported as malicious IPs.

Actions performed by attacker in the system

1. In the directory /root the following malicious files were found:
   • 2016ttfacai, ELF executable, 1223123 bytes, with date April, 2nd.
   • heifacai, ELF executable, 1223123 bytes. With date April 3rd, but with the same MD5 hash, so it seems a copy of the former file.
   • conf.n, 73 bytes. Apparently a config file for the previous executables, although it's a binary file.
   

   All three files were sent to virustotal. The report of the two first executablescan be seen in this link.

2. In /etc/init.d a script file named DbSecuritySpt was found that actually executed /root/heifacai previously mentioned. This script was executed at every system restart therefore ensuring its presence in memory.

3. In /etc/init.d another two files kzgbqvo and mowepretrf were found with similar contents.
4. In /bin at least three different commands 'ps', 'ss' and 'netstat' were found with modification dates between 25th and 28th of April. When running 'ps' no strange process where found and when running 'netstat' no strange connections where found, so it seems that the have the been substituted by a modified copy of the executable. When removed these files, they automatically appeared again:
   

   After downloading the 'ps' and 'netstat' executables from internet, the malicious heifacai process appeared.

5. Other malicious files were also found in the /bin directory (ovqbbgzk, kzgbbqvo, kzgbbqvo.sh)

Measures taken

Although eventually the system seemed to be cleaned, a recommendation of formatting and reinstalling the system was given to the owner. A speech of cybersecurity concientization was given to the users and in the newly installed system secure password policies were included.