Holy Cow, Batman! My Linux Server Got Hacked!

It was quiet, peaceful Tuesday night when I attempted to log into my Ubuntu Linux server as the root user. Permission denied, I got back. I tried it again. And again. Access denied. What the heck? I’m sure it was the correct password. I then tried all the passwords that it could be, from past to present. No luck. I got a sinking feeling. My root password was easy to remember and thus, easy to guess. I think my Linux server has been hacked!

No problem, I thought, I just have to reboot and log into single user mode… uh, unfortunately, my Linux server was sitting somewhere in cyberspace and I couldn’t do any of that. So I called up the service provider and they said, we didn’t touch your server and we don’t know who did. However, for $25, they can reset the root password. Having no choice, I accepted. Hours passed and finally I got an email with the new password.

I logged in, viewed the “/etc/passwd” file, and noticed two new users. Bad news, my server definitely got hacked. Good news, the hackers weren’t too sophisticated and had left a huge clue. I immediately copy the users’ info down and deleted them. Then it was off to google for what to do after a server hack. After looking at several websites, the most relevant one I found was How to restore a hacked Linux server.

What I got from the research was that once your Linux server is compromised, the best solution is to re-install the operating system. I did not want to do that because it would cost (a) money to ask the service provider to reinstall Ubuntu and (b) time for me to reinstall the applications. So I decided to play detective, find any damages, and fix them. Below are the steps I took.

1. Are there any unidentified users?

Run the following to list the users and groups:

Make a copy of the unknown users’ info, especially their usernames and ids, and delete them using the “userdel <unknown_username>” command. You should also check for and delete their home directories, located at “/home/username”.

2. Are there any strange processes running?

Run the following to list all the active processes started by the root or by the unidentified users:

Look for processes with weird names like “AkjwetSAG”; these are usually suspect. You can then run “lsof -p <process_id>” to get more info.

On my Ubuntu server, I found two processes, “avahi-daemon: chroot helper” and “/usr/sbin/console-kit-daemon –no-daemon”, that looked weird; but it turned out they were valid processes. After some quick online searches, I found that Avahi was used to detect network devices (not necessary for a server so it can be disabled or deleted) and console-kit was used for GUI related functions.

3. Are there any tracks left by the perpetrators?

Run the following to find any commands to change a password in the system:

This command returned several entries in the “auth.log” which included password changes for the unidentified user accounts. Strangely, there was no reference to a password change for the root user. By luck, there were two “Invalid user passwd from <ip_address>” messages containing the IP addresses of the perpetrators.

I then did a search on each of those two IP addresses:

I got back a ton of entries in “auth.log” which show numerous failed attempts using different usernames from these two IP addresses. I guess they kept trying until they got lucky. And they got lucky about five days ago.

Just for curiosity’s sake, I installed traceroute (it isn’t installed on Ubuntu by default) to see where the perpetrators came from:

One of the IP address came from “CHINA-NETCO.edge3.SanJose1.Level3.net” and the other from “DC-PT-01-te1-2.ip4.012.net.il”. China and Israel. But there was no guarantee that those were the origins because the perpetrators could have hacked some machines there and then used those machines to hack into my server.

4. Are there any surprises left behind?

To be on the safer side, I installed and ran two rootkit scanners recommended by the website I found:

Both reported a clean bill of health. There were one or two false positives which I investigated and cleared. Finally, some good news! It looks like the hackers didn’t do anything damaging (cross my fingers).

5. Any strange new files?

One of the recommended actions was to look for new files that were created since the break-in. But I didn’t find this useful at all; it was like looking for a needle in a haystack. (Funny, in real life, robbers take stuff. In cyberspace, they leave stuff… malicious stuff.)

On Ubuntu, I couldn’t find a way to search for newly created files since 5 days ago. So I did the next best thing, which was to search for files last modified in the past 5 days.

Unfortunately, the service provider had rebooted the machine as part of the password reset. So I got a ton of results back. Too much info to process.

6. What can I do to prepare for the next assault?

First, secure the root access. I changed the root password to an incomprehensible, random string of numbers, letters (lower and upper case), and special symbols. Try guessing that! And I will only share this root password with one other person as a backup.

Second, enable auditing of the server so we will have a record of every command issued by any user. This will make it easier to figure out what actions any user, legitimate or not, does on the server. I installed the Process Accounting (psacct or acct) service which this website recommended.

By default, the service is configured to start on bootup. To double-check, I tried using the “update-rc.d acct defaults” command but got back a strange warning “update-rc.d: warning: acct stop runlevel arguments (0 1 6) do not match LSB Default-Stop values (1)”. I didn’t know what it meant so decided to use an alternative check:

This command returned “acct on”, meaning that the Process Accounting service was configured to start on bootup.

Hopefully the above info will help you should you ever encounter the same misfortune. And maybe, the last section might help you to be better prepared.