Linux Server Forensics
Learn about digital forensics artefacts found on Linux servers by analysing a compromised server
Task 1: Deploy the first VM
You have been hired to investigate a data breach at ACME web design. Log in to the target machine using SSH with the following credentials:
- 10.49.129.60
- Username — ‘fred’
- Password — ‘FredRules!’
Task 2: Apache Log Analysis I
The most significant attack surface on the server is probably the web service; fortunately, the Apache access log keeps a history of all of the requests sent to the webserver and includes:
1. source address
2. response code and length
3. user-agent
Every request from a modern browser should contain a user-agent string that can be used to identify it roughly. The content of the user-agent string is then used to modify the experience by displaying the mobile version of a site or disabling features not supported on older browsers.
We can also use this string to identify traffic from potentially malicious tools as scanning tools like Nmap, SQLmap, DirBuster and Nikto leak their identity’s through default user-agent strings.
Once logged-in, Navigate to /var/log/apache2
Answer the questions below:
Q1: How many different tools made requests to the server?
We will look at access.log file as it stored logs related to HTTP requests processed by the web server. I used cut -d ' ' -f 12 to cut specific portion of the log file, sort to sort based on ascending order, uniq -c to print unique values. We won’t count Apache or Mozilla as they are legitimate services.
Answer: 2
Q2: Name a path requested by Nmap.
Use grep nmap to see logs related to nmap, one GET request stands out.
Answer: /nmaplowercheck1618912425
Task 3: Web Server Analysis
Web scanners are run against servers pretty much all the time, so this traffic is not out of the ordinary.
Have a look around the site for potential attack vectors.
Answer the below questions:
Q1: What page allows users to upload files?
Navigate to ~ home directory of the user, we see there is a directory named Templates having multiple html files. I decided to manually look at all the files to see which one allows for file uploading.
This php code block in contact.html is checking for file uploads.
Answer: contact.html
Q2: What IP uploaded files to the server?
We will once again analyze /var/log/apache2/access.log because the attacker interacted with the web server by uploading a file/files. We can filter for contact.html using cat access.log | grep contact.html. We see aHEAD request for /contact.php and we can then filter for POST requests we see the same IP.
Answer: 192.168.56.24
Q3: Who left an exposed security notice on the server?
Filter the logs and look for security we find security.html being enumerated which might have contained sensitive information, left by the user fred indicated by first -
Or we can look for /resources/development/2021/docs/SECURITY.md in browser
Answer: Fred
Task 4: Persistence Mechanisms I
It looks like our attacker got in via Remote File Inclusion (RFI). It’s best to look around the system itself now for any evidence of persistence mechanisms that could lead to a payload. There are multiple ways to maintain persistence in most Linux distributions including but not limited to:
- cron
- Services/systemd
- bashrc
- Kernel modules
- SSH keys
Cron is one of the more common methodologies, so it’s probably best to start there.
Answer the question below:
Q1: What command and option did the attacker use to establish a backdoor?
Checking the contents of crontab take a look at the last line, five asterisks this cronjob is scheduled to run (every minute, every hour, everyday,every month and every day of the week) to spawn an interactive shell and let the attacker create a Reverse Shell.
Answer: sh -i
Task 5: User Accounts
It looks like the command from the previous task was set up to run under the root2 account. This account doesn’t make any sense as there should only be one root account. Wonder how it got there?
Get WajidMir’s stories in your inbox
Join Medium for free to get updates from this writer.
There are a couple of locations where account information is stored on Linux distributions, including:
- /etc/passwd — contains the names of most of the accounts on the system. Should have open read permissions and should not contain password hashes.
- /etc/shadow — contains names but should also contain password hashes. Should have strict permissions.
Answer the questions below
Q1: What is the password of the second root account?
I pasted the WVLY0mgH0RtUI string in the browser and came across this article: https://materials.rangeforce.com/tutorial/2019/11/07/Linux-PrivEsc-SUID-Bit/
Answer: mrcake
Task 6: Deploy the second VM
Wow, it looks like they got hacked again! Better have another look; the credentials are the same as last time:
- MACHINE_IP
- ‘fred’
- ‘FredRules!’
Task 7: Apache Log Analysis II
The log file is a lot smaller this time around, so it looks like our attacker was a little more subtle. There also don’t appear to be obvious user agents anymore. Fortunately, there are a few other ways of identifying traffic originating from scanners. The time between each request is a good metric for most tools. You can also identify individual tools from signatures left in the requests; for example, Nmap will send HTTP requests with a random non-standard method when performing certain enumeration tasks. More aggressive tools can also be identified simply from the number of requests sent during any given attack; directory brute-forcing tools are a perfect example of this and are likely to fall foul of banning systems like fail2ban.
A poorly designed site may also freely grant valuable information without the need for aggressive tools. In this case, the site uses sequential IDs for all of the products making. It easily scrapes every single product or finds the total size of the product database by simply increasing the product ID until a 404 error occurs.
Q1: Name one of the non-standard HTTP Requests
I had some problems using grep, decided to use cut combined with sort and uniq.
GXWR looked out of place.
Answer: GXWR
Q2: At what time was the Nmap scan performed? (format: HH:MM:SS)
We can find answer to this in the same GXWR request.
Answer: 13:30:15
Task 8: Persistence Mechanisms II
The adversary has been a little smarter this time around. If any backdoor exists, it’s likely to be more subtle. SSH-keys are another excellent way of maintaining access, so it might be worth looking for additions to the authorized_keys file.
Answer the questions below
Q1: What username and hostname combination can be found in one of the authorized_keys files? (format: username@hostname)
Login as root, then navigate to /root directory, ls -la to list everything even hidden directories or files. There is .ssh directory, inside is authorized_keys
Answer: kali@kali
Task 9: Program Execution History
Of course, adding a public key to root’s authorized_keys requires root-level privileges so, it may be best to look for more evidence of privilege escalation. In general, Linux stores a tiny amount of programme execution history when compared to Windows but, there are still a few valuable sources, including:
- bash_history — Contains a history of commands run in bash; this file is well known, easy to edit and sometimes disabled by default.
- auth.log — Contains a history of all commands run using sudo.
- history.log (apt) — Contains a history of all tasks performed using apt — is useful for tracking programme installation and removal.
systemd services also keep logs in the journald system; these logs are kept in a binary format and have to be read by a utility like journalctl. This binary format comes with some advantages; however, each journal is capable of validating itself and is harder to modify.
Answer the questions below
Q1: What is the first command present in root’s bash_history file?
Simply check the .bash_history file in root directory
Answer: nano /etc/passwd
Task 10: Deploy The Final VM
This is getting a little annoying now. The credentials are still the same:
- MACHINE_IP
- ‘fred’
- ‘FredRules!’
Task 11: Persistence Mechanisms III
Malware can also maintain persistence using systemd as scripts run under systemd can run in the background and restart whenever the system is booted or whenever the script crashes. It is also relatively easy to conceal malicious scripts as they can blend in with other services. systemd services are defined in .service files which can contain:
- The command that runs whenever the service starts
- The user the service runs as
- An optional description
In this case, the malware is pretty obvious as it seems to be printing errors to the screen so, there is no way that it is dormant. Running systemctl will list all of the services loaded into the system. And much like Windows, there's usually a lot of them. It might be worth adding --type=service --state=active to the command as it will reduce the list to services that are running. Once the name of a suspicious service is found, more information can then be extracted by running systemctl status <service name> .
Answer the questions below
Q1: Figure out what’s going on and find the flag.
Use systemctl status IpManager we notice a few things such as Main PID is usually written in C or Go, bash script indicates persistence mechanism sleep 10 script waits 10 seconds and then re-opens the connection.
We can check /etc/network/ZGtsam5hZG1ua2Fu.sh
Answer: [gh0st_1n_the_machine]
Give a like. Follow for more
