Logs

Project Overview

Log files (also spelled logfiles), are files used by computers to log events. Software programs, background processes, services, or transactions between services, including the operating system itself, may generate
such events. Log files are dependent on the application that generates them. It is up to the application developer to conform to log file convention. Software documentation should include information on its log
files.
Because log files are essentially a way to track specific events, the type of information stored varies depending of the application or services generating the events.

(a) Consider the single log entry below. It was generated by Apache, a popular web server.
[Wed Mar 22 11:23:12.207022 2017] [core:error] [pid 3548:tid 4682351596] [client 209.165.200.230] File does not exist: /var/www/apache/htdocs/favicon.ico
The single log entry above represents a web event recorded by Apache. A few pieces of information are important in web transactions, including client IP address, time and details of the transaction. The entry
above can be broken down into five main parts:
Timestamp: This part records when the event took place. It is very important that the server clock is correctly synchronized as it allows for accurately cross-referencing and tracing back events.
Type: This is the type of event. In this case, it was an error.
PID: This contains information about the process ID used by Apache at the moment.
Client: This records the IP address of the requesting client.
Description: This contains a description of the event.
Question:
Based on the log entry above, describe what happened.

At 11:23:12 AM on March 22, 2017, the Apache web server recorded an error event.

Here’s what occurred step-by-step:

A client with IP address 209.165.200.230 sent a request to the web server.

The client was trying to access a file called favicon.ico.

Apache attempted to locate this file in the directory:

/var/www/apache/htdocs/favicon.ico

The file did not exist in that location.

Because the file was missing, Apache generated a core error and logged it.

In Simple Terms

A user (or browser) tried to load the website’s favicon (the small icon shown in a browser tab), but the server couldn’t find that file on the server. As a result, Apache logged an error saying the file does not exist.

(b) Using the cat command below to list a web server sample log file. The sample file is located at /var/log:

[analyst@secOps ~]$ cat /var/log/logstash-tutorial.log
83.149.9.216 — — [04/Jan/2015:05:13:42 +0000] “GET /presentations/logstash-monitorama-
2013/images/kibana-search.png HTTP/1.1” 200 203023
“http://semicomplete.com/presentations/logstash-monitorama-2013/" “Mozilla/5.0
(Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/32.0.1700.77 Safari/537.36”
83.149.9.216 — — [04/Jan/2015:05:13:42 +0000] “GET /presentations/logstash-monitorama-
2013/images/kibana-dashboard3.png HTTP/1.1” 200 171717
“http://semicomplete.com/presentations/logstash-monitorama-2013/" “Mozilla/5.0
(Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/32.0.1700.77 Safari/537.36”
83.149.9.216 — — [04/Jan/2015:05:13:44 +0000] “GET /presentations/logstash-monitorama-
2013/plugin/highlight/highlight.js HTTP/1.1” 200 26185
“http://semicomplete.com/presentations/logstash-monitorama-2013/" “Mozilla/5.0
(Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/32.0.1700.77 Safari/537.36”
<some output omitted>

Is the output above still considered a web transaction?

Yes, it is still a web transaction.

Even though the format looks different, each line represents a web request from a client to a web server, which makes it a web transaction.

For example, take this entry:

83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] 
"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 
200 203023 
"http://semicomplete.com/presentations/logstash-monitorama-2013/" 
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1)..."

This shows:

Client IP address → 83.149.9.216

Timestamp → 04/Jan/2015:05:13:42 +0000

HTTP method → GET

Requested resource → image file (kibana-search.png)

HTTP version → HTTP/1.1

Status code → 200 (success)

Bytes sent → 203023

Referrer URL

User-Agent (browser + OS info)

That is clearly a complete HTTP request/response exchange which is exactly what a web transaction is.

Explain why the output of the cat command is in a different format than the single entry shown in item (a)?

The output of the cat command differs in format from the single entry in item (a) because it comes from an Apache access log (which records all incoming requests, successful or not), whereas the entry in (a) is from an Apache error log (which specifically captures server-side errors or issues, like a missing file). Access logs typically follow the Combined Log Format (CLF), a standardized structure that includes fields like client IP, timestamp, HTTP request method and path, status code, response size, referrer URL, and user-agent string — all visible in these lines for tracking general web activity. In contrast, error logs use a more flexible, module-specific format focused on diagnostic details like error type, process/thread IDs, and a descriptive message, without the same emphasis on request specifics.

Operating system log file example
Any software can keep log files, including the operating system itself. Conventionally, Linux uses the /var/log directory to stores various log files, including operating system logs. Modern operating systems are complex
pieces of software and therefore use several different files to log events. This section takes a quick look at the
/var/log/messages file.
a. Stored under /var/log, the messages file stores various system events. The connection of new USB drive, a network card becoming available, and too many missed root login attempts, are a few examples of
events logged to the /var/log/messages file. Use the more command to display the contents of the /var/log/messages file. Unlike the cat command, more allows for a paced navigation through the file.
Press ENTER to advance line-by-line or SPACE to advance an entire page. Press q or CTRL + C to abort and exit more.
Note: the sudo command is required because the messages file belongs to the root user.
[analyst@secOps ~]$ sudo more /var/log/messages
[sudo] password for analyst:
Mar 20 08:34:38 secOps kernel: [6.149910] random: crng init done
Mar 20 08:34:40 secOps kernel: [8.280667] floppy0: no floppy controllers found
Mar 20 08:34:40 secOps kernel: [8.280724] work still pending
Mar 20 08:35:16 secOps kernel: [ 44.414695] hrtimer: interrupt took 5346452 ns
Mar 20 14:28:29 secOps kernel: [21239.566409] pcnet32 0000:00:03.0 enp0s3: link down
Mar 20 14:28:33 secOps kernel: [21243.404646] pcnet32 0000:00:03.0 enp0s3: link up,
100Mbps, full-duplex
Mar 20 14:28:35 secOps kernel: [21245.536961] pcnet32 0000:00:03.0 enp0s3: link down
Mar 20 14:28:43 secOps kernel: [21253.427459] pcnet32 0000:00:03.0 enp0s3: link up,
100Mbps, full-duplex
Mar 20 14:28:53 secOps kernel: [21263.449480] pcnet32 0000:00:03.0 enp0s3: link down
Mar 20 14:28:57 secOps kernel: [21267.500152] pcnet32 0000:00:03.0 enp0s3: link up,
100Mbps, full-duplex
Mar 20 14:29:01 secOps kernel: [21271.551499] pcnet32 0000:00:03.0 enp0s3: link down
Mar 20 14:29:05 secOps kernel: [21275.389707] pcnet32 0000:00:03.0 enp0s3: link up,
100Mbps, full-duplex
Mar 22 06:01:40 secOps kernel: [0.000000] Linux version 4.8.12–2-ARCH
(builduser@andyrtr) (gcc version 6.2.1 20160830 (GCC) ) #1 SMP PREEMPT Fri Dec 2
20:41:47 CET 2016
Mar 22 06:01:40 secOps kernel: [0.000000] x86/fpu: Supporting XSAVE feature 0x001:
‘x87 floating point registers’
Mar 22 06:01:40 secOps kernel: [0.000000] x86/fpu: Supporting XSAVE feature 0x002:
‘SSE registers’
Mar 22 06:01:40 secOps kernel: [0.000000] x86/fpu: Supporting XSAVE feature 0x004:
‘AVX registers’
Mar 22 06:01:40 secOps kernel: [0.000000] x86/fpu: xstate_offset[2]: 576,
xstate_sizes[2]: 256
Mar 22 06:01:40 secOps kernel: [0.000000] x86/fpu: Enabled xstate features 0x7,
context size is 832 bytes, using ‘standard’ format.
Mar 22 06:01:40 secOps kernel: [0.000000] x86/fpu: Using ‘eager’ FPU context switches.
<some output omitted>
Notice that the events listed above are very different from the web server events. Because the operating system itself is generating this log, all recorded events are in relation to the OS itself.
b. If necessary, enter Ctrl + C to exit out of the previous command.
c. Log files are very important for troubleshooting. Assume that a user of that specific system reported that all network operations were slow around 4:20 am on May 19.
Question:
Can you find evidence of that in the log entries shown above? If so, in what lines? Explain.

1. Locating Log Files in Unknown Systems

The CyberOps Workstation VM includes nginx, a lightweight web server. This section will show how to find and display nginx logs using the CyberOps Workstation VM.
With default settings, its global configuration file is located under /etc/nginx/nginx.conf, its access log file is at
/var/log/nginx/access.log, and errors are redirected to the terminal window. However, it is common for a security analyst to work on computers in which the installation details for tool and services are unknown. This section describes the process of locating such files described for nginx but is by no means complete. Nevertheless, it should be a good exercise about locating and displaying log files on unfamiliar systems.

a. When working with new software, the first step is to look at the documentation. It provides important information about the software, including information about its log files. Use the man command to display
the nginx manual page:
[analyst@secOps ~]$ man nginx

b. Scroll down the page to locate the nginx logging section. The documentation makes it clear that nginx supports logging, with the location of its log files defined at compilation time

c. The manual page also contains information on the files used by nginx. Scroll down further to display the nginx operating files under the Files section:

The outputs above help you to conclude that nginx supports logging and that it can save to log files. The output also hints at the existence of a configuration file for nginx.

d. Before looking for nginx files, use the ps and the grep commands to ensure nginx is running in the VM.
Note: Use man to learn more about ps and grep commands:

ps ax | grep nginx

The output above confirms that nginx is running. In addition, the output also displays the parameters used when nginx was started. nginx process ID is being stored in /run/nginx.pid and error messages are being
redirected to the terminal.
Note: If nginx is not running, enter the sudo /usr/sbin/nginx at the prompt to start the service using the default configuration.
Note: If you need to restart nginx, you can kill the service by using the sudo pkill nginx command. To start nginx with the custom configuration from a previous lab, run the following command: sudo nginx -c
custom_server.conf, and test the server by opening a web browser and going to URL: 127.0.0.1:8080. If you wish to start nginx with a default configuration you can start it with the command: sudo
/usr/sbin/nginx, and open a web browser and go to URL: 127.0.0.1.
Because the location to the log files was not specified, the global nginx configuration file should be checked for the location of the log files.

e. By design, the CyberOps Workstation VM utilizes default locations and definitions as much as possible.
Conventionally, the /var/log directory holds various log files for various applications and services while configuration files are stored under the /etc directory. While the nginx manual page did not provide an exact location for its log files, it not only confirmed that nginx supports logging but also hinted at the location of a configuration file. Because the log file locations can often be customized in configuration files, a logical next step is to use the ls command to look under /etc and look for a nginx configuration file:

ls /etc/

f. Notice the nginx folder under /etc in the output above. Using ls again, we find a number of files, including one named nginx.conf.

ls -l /etc/nginx/

g. Use the cat command to list the contents of /etc/nginx/nginx.conf. You can also use more or less to view the file and nano or SciTE to edit it. These tools make it easier to navigate through long text files (only the output of cat is displayed below).

cat /etc/nginx/nginx.conf

h. A quick look at the configuration file reveals that it is an nginx configuration file. Because there is no direct mention to the location of nginx log files, it is very likely that nginx is using default values for it. Following the convention of storing log files under /var/log, use the ls command to list its contents

ls -l /var/log/

i. As shown above, the /var/log directory has a subdirectory named nginx. Use the ls command again to list the contents of /var/log/nginx.
Note: Because the /var/log/nginx belongs to the http user, you must execute ls as root by preceding it with the sudo command.

sudo ls -l /var/log/nginx

These are very likely to be the log files in use by nginx. Move on to the next section to monitor these files
and get confirmation that they are indeed nginx log files.

2. Monitoring Log Files in Real Time

As seen in the previous sections, log files can be displayed with many text-presentation tools. While cat, more, less, and nano can be used to work with log files, they are not suitable for log file real-time monitoring.
Developers designed various tools that allow for log file real-time monitoring. Some tools are text-based while others have a graphical interface. This lab focuses on tail, a simple but efficient tool, available in practically every Unix-based system.
The CyberOps Workstation VM uses a log rotating system to ensure that older logs are archived. By the time this lab gets used in class, some time will have passed and the log files will likely have been rotated. The
result is that some log files, including the access.log file, could appear empty. To work around this problem and create some entries in access.log, simply open Firefox in the VM, point it to 127.0.0.1 and reload the
page a few times.

Get Elijah Chimera’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Step 1: Using the tail command
The tail command displays the end of a text file. By default, tail will display the last ten (10) lines of a text file.
Note: If you do not see any log entries, navigate to 127.0.0.1 in a web browser and refresh the page a few time.
a. Use the tail command to display the end of the /var/log/nginx/access.log.

sudo tail /var/log/nginx/access.log

If you do not see any log entries, navigate to 127.0.0.1 in a web browser and refresh the page a few time.

b. Use the –n option to specify how many lines from the end of a file, tail should display.

sudo tail -n 5 /var/log/nginx/access.log

c. You can use the tail command with the -f option to monitor the nginx access.log in real-time. Short for follow, -f tells tail to continuously display the end of a given text file. In a terminal window, issue tail with the –f option:

sudo tail -f /var/log/nginx/access.log

As before, tail displays the last 10 lines of the file. However, notice that tail does not exit after displaying the lines; the command prompt is not visible, indicating that tail is still running.
Note: Your /var/log/access.log file may be empty due to log rotation. Continue following the lab as an empty /var/log/access.log file will not impact the lab.

d. With tail still running on the terminal window, click the web browser icon on the Dock to open a web browser window. Re-size the web browser window in a way that it allows you to see the bottom of the
terminal window where tail is still running.
Note: In the screenshot below, the Enter key was pressed a few times in the terminal window running tail. This is for visualization only as tail does not process any input while running with –f. The extra empty lines make it easier to detect new entries, as they are displayed at the bottom of the terminal window.

e. In the web browser address bar, enter 127.0.0.1 and press Enter. This is the address of the VM itself, which tells the browser to connect to a web server running on the local computer. A new entry should be recorded in the /var/log/nginx/access.log file. Refresh the webpage to see new entries added to the log.

127.0.0.1 — — [23/Mar/2017:09:48:36 -0400] “GET / HTTP/1.1” 200 612 “-” “Mozilla/5.0 (X11; Linux i686; rv:50.0) Gecko/20100101 Firefox/50.0” Because tail is still running, it should display the new entry at the bottom of the terminal window. Aside from the timestamp, your entry should look like the one above

Note: Firefox stores pages in cache for future use. If a page is already in cache, force Firefox to ignore the cache and place web requests, reload the page by pressing <CTRL+SHIFT+R>.

f. Because the log file is being updated by nginx, we can state with certainty that /var/log/acess.log is in fact the log file in use by nginx.

g. Enter Ctrl + C to end the tail monitoring session.

Step 2: BONUS TOOL: Journalctl

The CyberOps Workstation VM is based on Arch Linux. Categorized as a Linux distribution, Arch Linux is designed to be lightweight, minimalist and simple. As part of this design philosophy, Arch Linux uses systemd
as its init system. In Linux, the init process is the first process loaded when the computer boots. Init is directly or indirectly, the parent of all processes running on the system. It is started by the kernel at boot time and continues to run until the computer shuts down. Typically, init has the process ID 1.

An init system is a set of rules and conventions governing the way the user space in a given Linux system is created and made available to the user. Init systems also specify system-wide parameters such as global
configuration files, logging structure and service management.

Systemd is a modern init system designed to unify Linux configuration and service behavior across all Linux distributions and has been increasingly adopted by major Linux distributions. Arch Linux relies on systemd for init functionality. The CyberOps Workstation VM also uses systemd.
system-journald (or simply journald) is systemd’s event logging service and uses append-only binary files serving as its log files. Notice that journald does not impede the use of other logging systems, such as syslog and rsyslog.
This section provides a brief overview of journalctl, a journald utility used for log viewing and real-time monitoring.

a. In a terminal window in the CyberOps Workstation VM, issue the journalctl command with no options to display all journal log entries (it can be quite long):

journalctl

journalctl includes a number of functionalities such as page scrolling, color-coded messages and more.
Use the keyboard up/down arrow keys to scroll up/down the output, one line at a time. Use the left/right keyboard arrow keys to scroll sideways and display log entries that span beyond the boundaries of the
terminal window. The <ENTER> key displays the next line while the space bar displays the next page in the output. Press the q key to exit journalctl.
Notice the hint message provided by journalctl:
Hint: You are currently not seeing messages from other users and the system.
Users in groups ‘adm’, ‘systemd-journal’, ‘wheel’ can see all messages.
Pass -q to turn off this notice.
This message reminds you that, because analyst is a regular user and not a member of either the adm, systemd-journal or wheel groups, not all log entries will be displayed by journalctl. It also states that running journalctl with the –q option suppresses the hint message.

How can you run journalctl and see all log entries?

One can use the following commands:

# This runs journalctl with root privileges and displays all system and user logs:
sudo journalctl


# For permanent access add the user to the appropriate group:
sudo usermod -aG systemd-journal analyst

#If you also want to remove the hint message:
sudo journalctl -q

b. journalctl includes options to help in filtering the output. Use the –b option to display boot-related log entries:

sudo journalctl -b

c. To see entries related to the last boot, add the -1 to the command above. To see entries related to the two last boots, add the -2 option.

sudo journalctl -b -2

d. Use the — list-boots option to list previous boots:

sudo journalctl — list-boots

e. Use the — since “<time range>” to specify the time range of which log entries should be displayed. The two commands below display all log entries generated in the last two hours and in the last day, respectively:

sudo journalctl --since "2 hours ago"

sudo journalctl --since "1 day ago"

f. journalctl also allows for displaying log entries related to a specific service with the –u option. The command below displays logs entries related to nginx:

sudo journalctl -u nginx.service

Note: As part of systemd, services are described as units. Most service installation packages create units and enable units during the installation process.

Similar to tail –f, journalctl also supports real-time monitoring. Use the –f option to instruct journalctl to follow a specific log. Press Ctrl + C to exit.

sudo journalctl -f

h. journalctl also supports mixing options to achieve the desired filter set. The command below monitors nginx system events in real time.

sudo journalctl -u nginx.service -f

i. Keep the command above running, open a new web browser window and type 127.0.0.1 (default configuration) or 127.0.0.1:8080 (custom_server.conf) in the address bar. journalctl should display an error related to a missing favicon.ico file in real-time. Use Ctrl+C to quit journalctl.

Reflection
Log files are extremely important for troubleshooting.
Log file location follows convention but ultimately, it is a choice of the developer.
More often than not, log file information (location, file names, etc.) is included in the documentation. If the documentation does not provide useful information on log files, a combination of web research, and system
investigation should be used.
Clocks should always be synchronized to ensure all systems have the correct time. If clocks are not correctly set, it is very difficult to trace back events.
It is important to understand when specific events took place. In addition to that, events from different sources are often analyzed at the same time.