Sitemap
Ankit Juneja

Ankit Juneja

BASIC linux commands.

13 min readJul 2, 2023

--

  1. awk: Awk is a powerful text processing tool that allows you to extract and manipulate data based on patterns. Here's an example:
bashCopy code
awk '{ print $1 }' <filename>

This command will print the first column of data from each line in the file.

  1. sort: Sort is used to sort the lines of a text file. Here's an example:
bashCopy code
sort -k 2 -n <filename>

This command will sort the lines based on the second column in numerical order.

  1. cut: Cut is used to extract specific columns from a file. Here's an example:
bashCopy code
cut -d ',' -f 1,3 <filename>

This command will extract the first and third columns from a comma-separated file.

  1. sed: Sed is a stream editor used for filtering and transforming text. Here's an example:
bashCopy code
sed 's/foo/bar/g' <filename>

This command will replace all occurrences of “foo” with “bar” in the file

Head:

The head command in Linux is used to display the beginning or the top lines of a text file or input provided through a pipeline. By default, it shows the first 10 lines of the file, but you can specify a different number of lines if needed.

The basic usage of the head command is as follows:

cssCopy code
head [options] [file]
  • options: You can specify various options to control the behavior of the head command, such as the number of lines to display.
  • file: This is the name of the file you want to display the beginning of. If you don't specify a file, head will read from standard input (e.g., when used in a pipeline).

For example, to display the first 10 lines of a file named “example.txt,” you can use the following command:

bashCopy code
head example.txt

If you want to display a different number of lines, you can use the -n option, like this:

bashCopy code
head -n 20 example.txt

This command would display the first 20 lines of the “example.txt” file.

Regenerate

How to fetch 10th line of file in unix?

head command:

head -n 10 <file_path> | tail -n 1
tail -n +10 <file_path> | head -n 1

less= tail command

Explanation: The tail command with -n +10 displays all lines starting from the 10th line to the end. Then, the head -n 1 command displays the first line from the output of tail, which corresponds to the 10th line of the file.Using less command:

less +10 <file_path> | sed -n '1p'

Explanation: The less command opens the file for viewing, starting from the 10th line. The sed command with '1p' prints only the first line of the displayed content, which is the 10th line of the file.

Using sed command:

sed -n '10p' <file_path>

Explanation: The sed command is used for stream editing. With the -n option, it suppresses automatic printing, and '10p' specifies to print only the 10th line.

  1. Using awk command
awk 'NR==10' <file_path>

Explanation: The awk command processes files line by line. 'NR==10' is a condition that matches the 10th line and prints it.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -

To generate random data with 4 rows and 4 columns, where each column has a header, you can use various programming languages or tools. Here’s an example using Python with the Pandas library:

pythonCopy code
import pandas as pd
import numpy as np
# Generate random data
data = np.random.rand(4, 4)
# Create column headers
headers = ['Column1', 'Column2', 'Column3', 'Column4']
# Create a DataFrame with data and headers
df = pd.DataFrame(data, columns=headers)
# Print the DataFrame
print(df)

This code snippet uses the numpy library to generate random data with a shape of (4, 4), representing 4 rows and 4 columns. It then creates a list of column headers. Finally, it creates a Pandas DataFrame using the generated data and column headers, and prints the DataFrame.

The output will be a 4x4 DataFrame with randomly generated values and column headers like this:

markdownCopy code
Column1   Column2   Column3   Column4
0 0.136073 0.556845 0.100583 0.912385
1 0.366551 0.498202 0.858379 0.203587
2 0.063897 0.129738 0.469038 0.718652
3 0.704232 0.452885 0.636814 0.303469

You can modify the code to suit your specific requirements, such as changing the number of rows and columns or customizing the column headers.

suppose for the above data the file name is file1………….what will it extract if i use this linux command………………..awk ‘{print $2}’ file1

If you use the following Linux command:

bashCopy code
awk '{print $2}' file1

It will extract the second column from the file named file1. The awk command processes files line by line and separates each line into fields based on whitespace (by default). The print $2 statement specifies to print the second field/column of each line.

Assuming your file file1 contains the following data:

Copy code
Column1   Column2   Column3   Column4
0.136073 0.556845 0.100583 0.912385
0.366551 0.498202 0.858379 0.203587
0.063897 0.129738 0.469038 0.718652
0.704232 0.452885 0.636814 0.303469

Running the awk '{print $2}' file1 command will output:

Copy code
Column2
0.556845
0.498202
0.129738
0.452885

The command prints the second column header Column2 followed by the second column values from each line of the file.

what is awk specially used for in linux

Awk is a versatile and powerful command-line tool in Linux used for text processing. It allows you to search, filter, manipulate, and extract information from text files based on patterns and actions specified in the Awk language.

Here are some common use cases for Awk:

  1. Text processing: Awk is often used to process structured or delimited text files, such as CSV files. It can split input lines into fields, perform calculations, and transform the data based on specified conditions.
  2. Pattern matching: Awk excels at pattern matching and can search for specific patterns or regular expressions in text files. It allows you to define rules or conditions to identify and process lines that match the specified patterns.
  3. Data extraction and reporting: With Awk, you can extract specific columns, fields, or sections of data from files. It provides flexibility in selecting and printing specific information, making it useful for generating reports or extracting specific data points.
  4. Data manipulation and transformation: Awk supports a wide range of built-in functions and operators for performing mathematical calculations, string manipulations, and conditional operations. This allows you to transform and manipulate data within the Awk script.
  5. One-liner scripts: Awk is well-suited for writing concise, powerful one-liner scripts directly on the command line. It enables you to perform complex text processing tasks without the need for writing a separate script file.

Overall, Awk is a valuable tool for text processing, data extraction, and manipulation in Linux. Its simplicity, expressive syntax, and powerful capabilities make it a popular choice for various tasks involving structured text data.

awk ‘{ print NR, $0}’ file1

The command awk '{ print NR, $0 }' file1 will print each line of the file file1 preceded by the line number.

Assuming your file file1 contains the following data:

Copy code
Column1   Column2   Column3   Column4
0.136073 0.556845 0.100583 0.912385
0.366551 0.498202 0.858379 0.203587
0.063897 0.129738 0.469038 0.718652
0.704232 0.452885 0.636814 0.303469

Running the awk '{ print NR, $0 }' file1 command will output:

Copy code
1 Column1   Column2   Column3   Column4
2 0.136073 0.556845 0.100583 0.912385
3 0.366551 0.498202 0.858379 0.203587
4 0.063897 0.129738 0.469038 0.718652
5 0.704232 0.452885 0.636814 0.303469

The command uses the print statement in Awk to output two values: NR (the line number) and $0 (the entire line). By specifying NR and $0 inside the print statement separated by a comma, Awk will print the line number followed by the entire line for each line in the file.

awk ‘{ print NR, “:” $0}’ file1

The command awk '{ print NR, ":" $0 }' file1 will print each line of the file file1 preceded by the line number and a colon.

Assuming your file file1 contains the following data:

Copy code
Column1   Column2   Column3   Column4
0.136073 0.556845 0.100583 0.912385
0.366551 0.498202 0.858379 0.203587
0.063897 0.129738 0.469038 0.718652
0.704232 0.452885 0.636814 0.303469

Running the awk '{ print NR, ":" $0 }' file1 command will output:

yamlCopy code
1 : Column1   Column2   Column3   Column4
2 : 0.136073 0.556845 0.100583 0.912385
3 : 0.366551 0.498202 0.858379 0.203587
4 : 0.063897 0.129738 0.469038 0.718652
5 : 0.704232 0.452885 0.636814 0.303469

The command uses the print statement in Awk to output two values: NR (the line number) and ":" $0 (a colon followed by the entire line). By concatenating ":" with $0 inside the print statement, Awk will print the line number followed by a colon and the entire line for each line in the file.

generate a csv data for 5 colums and 5 rows of type integegr and string with number in the range of 1 to 10

To generate a CSV data file with 5 columns and 5 rows containing integer and string values in the range of 1 to 10, you can use Python. Here’s an example code snippet:

pythonCopy code
import csv
import random
# Define the column headers
headers = ['Column1', 'Column2', 'Column3', 'Column4', 'Column5']
# Generate random data
data = []
for _ in range(5):
row = []
for _ in range(5):
if random.random() < 0.5: # Randomly choose between integer and string
value = random.randint(1, 10) # Generate random integer between 1 and 10
else:
value = ''.join(random.choice('abcdefghijklmnopqrstuvwxyz') for _ in range(5)) # Generate random string
row.append(value)
data.append(row)
# Write data to CSV file
with open('data.csv', 'w', newline='') as csvfile:
writer = csv.writer(csvfile)
writer.writerow(headers) # Write column headers
writer.writerows(data) # Write data rows

Running this code will generate a CSV file named data.csv with 5 columns and 5 rows of randomly generated integer and string values in the range of 1 to 10.

Get Ankit Juneja’s stories in your inbox

Join Medium for free to get updates from this writer.

Here’s an example of the generated CSV file:

Copy code
Column1,Column2,Column3,Column4,Column5
8,abcde,2,6,klmno
4,ghijk,1,9,pqrst
3,lmnop,7,2,fghij
6,qwxyz,5,10,uvwxy
9,defgh,8,5,mnopq

The column headers are listed in the first row, followed by the randomly generated data in subsequent rows.

do it without using random function…gve a specific input

Certainly! Here’s an example of generating a CSV file with 5 columns and 5 rows of specific integer and string values:

pythonCopy code
import csv
# Define the column headers
headers = ['Column1', 'Column2', 'Column3', 'Column4', 'Column5']
# Define the data
data = [
[1, 'apple', 3, 'orange', 5],
[2, 'banana', 4, 'grape', 6],
[3, 'cherry', 5, 'mango', 7],
[4, 'pear', 6, 'kiwi', 8],
[5, 'watermelon', 7, 'pineapple', 9]
]
# Write data to CSV file
with open('data.csv', 'w', newline='') as csvfile:
writer = csv.writer(csvfile)
writer.writerow(headers) # Write column headers
writer.writerows(data) # Write data rows

In this example, we explicitly define the data in a 2D list data, where each inner list represents a row of values. The values consist of specific integers and strings.

Running this code will generate a CSV file named data.csv with the specified data.

Here’s an example of the generated CSV file:

Copy code
Column1,Column2,Column3,Column4,Column5
1,apple,3,orange,5
2,banana,4,grape,6
3,cherry,5,mango,7
4,pear,6,kiwi,8
5,watermelon,7,pineapple,9

The column headers are listed in the first row, followed by the specified data in subsequent rows.

suppose file 2 is the file name for csv ouput …….awk -F , ‘{ print $4}’ file2

If file2 is the name of the CSV file, you can use the awk command with the -F option to specify the field separator as a comma (,) and then print the fourth field.

Assuming file2 contains the following CSV data:

Copy code
Column1,Column2,Column3,Column4,Column5
1,apple,3,orange,5
2,banana,4,grape,6
3,cherry,5,mango,7
4,pear,6,kiwi,8
5,watermelon,7,pineapple,9

Running the command awk -F, '{ print $4 }' file2 will output:

Copy code
Column4
orange
grape
mango
kiwi
pineapple

The command separates each line of the file based on the comma delimiter specified with -F,. The print $4 statement prints the fourth field from each line, which corresponds to the values in Column4 in the CSV file.

awk {if ($1 == 1)’ print $0 }’ file2

In the given command, it seems that there is a syntax error. The correct command to achieve the desired result would be:

bashCopy code
awk '{if ($1 == 1) print $0}' file2

This command uses awk to check if the value in the first field ($1) of each line is equal to 1. If the condition is true, it will print the entire line ($0).

Assuming file2 contains the following data:

Copy code
Column1,Column2,Column3,Column4,Column5
1,apple,3,orange,5
2,banana,4,grape,6
3,cherry,5,mango,7
4,pear,6,kiwi,8
5,watermelon,7,pineapple,9

Running the corrected command awk '{if ($1 == 1) print $0}' file2 will output:

Copy code
1,apple,3,orange,5

This is the line that matches the condition where the value in the first field is equal to 1.

awk ‘ /mango/ {print $0}’ file2

The command awk '/mango/ {print $0}' file2 will search for lines in file2 that contain the string "mango" and print the entire line.

Assuming file2 contains the following data:

Copy code
Column1,Column2,Column3,Column4,Column5
1,apple,3,orange,5
2,banana,4,grape,6
3,cherry,5,mango,7
4,pear,6,kiwi,8
5,watermelon,7,pineapple,9

Running the command awk '/mango/ {print $0}' file2 will output:

Copy code
3,cherry,5,mango,7

This is the line that contains the string “mango”, and awk will print the entire line.

awk ‘NR == 3’ file2

The command awk 'NR == 3' file2 will print the third line from the file file2 using the NR variable, which represents the current record number.

Assuming file2 contains the following data:

Copy code
Column1,Column2,Column3,Column4,Column5
1,apple,3,orange,5
2,banana,4,grape,6
3,cherry,5,mango,7
4,pear,6,kiwi,8
5,watermelon,7,pineapple,9

Running the command awk 'NR == 3' file2 will output:

Copy code
3,cherry,5,mango,7

This is the third line in the file, and awk will print it based on the condition NR == 3, where NR represents the line number and 3 is the desired line number to print.

what if the input was awk ‘NR ==”3"’ file2

If you modify the command to use a string comparison instead of a numerical comparison, as in awk 'NR == "3"' file2, the comparison will evaluate to false, and no output will be produced.

In awk, the NR variable represents the current record number (line number). By default, it is of numeric type. Therefore, when you use NR == "3", you are comparing a numeric value to a string value, which will result in a false comparison.

If you want to match a string value for the record number, you can enclose the comparison value in quotes, like this:

bashCopy code
awk 'NR == "3"' file2

In this case, the command will output the third line from the file file2 if the record number (NR) matches the string "3". However, note that NR is typically treated as a numeric value, so using it as a string comparison may not yield the expected results in most scenarios.

aw

k ‘NR == 3’ {print $0} file2

The correct syntax for your command would be:

bashCopy code
awk 'NR == 3 {print $0}' file2

This command will print the entire third line from the file file2 when the record number (NR) is equal to 3.

Assuming file2 contains the following data:

Copy code
Column1,Column2,Column3,Column4,Column5
1,apple,3,orange,5
2,banana,4,grape,6
3,cherry,5,mango,7
4,pear,6,kiwi,8
5,watermelon,7,pineapple,9

Running the corrected command awk 'NR == 3 {print $0}' file2 will output:

Copy code
3,cherry,5,mango,7

This is the third line in the file, and awk will print it based on the condition NR == 3, where NR represents the line number and 3 is the desired line number to match. The $0 represents the entire line.

— — — — — — — — — — — — — — — — — — — — — — — — —

how to grep the time response in tomcat logs

Certainly! Here’s an updated version of the pattern with random parameters, API, and random words:

023-07-07 10:05:15	
10.60.119.238 - - [07/Jul/2023:10:05:14 +0530] "POST /api/v1/users?user-id=987621&timestamp=1625685363 HTTP/1.1" 200 1865 "-" "-" "0.062" "10.60.18.168:80" "-" "-" "-" "-" "-" "10.60.119.238" "c-1531975263799624588b398a-9ade-47a3-a3b6-5abaa39f70d1" "shopping" "PayNow" "1411678391" "-" "-" "-"Show context
2023-07-07 10:04:59
10.188.102.71 - - [07/Jul/2023:10:04:59 +0530] "POST /api/v1/users?user-id=126789&timestamp=1625685359 HTTP/1.1" 200 424 "-" "theia" "0.003" "10.60.26.214:80" "-" "-" "-" "-" "-" "10.188.102.71" "c-1532300145081480a129c23a-c2e0-4507-a9d2-fac07120630e" "PG" "Transaction" "-" "-" "-" "-"
2023-07-07 10:04:50
10.60.76.222 - - [07/Jul/2023:10:04:49 +0530] "POST /api/v1/users?user-id=98321&timestamp=1625685349 HTTP/1.1" 200 2600 "-" "-" "0.004" "10.60.30.92:80" "-" "-" "-" "-" "-" "10.60.76.222" "c-1532074615312606efc3fc0b-ecb8-4f1d-9db8-095a16d6d21d" "shopping" "OnlinePayment" "1141871914" "-" "-" "-"
2023-07-07 10:04:49
10.60.76.231 - - [07/Jul/2023:10:04:49 +0530] "POST /api/v1/users?user-id=9321&timestamp=1625685349 HTTP/1.1" 200 2600 "-" "-" "0.003" "10.60.30.92:80" "-" "-" "-" "-" "-" "10.60.76.231" "c-153207430803231303b3f42a-b5f6-44b0-9641-1144582ea339" "shopping" "PayNow" "1141871914" "-" "-" "-"
2023-07-07 10:04:49
10.60.76.222 - - [07/Jul/2023:10:04:49 +0530] "POST /api/v1/users?user-id=982341&timestamp=1625685349 HTTP/1.1" 200 2600 "-" "-" "0.003" "10.60.30.92:80" "-" "-" "-" "-" "-" "10.60.76.222" "c-15320739010206582383de03-826a-4a4b-84c6-04fe22cb7f7f" "shopping" "PayNow" "1141871914" "-" "-" "-"

to grep the user_id

grep -oE ‘user-id=[0–9]+’ file.txt | awk -F’=’ ‘{print $2}’

to grep user_id ,timestamp, api and the repsonse time taken

ChatGPT

To grep the user-id, timestamp, api, and the response time from the provided text, you can use the following command:

grep -oE 'user-id=[0-9]+|timestamp=[0-9]+|(GET|POST) [^ ]+|"[0-9]+\.[0-9]+"' file.txt

Explanation of the modified pattern:

  1. user-id=[0-9]+: Matches the user ID.
  2. timestamp=[0-9]+: Matches the timestamp.
  3. (GET|POST) [^ ]+: Matches either "GET" or "POST" followed by a space and one or more non-space characters. This matches the API endpoint when it starts with either "GET" or "POST".
  4. "[0-9]+\.[0-9]+": Matches the response time in the format "0.062".

To grep the time and response time taken by the API from the provided data, you can use the following Linux command:

grep -oE '(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})|("[0-9]+\.[0-9]+")' <filename>
  • grep is a command-line utility in Linux used for searching and matching patterns within files.
  • -o option tells grep to only output the matching part of the line rather than the entire line.
  • -E option enables extended regular expressions, allowing us to use the pattern syntax.
  • (\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) is a regular expression pattern that matches the timestamp in the format YYYY-MM-DD HH:MM:SS. Here, \d represents a digit and {4} specifies four occurrences of the preceding digit pattern.
  • | is a logical OR operator that allows us to match either the timestamp or the response time.
  • ("[0-9]+\.[0-9]+") is a regular expression pattern that matches the response time in the format "0.062". Here, [0-9]+ represents one or more digits, \. matches the decimal point, and [0-9]+ matches one or more digits after the decimal point.

write a linux command to filter the time response

grep -oE '"[0-9]+\.[0-9]+"' file.txt | awk -F'"' '$2 >= 0.001 && $2 <= 0.005 { print }'

This command will output the response times great

To include the lines where the response time is greater than or equal to “0.002” or equal to “0.001”

if time = 0.003 then print exception

If you want to print an exception message when the response time is equal to 0.003, you can modify the command as follows:

bashCopy code
grep -oE '"[0-9]+\.[0-9]+"' file.txt | awk -F'"' '{ if ($2 >= 0.001 && $2 <= 0.002) print; else if ($2 == 0.003) print "Exception" }'

sum all the timeresponse in the new filter

To sum up all the response times in the filtered output, you can modify the command as follows:

bashCopy code
grep -oE '"[0-9]+\.[0-9]+"' file.txt | awk -F'"' '{ if ($2 >= 0.001 && $2 <= 0.002) sum += $2; else if ($2 == 0.003) print "Exception" } END { print "Sum: " sum }'

--

--