Filters

Filters are programs that take plain text (stored in a file or produced by another program) as standard input, transform it into a meaningful format, and then return it as standard output. Linux has a multitude of filters.

Cat

The primary purpose of the cat command is to display the contents of one or more text files on the terminal. This command allows quick viewing of file contents.

root@hackerbox:~$ cat /etc/ssh/sshd_config
# Port 22
# AddressFamily any
# ListenAddress 0.0.0.0
# ListenAddress ::
PermitRootLogin no
PasswordAuthentication yes
PermitEmptyPasswords no
ChallengeResponseAuthentication no
UsePAM yes

In the example above, the configuration file for the SSH service located at /etc/ssh/sshd_config is displayed on the terminal using the cat command.

Head

The head command is used to display the first few lines of a specified file. By default, the head command shows the first 10 lines, but this number can be changed using the -n parameter.

This command is handy when you want to quickly review the beginning of a file without displaying its entire content.

root@hackerbox:~$ head -n 3 /var/log/apache2/access.log
192.168.1.1 - - [15/Mar/2024:10:00:00 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
192.168.1.2 - - [15/Mar/2024:10:00:02 +0000] "POST /login.php HTTP/1.1" 200 452 "http://example.com/login" "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N)"
192.168.1.3 - - [15/Mar/2024:10:00:03 +0000] "GET /wp-admin HTTP/1.1" 403 497 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6)"

In the example above, the first 3 lines of the log file located at /var/log/apache2/access.log for the Apache2 Web Server service are displayed on the terminal.

Tail

The tail command is used to display the last few lines of a specified file. By default, the tail command shows the last 10 lines, but this number can be changed using the -n parameter.

This command is extremely useful for monitoring the most recently added content to continuously growing files, such as log files.

root@hackerbox:~$ tail -n 3 /var/log/auth.log
Mar 15 12:00:00 servername sshd[23456]: Failed password for invalid user admin from 192.168.1.1 port 54321 ssh2
Mar 15 12:01:00 servername sshd[23457]: Accepted password for user1 from 192.168.1.2 port 65432 ssh2
Mar 15 12:02:00 servername sshd[23458]: Failed password for user2 from 192.168.1.3 port 76543 ssh2

In the example above, the last three lines of the auth.log file are displayed. The auth.log file logs events related to user authentication on a Linux system, including user logins and logouts, sudo command usage, SSH sessions, and other authentication-related events.

Sort

The sort command sorts the contents of a given file alphabetically.

root@hackerbox:~$ cat names.txt
Bob
Charlie
Alice


root@hackerbox:~$ sort names.txt
Alice
Bob
Charlie

In the example above, the contents of "names.txt" are sorted alphabetically.

Uniq

The uniq command filters out consecutive duplicate lines from a file and shows the unique lines.

It is often used in conjunction with the sort command because, when used alone, it only detects consecutive duplicate lines. To address duplicates throughout the file, it's recommended to first sort the data.

root@hackerbox:~$ cat names.txt
Alice
Charlie
Alice
Bob

root@hackerbox:~$ uniq names.txt
Alice
Charlie
Alice
Bob

In the example above, although "Alice" appears twice, the uniq command does not remove non-consecutive duplicates.

Therefore, to eliminate non-consecutive duplicates, first sort the file and then apply the uniq command.

root@hackerbox:~$ sort names.txt | uniq
Alice
Bob
Charlie

Grep

The grep command searches files for specific text strings, filters lines, and displays matching results.grep is a powerful tool commonly used to search log files, configuration files, or any text file.

root@hackerbox:~$ grep '192.168.1.1' /var/log/apache2/access.log

This command will display all records in the Apache 2 web server access logs located at /var/log/apache2/access.log that contain the IP address 192.168.1.1.

root@hackerbox:~$ grep '192.168.1.1' /var/log/apache2/access.log192.168.1.1 - - [15/Mar/2024:10:00:00 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"192.168.1.1 - - [15/Mar/2024:10:00:02 +0000] "POST /login.php HTTP/1.1" 200 452 "http://example.com/login" "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N)"192.168.1.1 - - [15/Mar/2024:10:00:03 +0000] "GET /wp-admin HTTP/1.1" 403 497 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6)"

Wc

The wc (word count) command quickly determines how large a file is or how much data it contains.

root@hackerbox:~$ wc /etc/passwd46   
67 2544 /etc/passwd

In the example above, the wc command returns the number of lines, words, and characters respectively in the/etc/passwd file, which contains the list of registered users on a Linux system.

Column Value	Description
46	Number of lines
67	Number of words
2544	Number of characters
/etc/passwd	File path

The wc command has various parameters:

• -l: Displays only the number of lines.

• -w: Displays only the number of words.

• -c: Displays only the number of bytes.

• -m: Displays only the number of characters (useful for multi-byte character sets).

For example, to see the total number of log entries recorded by the Apache 2 web server to date, you can use the -l parameter with the following command:

root@hackerbox:~$ wc -l /var/log/apache2/access.log
54230 /var/log/apache2/access.log

The output indicates that there are a total of 54,230 log entries.

Sed

The sed(stream editor) command is a tool capable of performing various text edits such as processing, modifying, adding, deleting, or replacing texts between files.

The sed command is commonly used to filter and transform texts.

root@hackerbox:~$ cat names.txt
Alice
Charlie
Bob


root@hackerbox:~$ sed 's/Alice/George/' names.txt
George
Charlie
Bob

In the example above, the name Alice in names.txt is replaced with George using the sed command. However, note that thesedcommand only prints the change to the screen and does not save it to the file.

Awk

The awk command is designed for text and data processing tasks, and it is especially effective when working with column-based data. It reads files line by line, splits each line into fields (columns), and processes them based on specified conditions.awk offers numerous functions and control structures for complex text processing.

root@hackerbox:~$ cat names.txt
John Doe
Emily Clark
Alex Turner


root@hackerbox:~$ awk '{print $1}' names.txt
John
Emily
Alex

In this example, the file names.txt contains three name-surname pairs: John Doe, Emily Clark, and Alex Turner. The command awk '{print $1}' names.txt processes the content of this file using the awk program.awk reads the text files line by line, splitting each line into fields separated by spaces or tabs. In this case, the expression{print $1} instructs awk to print only the first field (the first name) of each line.

Using these useful Linux filters, you can process, search, and transform text files quickly and efficiently according to your specific need.