Linux has a lot of powerful tools built right in, so many little tricks that you can leverage to make your job easier. But few of them compare to the wizardry that is the combination of grep, sed, and awk. With these three tools you can accomplish almost anything that has to do with text manipulation! In our 200th episode of Into the Terminal, Nate and Scott walk you through their favorite moves with these ancient and powerful tools. Watch the full episode here or practice your skills in the interactive labs.
The Essentials
Before you write a custom Python or Bash script to parse a CSV or log file, ask yourself if you can just solve it in one line with the classics:
| Command |
What it does |
grep |
Filters text to show only lines containing a specific pattern |
sed |
The Stream Editor — manipulates text in-stream (e.g., find and replace) |
awk |
A full text-processing language, incredible for isolating columns and fields of data |
zcat |
Functions exactly like cat, but automatically uncompresses .gz files on the fly |
Advanced Features
Leveling Up Your grep Game
Everyone knows grep <string> <file>, but a few flags can make it exponentially more useful when hunting down errors across a system:
- grep -v "Console" file.csv: The reverse grep. Shows you every line that does not contain the match.
- **grep -i "error" file.csv: Case-insensitive matching. It will match "error", "Error", and "ERROR".
- **grep -r -i "error" /var/log/: The recursive grep. This is perfect when you know an error occurred, but you have no idea which log file caught it. It will search every file in the directory tree and prepend the match with the filename where it was found!
The Power of sed for Find and Replace
sed is often used to strip out useless data before handing it off to another command.
Example: You have a CSV file where the last column is full of literal null strings, sometimes with trailing commas.
You can strip the word "null" with standard sed regex:
grep "Console" lab-data.csv | sed -e 's/null//g'
To remove the word "null" and the optional trailing comma, we can use Extended Regex (-E) and chain multiple expressions (-e):
grep "Console" lab-data.csv | sed -E -e 's/null,?//g' -e 's/,$//g'
Pro Tip: To apply these edits permanently to the file itself instead of just printing them to standard output, use the -i (in-place) flag: sed -i -E ... file.csv
Field Manipulation with awk
If your data has columns (like a space-separated log file or a comma-separated CSV), awk is your best friend.
Extracting specific columns:
cat httpd_access.log | awk '{print $1}'
This isolates just the very first column of data (usually the IP address in an Apache log). By default, awk separates fields by spaces, but you can define a custom delimiter with -F:
cat httpd_access.log | awk -F'- -' '{print $2}'
Working from the end of the line:
Sometimes lines are different lengths, making it impossible to count from the left. awk provides the NF (Number of Fields) variable, letting you select columns from the right!
- Print the very last column: awk '{ print $(NF) }'
- Print the second-to-last column: awk '{ print $(NF-1) }'
Quick Reference Card
```bash
Read a compressed log file and find all occurrences of "root"
zcat secure_sample.gz | grep root
Recursively search all log files for "error", ignoring case
grep -r -i "error" /var/log/
Find and replace "null" with an empty string in-place
sed -i -e 's/null//g' lab-data.csv
Print only the first column of a space-separated file
cat httpd_access.log | awk '{print $1}'
Print the last column of a space-separated file
cat httpd_access.log | awk '{ print $(NF) }'
Chain them together! Find GET requests and print the second field based on a custom delimiter
cat httpd_access_sample | awk -F'- -' '{print $2}' | grep GET
```
Links & Resources
Into the Terminal is a show dedicated to helping you grow your knowledge of critical administration skills for Red Hat Enterprise Linux. Whether you are new to Linux or new to RHEL, join us for a hands-on look at commands, processes, and tools.
