Linux Text Processing Guide: Master grep, sed, and awk Commands

Text processing is a fundamental skill in Linux. This guide covers essential commands for searching, manipulating, and transforming text data.

Table of Contents

Pattern Matching with grep

Basic Search

grep "pattern" file.txt

Output:

$ grep "error" system.log
2024-11-02 10:15:23 ERROR Database connection failed
2024-11-02 10:15:25 ERROR Retry attempt 1

Advanced grep Options

grep -i "WARNING"      # Case-insensitive
grep -r "TODO"         # Recursive search
grep -v "exclude"      # Inverse match
grep -n "line number"  # Show line numbers

Text Manipulation with sed

Basic Substitution

sed 's/old/new/' file.txt

Multiple Operations

sed -e 's/one/1/g' -e 's/two/2/g' numbers.txt

Output:

$ cat numbers.txt
one two three
$ sed -e 's/one/1/g' -e 's/two/2/g' numbers.txt
1 2 three

Interactive Quiz

Question 1

How would you search for all lines containing either “error” or “warning” (case-insensitive)?

grep -i -E "error|warning" logfile.txt

Question 2

How can you replace all occurrences of “foo” with “bar” in a file?

sed 's/foo/bar/g' file.txt

Data Processing with awk

Basic Usage

awk '{print $1, $3}' data.txt

Field Processing

awk -F',' '{sum+=$2} END {print "Total:", sum}' sales.csv

Practical Examples

Log Analysis

# Find all failed login attempts
grep "Failed password" /var/log/auth.log | \
awk '{print $11}' | sort | uniq -c

CSV Processing

# Calculate average of third column
awk -F',' '{sum+=$3} END {print "Average:", sum/NR}' data.csv

Best Practices

1. Use appropriate regex patterns
2. Test commands on sample data first
3. Make backups before bulk changes
4. Consider using -i for case-insensitive matches
5. Use proper quoting for patterns

Conclusion

Mastering text processing commands enables efficient handling of logs, configuration files, and data processing tasks in Linux.

What’s Next in the Series?

• Network Commands Guide
• System Monitoring Tools
• Shell Scripting Basics