5.1.3. Regular Expressions and Text Processing in Scripts

💡 First Principle: Regular expressions are a pattern language for describing text. They appear in grep, sed, awk, [[ =~ ]], and most modern programming languages. Learning regex once unlocks text processing everywhere. The key insight: regex patterns describe the shape of text (what characters, how many, in what sequence) rather than specific content.

Regex Fundamentals:

# Anchors
^                  # Start of line
$                  # End of line

# Character classes
.                  # Any single character (except newline)
[abc]              # a, b, or c
[a-z]              # Any lowercase letter
[A-Z0-9]           # Uppercase letter or digit
[^abc]             # NOT a, b, or c

# Quantifiers
*                  # 0 or more of previous
+                  # 1 or more (extended regex / ERE)
?                  # 0 or 1 (optional)
{3}                # Exactly 3
{2,5}              # 2 to 5
{3,}               # 3 or more

# Groups and alternation
(abc)              # Capture group
abc|def            # abc OR def

# Common patterns
^[0-9]+$           # Entire line is digits
^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$   # Email (simplified)
^([0-9]{1,3}\.){3}[0-9]{1,3}$                        # IPv4 address

grep with regex:

grep -E "^[0-9]+" file           # Extended regex (ERE) — same as egrep
grep -P "\d{4}-\d{2}-\d{2}" file # Perl-Compatible Regex (PCRE)
grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" file  # Extract IPs only
grep -c "^ERROR" logfile         # Count lines starting with ERROR

sed for transformation:

# Substitution
sed 's/old/new/g' file               # Replace all occurrences
sed 's/old/new/2' file               # Replace 2nd occurrence only
sed -i 's/old/new/g' file            # In-place (edit file directly)
sed -i.bak 's/old/new/g' file        # In-place with backup (.bak)

# Address ranges
sed '5,10s/foo/bar/g' file           # Lines 5-10 only
sed '/^#/d' file                     # Delete comment lines
sed -n '/ERROR/,/RESOLVED/p' file    # Print block from ERROR to RESOLVED

# Multiple commands
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file

awk for structured data:

# Basic field processing
awk '{print $1, $NF}' file           # Print first and last field
awk -F: '{print $1, $3}' /etc/passwd # Colon delimiter; print user and UID

# Conditions
awk '$3 > 1000 {print $1}' /etc/passwd   # Users with UID > 1000
awk '/ERROR/ {print NR": "$0}' log.txt   # Print line number + ERROR lines

# BEGIN/END blocks
awk 'BEGIN{count=0} /ERROR/{count++} END{print "Total errors:", count}' log.txt

# Accumulate values
awk '{sum += $3} END {print "Total:", sum}' data.txt

⚠️ Exam Trap: sed -i without a backup suffix edits the file in place with no recovery option. Always use sed -i.bak in production to preserve the original, or test your sed command without -i first and redirect to a new file, then verify before replacing the original.

Reflection Question: Write a one-liner using awk and sort that reads /etc/passwd, prints all usernames with UID >= 1000, and sorts them alphabetically.