Lesson 6: Text Streams & Pipes
Lesson 6: Text Streams & Pipes
Standard Streams
Every process has 3 streams: | FD | Name | Default | Purpose | |----|------|---------|---------| | 0 | stdin | Keyboard | Input | | 1 | stdout | Terminal | Normal output | | 2 | stderr | Terminal | Error messages |
Pipes (|)
Connect stdout → stdin:
cat /etc/passwd | grep bash
ls -la | head -20
ps aux | grep nginx | grep -v grep
Redirection
cmd > file.txt # stdout → file (overwrite)
cmd >> file.txt # stdout → file (append)
cmd 2> file.txt # stderr → file
cmd > out.txt 2>&1 # stdout & stderr → file
cmd &> out.txt # Same (bash shortcut)
cmd < input.txt # stdin ← file
grep — Search
grep 'pattern' file.txt # Basic
grep -i 'pattern' file.txt # Case-insensitive
grep -r 'pattern' /dir/ # Recursive
grep -v 'pattern' file.txt # Invert (exclude)
grep -n 'pattern' file.txt # Show line numbers
grep -E 'pat1|pat2' file.txt # Extended regex (OR)
grep -c 'pattern' file.txt # Count matches
sed — Stream Editor
sed 's/old/new/' file.txt # First match per line
sed 's/old/new/g' file.txt # Global (all matches)
sed -i 's/old/new/g' file.txt # In-place edit (backup: -i.bak)
sed '/^#/d' file.txt # Delete lines starting with #
sed -n '10,20p' file.txt # Print lines 10-20
awk — Pattern Processing
awk '{print $1}' file.txt # Column 1 (whitespace-delimited)
awk -F: '{print $1, $3}' /etc/passwd # Colon-delimited, cols 1 & 3
awk '$3 > 1000' /etc/passwd # Filter: UID > 1000
awk '{sum += $1} END {print sum}' # Sum column 1
cut — Column Extraction
cut -d: -f1 /etc/passwd # Col 1, colon-delimited
cut -d' ' -f1,3 file.txt # Cols 1 & 3, space-delimited
cut -c1-10 file.txt # Characters 1-10
sort, uniq, wc
sort file.txt # Sort lines
sort -n file.txt # Numeric sort
sort -u file.txt # Sort + unique
uniq file.txt # Remove adjacent duplicates (needs sort first)
sort file.txt | uniq -c # Count occurrences
wc -l file.txt # Line count
wc -w file.txt # Word count
wc -c file.txt # Byte count
Regex Basics
| Pattern | Meaning |
|---|---|
. |
Any character |
^ |
Start of line |
$ |
End of line |
* |
Zero or more |
+ |
One or more |
? |
Zero or one |
[abc] |
a, b, or c |
[^abc] |
Not a, b, or c |
\d |
Digit (grep -P / awk) |
\s |
Whitespace |
Practice Pipeline
# Find top 10 IPs in nginx access log
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10
# Count errors per hour in syslog
grep 'error' /var/log/syslog | awk '{print $3}' | cut -d: -f1 | sort | uniq -c
# Find largest files in /var/log
find /var/log -type f -name "*.log" -exec ls -lh {} + | awk '{print $5, $9}' | sort -hr | head -5
💻 实战命令速查
grep 'error' /var/log/syslog
Search for pattern in file
$ grep 'error' /var/log/syslog
Nov 1 10:00:01 server systemd[1]: error starting servicegrep -r 'TODO' /home/ubuntu/projects
Recursive search in directory
$ grep -r 'TODO' .
./src/main.py:10:# TODO: add auth
./tests/test.py:5:# TODO: write testscat file.txt | grep 'pattern'
Pipe: pass output to another command
$ cat /etc/passwd | grep bash
root:x:0:0:root:/root:/bin/bash
ubuntu:x:1000:1000:ubuntu:/home/ubuntu:/bin/bashecho 'hello world' | sed 's/world/Linux/'
sed: stream editor (find/replace)
$ echo 'hello world' | sed 's/world/Linux/'
hello Linuxawk '{print $1}' /etc/passwd
awk: print first column (space-delimited)
$ awk -F: '{print $1}' /etc/passwd
root
daemon
ubuntucut -d: -f1,3 /etc/passwd
cut: extract columns by delimiter
$ cut -d: -f1,3 /etc/passwd
root:0
daemon:1
ubuntu:1000sort file.txt | uniq
sort + uniq: remove duplicates
$ echo -e 'b
a
b
c
a' | sort | uniq
a
b
cwc -l /etc/passwd
wc: count lines, words, chars
$ wc -l /etc/passwd
42 /etc/passwdcommand > output.txt
Redirect stdout to file (overwrite)
$ ls -la > listing.txtcommand >> output.txt
Append stdout to file
$ date >> log.txtcommand 2> errors.txt
Redirect stderr to file
$ ls /nonexistent 2> err.txtcommand > out.txt 2>&1
Redirect both stdout and stderr
$ find / -name '*.conf' > found.txt 2>&1