Regex and text processing

Why this matters

Regex is the universal hammer for text. Used well it's two lines instead of fifty; used poorly it's a maintenance disaster. This lesson teaches you Python's re module, modern features, when to reach for regex (third-party) or pyparsing, and when to not use regex at all (HTML, recursive grammars, things that have a real parser).

Learning objectives

Use the re module's core API.
Read and write common regex patterns.
Use named groups, lookaround, non-greedy quantifiers.
Avoid catastrophic backtracking.
Pick between re, regex, and a real parser.

1. `re` module — core API

python

import re

re.search(pattern, string)        # first match anywhere; returns Match or None
re.match(pattern, string)         # match at the START only
re.fullmatch(pattern, string)     # entire string must match
re.findall(pattern, string)       # list of strings or tuples (if groups)
re.finditer(pattern, string)      # iterator of Match objects
re.sub(pattern, replacement, string)
re.subn(pattern, replacement, string)   # also returns count
re.split(pattern, string)

Compile for re-use

python

EMAIL = re.compile(r"[^@\s]+@[^@\s]+\.[^@\s]+")
EMAIL.findall(text)

Pre-compiled patterns are slightly faster and document intent. Always do this for patterns used more than once.

Match objects

python

m = re.search(r"(\w+)@(\w+\.\w+)", "ada@example.com")
m.group(0)                        # 'ada@example.com'
m.group(1)                        # 'ada'
m.group(2)                        # 'example.com'
m.groups()                        # ('ada', 'example.com')
m.span()                          # (0, 15)
m.start(), m.end()
m.groupdict()                     # if named groups

2. The 30 metacharacters you'll use

Pattern	Matches
`.`	Any char except newline (use `re.DOTALL` to include `\n`)
`\d`	Digit `[0-9]`
`\D`	Non-digit
`\w`	Word char `[a-zA-Z0-9_]` (Unicode-aware)
`\W`	Non-word
`\s`	Whitespace
`\S`	Non-whitespace
`\b`	Word boundary (zero-width)
`\B`	Non-boundary
`^`	Start of string (or line with `re.MULTILINE`)
`$`	End of string (or line with `re.MULTILINE`)
`[abc]`	Char class
`[^abc]`	Negated class
`[a-z]`	Range
`a\|b`	Alternation
`a?`	0 or 1
`a*`	0 or more
`a+`	1 or more
`a{3}`	Exactly 3
`a{3,5}`	3-5
`a*?`, `a+?`, `a??`	Non-greedy variants
`(...)`	Capturing group
`(?:...)`	Non-capturing group
`(?P<name>...)`	Named group
`(?=...)`	Positive lookahead (zero-width)
`(?!...)`	Negative lookahead
`(?<=...)`	Positive lookbehind
`(?<!...)`	Negative lookbehind
`\1`, `\2`	Backreference
`(?:a)+?`	Non-greedy non-capturing group, 1+

Greedy vs non-greedy

python

re.findall(r"<.+>", "<a><b>")          # ['<a><b>']    greedy: matches longest
re.findall(r"<.+?>", "<a><b>")         # ['<a>', '<b>']  non-greedy

Tend to non-greedy (*?, +?) when matching delimited spans.

Lookaround

Zero-width: matches a position, not characters. Useful for "find X not preceded/followed by Y."

python

re.findall(r"(?<=\$)\d+", "price $100 and €50")    # ['100']  — must follow $
re.findall(r"\d+(?!\.\d)", "1, 2.5, 3, 4.0")        # ['1', '3'] — not followed by decimal

3. Flags

Pass via flags= or use inline (?flag):

python

re.IGNORECASE       # re.I — case insensitive
re.MULTILINE        # re.M — ^ and $ match line boundaries
re.DOTALL           # re.S — . matches \n
re.VERBOSE          # re.X — ignore whitespace + comments in pattern
re.ASCII            # re.A — \w, \d, \s match ASCII only
re.UNICODE          # default

python

PHONE = re.compile(r"""
    \(? (\d{3}) \)?        # area code
    [\s\-]?                # separator
    (\d{3})                # exchange
    [\s\-]?                # separator
    (\d{4})                # line
""", re.VERBOSE)
PHONE.search("(555) 123-4567").groups()    # ('555', '123', '4567')

re.VERBOSE is the readability superpower. Use it for any pattern longer than 30 characters.

4. Named groups (PEP 8 says yes)

python

PHONE = re.compile(r"(?P<area>\d{3})-(?P<exchange>\d{3})-(?P<line>\d{4})")
m = PHONE.search("123-456-7890")
m.groupdict()                          # {'area': '123', 'exchange': '456', 'line': '7890'}
m["area"]                              # subscript works

Named groups document intent and survive refactors better than positional indices.

Backreferences

python

re.findall(r"(\w+) \1", "the the dog ran ran fast")    # ['the', 'ran']

\1 matches whatever group 1 captured. Useful for "doubled word" / palindrome-ish patterns.

Replace with backreferences

python

re.sub(r"(\w+)@(\w+)", r"\2:\1", "ada@example.com")    # 'example:ada'
re.sub(r"(?P<first>\w+) (?P<last>\w+)", r"\g<last>, \g<first>", "Ada Lovelace")
# 'Lovelace, Ada'

\g<name> is the safe form for named groups (avoids ambiguity with digit backreferences).

5. `re.sub` with a function

python

def double_digit(m):
    return str(int(m.group(0)) * 2)

re.sub(r"\d+", double_digit, "a1 b22 c333")    # 'a2 b44 c666'

Powerful for non-trivial replacements.

6. Catastrophic backtracking — the regex bomb

Some patterns cause exponential blow-up:

python

re.search(r"^(a+)+$", "a" * 20 + "X")            # very slow
re.search(r"(.*a){10}", "aaaa..." + "X")          # very slow

Symptoms: simple-looking pattern, exponential blow-up on certain inputs. Causes: nested quantifiers on overlapping content.

Mitigations:

Use atomic groups or possessive quantifiers — re doesn't support them, but the third-party regex module does (uv add regex).
Restructure the pattern to avoid ambiguity.
Apply timeouts (Python's re has none — regex has timeout=).

If you're running regex on untrusted input, this is a ReDoS DoS vector. Either use regex with timeout, or validate input lengths.

python

import regex
regex.search(r"^(a+)+$", "a"*40 + "X", timeout=1.0)    # raises TimeoutError

7. When NOT to use regex

HTML / XML: use lxml, beautifulsoup4, selectolax.
JSON: use json or msgspec.
Recursive grammars (matching balanced parens, function calls): use pyparsing or write a real parser (lark, ply).
URLs: use urllib.parse.
Emails for validation (vs filtering): use email-validator package.
Code parsing: use ast for Python, tree-sitter for everything else.

Regex is great for "extract / replace small patterns in flat text." For anything with structure, use a parser.

8. The third-party `regex` module

uv add regex then import regex (drop-in replacement, plus more).

Extras:

Atomic groups (?>...), possessive quantifiers *+, ++, ?+.
Variable-length lookbehind (re requires fixed length).
Better Unicode (named scripts: \p{Greek}, \p{Letter}).
timeout= parameter.
Recursion (?R).

Use regex when you need any of these. Most code stays on re.

9. Common patterns library

python

# Whitespace cleanup
re.sub(r"\s+", " ", text).strip()

# Trim trailing whitespace on every line
re.sub(r"[ \t]+$", "", text, flags=re.MULTILINE)

# URLs (rough)
URL = re.compile(r"https?://\S+")

# IPv4 (rough; for validation use ipaddress.ip_address)
IPV4 = re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b")

# Hex color
HEX = re.compile(r"#(?:[0-9a-fA-F]{3}){1,2}\b")

# CamelCase → snake_case
def camel_to_snake(name):
    s1 = re.sub("(.)([A-Z][a-z]+)", r"\1_\2", name)
    return re.sub("([a-z0-9])([A-Z])", r"\1_\2", s1).lower()
camel_to_snake("HTTPRequestHandler")           # 'http_request_handler'

# Slugify
def slugify(s):
    s = re.sub(r"[^\w\s-]", "", s.lower())
    return re.sub(r"[-\s]+", "-", s).strip("-_")

# Strip ANSI escape codes
ANSI = re.compile(r"\x1b\[[0-9;]*[a-zA-Z]")
clean = ANSI.sub("", terminal_output)

Save these in your text/utils.py.

10. `str.translate` — faster char remap than regex

python

# Build a translation table
table = str.maketrans({"a": "A", "b": "B", "c": None})    # None deletes
"abc xyz".translate(table)                     # 'AB xyz'

# Build from two strings of equal length
table = str.maketrans("aeiou", "AEIOU")
"hello".translate(table)                       # 'hEllO'

For pure character-level substitutions on large strings, translate beats re.sub by 10×+.

11. Unicode considerations

python

import unicodedata

unicodedata.normalize("NFC", text)             # canonical composition
unicodedata.normalize("NFKD", text)            # compatibility decomposition (strips diacritics)

# Strip accents (e.g., "résumé" → "resume")
def strip_accents(s):
    return "".join(c for c in unicodedata.normalize("NFKD", s)
                   if not unicodedata.combining(c))

# Case-insensitive comparison
"Straße".casefold() == "strasse".casefold()    # True

Always normalise before string comparisons across user input from different keyboards / OSes.

12. Worked example: extract structured data from log lines

python

import re
from dataclasses import dataclass
from datetime import datetime

LINE = re.compile(r"""
    ^(?P<ts>\d{4}-\d{2}-\d{2}\ \d{2}:\d{2}:\d{2}(?:\.\d+)?)\s+    # timestamp
    (?P<level>DEBUG|INFO|WARNING|ERROR|CRITICAL)\s+               # level
    (?P<logger>[\w.]+)                                            # logger name
    \s*:\s*
    (?P<message>.+)$                                              # message
""", re.VERBOSE)

@dataclass
class LogRecord:
    ts: datetime; level: str; logger: str; message: str

def parse(line: str) -> LogRecord | None:
    m = LINE.match(line)
    if not m: return None
    return LogRecord(
        ts=datetime.fromisoformat(m["ts"].replace(" ", "T")),
        level=m["level"], logger=m["logger"], message=m["message"],
    )

for line in open("app.log", encoding="utf-8"):
    rec = parse(line.rstrip("\n"))
    if rec and rec.level in {"ERROR", "CRITICAL"}:
        print(rec)

Verbose regex + named groups + dataclass. Reads top-to-bottom; refactors cleanly.

Hands-on lab (1.5 hours)

Write a regex that extracts all email addresses from a text file; print unique results.
Convert CamelCase to snake_case (and back) using two functions.
Build a slugify that handles unicode (use unicodedata.normalize).
Parse nginx-style access logs with re.VERBOSE + named groups; load into dataclasses.
Find sentences (split on .!? not followed by ., lookahead). Compare to nltk.sent_tokenize.
Try to trigger catastrophic backtracking with ^(a+)+$; measure with time.perf_counter. Then switch to regex with timeout=.
Bonus: build a tiny calculator for 1 + 2 * 3 using regex tokenisation + a recursive descent parser. (Notice when regex stops being enough.)

Common pitfalls

Forgetting raw strings (r"...") → \n becomes newline before regex sees it.
re.match (anchored at start) when you wanted re.search.
Greedy .* eating too much.
Catastrophic backtracking on user input.
Parsing HTML / JSON / CSV with regex.
Forgetting re.VERBOSE for long patterns; future-you can't read them.

Self-check

re.match vs re.search vs re.fullmatch?
What does (?P<name>...) do?
What is catastrophic backtracking?
Name two cases where regex is wrong.
What does unicodedata.normalize("NFKD", s) do?

References

Python re HOWTO: https://docs.python.org/3/howto/regex.html.
Mastering Regular Expressions, Jeffrey Friedl.
regex module docs: https://github.com/mrabarnett/mrab-regex.
regex101.com (Python flavour) for interactive testing.
OWASP, "Regular Expression Denial of Service."

Why this matters

Learning objectives

1. re module — core API

Compile for re-use

Match objects

2. The 30 metacharacters you'll use

Greedy vs non-greedy

Lookaround

3. Flags

4. Named groups (PEP 8 says yes)

Backreferences

Replace with backreferences

5. re.sub with a function

6. Catastrophic backtracking — the regex bomb

7. When NOT to use regex

8. The third-party regex module

9. Common patterns library

10. str.translate — faster char remap than regex

11. Unicode considerations

12. Worked example: extract structured data from log lines

Hands-on lab (1.5 hours)

Common pitfalls

Self-check

References

1. `re` module — core API

5. `re.sub` with a function

8. The third-party `regex` module

10. `str.translate` — faster char remap than regex