Property-based testing with hypothesis — Python Mastery

Why this matters

Example-based tests only check the cases you thought of. Hypothesis generates hundreds of random inputs (then shrinks failing ones to a minimal example), finding edge cases human brains miss — empty lists, Unicode in usernames, the exact float that breaks your serialiser. Two hours invested here pays back for years.

Learning objectives

Write property-based tests.
Use core strategies (integers, text, lists, composite).
Define invariants ("for all X, this property holds").
Use stateful tests for state machines.
Interpret shrinking output.

1. Install

powershell

uv add --dev hypothesis

python

from hypothesis import given, strategies as st

@given(st.integers(), st.integers())
def test_add_commutative(a, b):
    assert a + b == b + a

@given decorates a test that takes generated values. By default, Hypothesis runs each test ~100 times.

2. Strategies — describe the input shape

python

st.integers()                              # any int
st.integers(min_value=0, max_value=100)
st.floats(allow_nan=False, allow_infinity=False)
st.text()                                  # any unicode str
st.text(min_size=1, max_size=20, alphabet="abcd")
st.booleans()
st.none()
st.binary()

st.lists(st.integers(), min_size=1)
st.tuples(st.integers(), st.text())
st.sets(st.integers())
st.frozensets(st.integers())
st.dictionaries(st.text(), st.integers())
st.fixed_dictionaries({"name": st.text(), "age": st.integers(min_value=0)})

st.one_of(st.integers(), st.text())        # union
st.sampled_from(["red", "green", "blue"])  # one of the listed
st.just(42)                                # always 42

st.uuids()
st.datetimes(); st.dates(); st.times()
st.decimals(); st.fractions()
st.ip_addresses()
st.emails()

Built-in for common types

python

st.from_type(int)                          # like st.integers()
st.from_type(User)                         # generate by inspecting type hints!

from_type(User) works if User is a pydantic / dataclass / attrs class with hintable fields. Magical.

3. Classic properties

Round-trip

python

import json
from hypothesis import given, strategies as st

@given(st.recursive(
    st.none() | st.booleans() | st.integers() | st.floats(allow_nan=False) | st.text(),
    lambda children: st.lists(children) | st.dictionaries(st.text(), children),
    max_leaves=20,
))
def test_json_roundtrip(value):
    assert json.loads(json.dumps(value)) == value

Round-trip: decode(encode(x)) == x is the canonical property for any (de)serializer.

Idempotence

python

@given(st.text())
def test_strip_idempotent(s):
    assert s.strip().strip() == s.strip()

@given(st.lists(st.integers()))
def test_sort_idempotent(xs):
    assert sorted(sorted(xs)) == sorted(xs)

Invariants

python

@given(st.lists(st.integers()))
def test_sort_preserves_length(xs):
    assert len(sorted(xs)) == len(xs)

@given(st.lists(st.integers()))
def test_sort_is_sorted(xs):
    result = sorted(xs)
    assert all(result[i] <= result[i+1] for i in range(len(result)-1))

@given(st.lists(st.integers()))
def test_sort_is_permutation(xs):
    from collections import Counter
    assert Counter(sorted(xs)) == Counter(xs)

Three small properties pin down "sort" pretty rigorously.

Equivalent implementations

python

@given(st.lists(st.integers()))
def test_my_sort_matches_builtin(xs):
    assert my_sort(xs) == sorted(xs)

If you have a slow correct reference and a fast new implementation, this is gold.

Algebraic laws

python

@given(st.integers(), st.integers(), st.integers())
def test_add_associative(a, b, c):
    assert (a + b) + c == a + (b + c)

@given(st.integers())
def test_add_identity(a):
    assert a + 0 == a

4. Composite strategies

For complex / dependent shapes:

python

from hypothesis import given, strategies as st

@st.composite
def user_strategy(draw):
    name = draw(st.text(min_size=1, max_size=20))
    age = draw(st.integers(min_value=0, max_value=120))
    email = draw(st.emails())
    return User(name=name, age=age, email=email)

@given(user_strategy())
def test_user_serializable(u):
    assert User.from_json(u.to_json()) == u

@st.composite lets you build values step by step, with later draws depending on earlier ones.

5. Assume — skip uninteresting inputs

python

from hypothesis import given, assume, strategies as st

@given(st.integers(), st.integers())
def test_divide(a, b):
    assume(b != 0)
    assert a == (a // b) * b + (a % b)

assume(cond) discards the example if cond is False. Use sparingly — too many discards slow the test and may signal a bad strategy.

6. Shrinking — minimal failing example

When a test fails, Hypothesis tries to make the failing input minimal (e.g., shortest string, smallest list, smallest int) before reporting. Output:

Falsifying example: test_func(
    xs=[0, 1],
)

That's the smallest input that triggers the bug, not the random one that originally failed. Often a textbook off-by-one or empty-list case.

7. Settings — verbosity, deadline, max examples

python

from hypothesis import given, settings, Verbosity

@settings(max_examples=500, deadline=200, verbosity=Verbosity.verbose)
@given(...)
def test_thorough(...): ...

max_examples: how many examples (default 100).
deadline: per-test timeout (ms); set None for slow tests.
verbosity: print each example for debugging.
phases: control test phases.

Profiles per environment:

python

settings.register_profile("ci", max_examples=1000, deadline=500)
settings.register_profile("dev", max_examples=20)
settings.load_profile(os.getenv("HYPOTHESIS_PROFILE", "dev"))

8. Stateful (rule-based) testing

For testing state machines (caches, queues, ORMs):

python

from hypothesis.stateful import RuleBasedStateMachine, rule, invariant

class CacheStateMachine(RuleBasedStateMachine):
    def __init__(self):
        super().__init__()
        self.model = {}                  # reference impl
        self.cache = LRUCache(capacity=5)

    @rule(key=st.text(min_size=1, max_size=3), value=st.integers())
    def put(self, key, value):
        self.model[key] = value
        self.cache.put(key, value)

    @rule(key=st.text(min_size=1, max_size=3))
    def get(self, key):
        if key in self.model:
            assert self.cache.get(key) in (self.model[key], None)   # may have been evicted

    @invariant()
    def size_ok(self):
        assert len(self.cache) <= 5

TestCache = CacheStateMachine.TestCase

Hypothesis generates random sequences of put/get and checks invariants after each step. Devastatingly good at finding concurrency / state-machine bugs.

9. Hypothesis + pandas / NumPy

python

from hypothesis.extra.numpy import arrays, array_shapes
from hypothesis.extra.pandas import data_frames, columns, range_indexes

@given(arrays(dtype=np.float64, shape=array_shapes(min_dims=1, max_dims=2)))
def test_normalise(a):
    if a.std() == 0: return
    z = (a - a.mean()) / a.std()
    assert abs(z.mean()) < 1e-6
    assert abs(z.std() - 1) < 1e-6

@given(data_frames(
    columns=[
        columns("x", dtype=int),
        columns("y", dtype=float),
    ],
    index=range_indexes(min_size=1, max_size=100),
))
def test_df_op(df):
    out = my_transform(df)
    assert list(out.columns) == ["x", "y", "z"]
    assert len(out) == len(df)

These extensions cover NumPy arrays, pandas DataFrames, JSON, datetimes, regex-matched strings, and more.

10. When to use property tests

Yes	No
Parsers, serialisers, codecs	UI / integration smoke tests
Algorithms with clear invariants	Code where "correct" is fuzzy or example-defined
Math / numeric routines	"Just exercise this once" tests
Data structures	Tests where each example needs hand-tuning
Validators / sanitisers	Anything with significant side effects

Property + example tests are complementary. Examples document the API; properties find bugs.

11. Worked example: testing a tokenizer

python

from hypothesis import given, strategies as st

@given(st.text())
def test_decode_encode_roundtrip(s):
    assert tokenizer.decode(tokenizer.encode(s)) == s

@given(st.text())
def test_encode_returns_ints(s):
    ids = tokenizer.encode(s)
    assert all(isinstance(i, int) for i in ids)
    assert all(0 <= i < tokenizer.vocab_size for i in ids)

@given(st.lists(st.text(min_size=1, max_size=5), min_size=1))
def test_concat_then_encode_equals_concat_of_encodings(parts):
    full = "".join(parts)
    assert tokenizer.encode(full) == sum((tokenizer.encode(p) for p in parts), [])
    # Note: this property is FALSE for BPE tokenizers — the test would find that bug.

The last property is a deliberate false claim — running it would falsify with a minimal counter-example, which is exactly what you want during exploration.

Hands-on lab (2 hours)

Write a property test asserting sorted(sorted(xs)) == sorted(xs).
Test JSON round-trip on the recursive strategy from this lesson.
Implement flatten(nested_list) and add three properties (length, no nested lists, elements preserved).
Write a composite strategy for a User(name, age, email) and a property that User.from_dict(u.to_dict()) == u.
Use assume to test integer division for b != 0.
Build a RuleBasedStateMachine for a stack (push/pop) with invariants.
Bonus: install hypothesis-jsonschema and generate values matching a Pydantic schema.

Common pitfalls

Strategy too narrow → finds nothing.
Strategy too broad → mostly tests garbage; slow.
Properties that only hold for "reasonable" inputs without using assume.
Asserting equality on floats without tolerance.
Non-deterministic test body (e.g., uses real time / global state).
Forgetting that shrinking takes time; long fixtures slow shrinks.

Self-check

What is the round-trip property?
What does @st.composite do?
What does Hypothesis "shrinking" mean?
When use RuleBasedStateMachine?
Why use assume carefully?

References

Hypothesis docs: https://hypothesis.readthedocs.io/.
David MacIver, "Hypothesis: property-based testing for Python" (talks, blog).
John Hughes, "QuickCheck" original paper.
Property-Based Testing with PropEr, Erlang, and Elixir (concepts transfer).
"Why Generative Testing?" — Carlos Bueno.