Review property-testing skills with real counterexamples

Review a property-testing skill by the guarantees its tests can falsify, the inputs it actually generates, and the failures it can explain. A large example count is useful evidence only when the property, domain and implementation are connected to a stated contract.

Choose a property before choosing a tool

Property-Based Testing & Counterexample Review provides Trail of Bits' instructions for writing and reviewing generated tests across several languages. It includes five complete references covering strategy design, refactoring, review, failure interpretation and library selection. The upstream directory and license context are preserved at a fixed commit; our separate synthetic probe demonstrates a small part of the approach.

A practical starting point is an operation that promises to preserve information or obey a transformation rule. A decoder should recover the bytes its paired encoder accepts. Sorting should retain every input element and its multiplicity while putting elements in order. A Unicode canonicalizer should preserve canonical equivalence while producing a stable normalized result. Write down that promise before asking an agent to generate tests.

An assertion that merely completes without raising can miss data loss. An assertion copied from the implementation can share the same defect. A passing property should constrain behavior independently enough that a plausible incorrect implementation would fail. That does not require an independent implementation of the entire system: a multiset count can constrain a sorting routine without reproducing its algorithm.

Existing example tests still matter. Keep explicit regressions for boundaries and previous defects, and use generated cases to explore a defined neighborhood. Property-based testing is different from Vitest test organization or browser workflow testing; use the method that fits the actual contract rather than replacing every test with a generator.

Define the input domain explicitly

Our native fixture uses four bounded domains. Bytes are at most 512 bytes long. Lists contain at most 64 integers between -1,000 and 1,000. Generated text has at most 32 Unicode scalar characters and excludes surrogate code points. JSON objects have at most 16 such text keys and values restricted to bounded integers, booleans or scalar text. This is a deliberately small synthetic model, not the entire domain of an application parser.

Place ordinary constraints in the generator. A nonempty object strategy can produce a valid object directly; repeatedly generating empty objects and discarding them wastes test effort. A valid list index can be generated from the list's length. Constraints should express the promised input domain, not erase awkward inputs merely because they reveal a defect.

The distinction is visible in our error classification. A lone surrogate is excluded from the fixture's scalar-text domain. An attempted UTF-8 encoding of that deliberately out-of-domain string raises UnicodeEncodeError. We record that observation as a strategy/domain example, not an in-domain defect. Empty bytes, zero bytes and duplicate integers remain valid inputs and cannot be filtered out to make the properties pass.

A production contract may accept a different domain. A JSON service might allow null values, nested objects, floating-point values or duplicate keys in source text. A Unicode service might document a narrower character set or security-sensitive identifier rules. Add appropriate strategies and contracts for those features; this fixture does not cover them.

Check preservation as well as the final shape

For each operation, decide which information must survive. A sorted list can look ordered while losing duplicate values. A normalized string can be stable while being empty for every input. A JSON object can remain valid JSON after a field has vanished. Those outputs satisfy weak shape checks while violating the preservation promise.

Our four property families ran successfully in an isolated local environment:

Family	Bounded guarantee checked	Calls recorded	Distinct inputs recorded
Base64	Decoding the encoded bytes returns the original bytes	202	200
JSON over UTF-8	Serialization, UTF-8 encoding and parsing preserve the object	202	197
Sorting	Order, length and element multiplicities are preserved	202	201
Unicode NFC	Normalization is stable and preserves NFD canonical equivalence	202	201

Each count includes two explicit examples. The four families recorded 808 function calls, including repeated inputs; we do not call that 808 distinct samples. Settings target 200 generated successful examples per family and include an empty case plus a meaningful boundary. The measured counts describe this run only. A finite generated run is not a proof about every permitted input.

Python's strict Base64 decoding is also a separate setting. We used validate=True and checked five malformed encodings. Python's Base64 documentation distinguishes strict validation from the default behavior that can discard non-alphabet characters. Record the actual options rather than generalizing the result to every decoder mode.

Use deliberate defects to challenge the assertions

The probe contains four intentionally incorrect variants. They are negative controls written for this example, not defects discovered in Trail of Bits, Hypothesis or Python. Hypothesis found and reduced witnesses for each one:

Deliberate fixture defect	Recorded witness	Incorrect result
Strip trailing zero bytes before encoding	One byte with hexadecimal value 00	Empty bytes
Remove duplicates while sorting	[0, 0]	[0]
Remove the last JSON field	An object whose empty-string key has value 0	Empty object
Return an empty string for every normalizer input	The string 0	Empty string

These controls answer a narrow question: do the chosen preservation properties reject these particular plausible mistakes? They do not establish that the assertions detect every bug or constitute a full mutation-testing campaign. A different plausible defect may survive, which is a reason to reconsider the property or domain.

The constant normalizer illustrates the weakness of idempotence alone. Applying the constant function twice gives the same result as applying it once, so that check passes. The canonical-preservation check rejects the result for a nonempty witness. Combining properties can make a claim more informative without claiming completeness.

Do not mutate a production function just to demonstrate this lesson. The bundled negative controls operate on synthetic data within a standalone example script. They do not change a repository, send findings to maintainers, invoke a contract engine or modify a user's test suite.

Classify the shrunk failure against the contract

A small counterexample is evidence to investigate. Ask whether the witness lies inside the promised input domain and whether the claimed invariant is actually part of the operation's guarantee. Then distinguish an implementation defect from an incorrect assertion, an overbroad strategy or an unresolved specification question.

For example, losing a zero byte is a defect in our byte-preserving contract. Refusing an unsupported string representation can be an expected error path. Treating every possible Python string as valid UTF-8 would make the strategy broader than this fixture's chosen scalar-text contract. Suppressing that exception and announcing a clean run would obscure the distinction.

Preserve useful witnesses as explicit examples once the relevant defect and contract are understood. An example prevents a later generator distribution from being the only route to that boundary. Hypothesis' API reference documents explicit examples, generated tests and settings. A shrunk witness is not a ranking of production impact; prioritize a real issue using reachable code, the intended contract and the effect on users.

There are 15 named observations in our record: four property families, four negative controls, one demonstration of weak idempotence, five malformed Base64 errors, and one excluded-surrogate classification. The negative controls are counted as successful detection checks, not as a claim that their deliberately wrong functions are correct.

Reproduce the bounded native fixture

The package includes the original examples/probe_property_fixture.py and its recorded native-property-evidence.json. Inspect both files through the resource's file list before running them. The script creates its new result JSON alongside itself, uses synthetic inputs and does not read project files, credentials, browser state or production data. It requires Hypothesis in a separate environment.

Our recorded environment was Python 3.12.4 on Windows 11 with Hypothesis 6.168.3 and sortedcontainers 2.4.0. Use a Python version supported by your chosen Hypothesis release; the exact version identifies the observation, not a recommendation to retain an old interpreter indefinitely. Keep an existing project's dependency decision with its owner.

For an independent throwaway environment, the following command sequence illustrates installation of the recorded library and execution of the inspected example. Invoke that environment's Python explicitly; activation is optional and differs by operating system:

python -m venv pbt-fixture-env
<venv-python> -m pip install hypothesis==6.168.3
<venv-python> property-based-testing/examples/probe_property_fixture.py

The probe uses database=None, derandomize=True and no timing deadline. Those settings avoid saved-example state and wall-clock thresholds in this small pure fixture. Code changes, Python versions and library versions can still alter the generated run or reduced witnesses. Reproducibility requires preserving the program and settings as well as version information.

The executed probe SHA-256 is 3cd2309427870c92d528112f5de1f409062d53a088d1a4ead9a8811a27513f59. The bundled record binds that digest, the environment, example counts and counterexamples. The resource's download checksum binds the ZIP itself. These are different integrity checks, and neither proves that unrelated code or instructions are safe.

Inspect the skill package and its license

We verified 13 upstream files against the pinned Git blobs and SHA-256 values. The package preserves all nine files in the skill directory, including its README, display metadata, asset and five references, plus four license/source-context files. Six local Markdown links from the original skill and references resolve. Repository-level relative links in the source context refer to the fixed upstream repository rather than a complete marketplace installation.

BB Skills adds source notes, review metadata, the original synthetic probe and its recorded result. The ZIP contains 17 files. The upstream AI evaluation harnesses, extra prompts, smart-contract fixtures and larger plugin runtime are excluded. Included display metadata is not an executed client integration.

The fixed upstream source uses CC BY-SA 4.0. The complete legal text and distinct publisher license grant are bundled, along with source links and attribution to Trail of Bits and contributors. The additions described above are also offered under CC BY-SA 4.0. Keep required credits, indicate adaptations and observe ShareAlike; a free instruction download does not erase redistribution conditions. Names and brand assets do not imply endorsement. Read the license terms.

The package review's runtime_tested flag remains false. The separate scenario says what the native Hypothesis fixture exercised. We did not invoke an AI client, evaluate the complete skill, run upstream model harnesses, test other language libraries, execute Solidity invariants or certify a production application. No paid model, resource purchase or cloud service was used.

Decide what the evidence supports next

Use this resource when you can state a preservation, inverse, canonicalization or state guarantee and can build valid inputs for it. Use an example test when the behavior is best described by a few specific cases, and use browser, integration or security review tools for different questions. The five references can help expose a pure calculation seam before deciding that the code has no useful property.

Before applying a generated test to a real repository, identify the contract, constrain the domain, inspect its assertions, retain boundaries, and challenge the property with plausible intentional defects in a disposable fixture. Review the resulting failures and the parts left unexplored. A large green count accompanied by vague claims is weaker evidence than a small reproducible test with a precise conclusion.

AI assistance was used to draft this guide and prepare the original fixture. BB Skills executed the bounded native program, verified the package and reviewed the reported observations. We did not run an AI client or a full skills benchmark. The guide and the independently authored example explain their scope rather than suggesting universal compatibility or safety. Material adapting the CC BY-SA source, including this attributed guide and packaged additions, is shared under CC BY-SA 4.0.