{ }
Resource profile / Property-Based Testing & Counterexample Review
About this skill

Workflow & requirements

Property-Based Testing

An example test asserts one point. A property asserts a rule over the whole input domain and lets the generator hunt for the counterexample. That trade is worth making when the code has an algebraic shape — an inverse, an invariant, an oracle — and not otherwise. Code with no such shape gets example tests; saying so is a valid outcome.

Check first whether the shape is missing or merely buried. A calculation wrapped in I/O, a string built by concatenation, an in-place mutation — each has a property and no seam to assert it through. See references/refactoring.md before concluding there is nothing to assert.

Property catalog

Property Formula Where it applies
Roundtrip decode(encode(x)) == x Serialization, conversion pairs
Inverse f(g(x)) == x encrypt/decrypt, compress/decompress
Oracle new(x) == reference(x) Optimization, refactoring, reimplementation
Idempotence f(f(x)) == f(x) Normalization, formatting, sorting
Invariant Holds before and after Any transformation, contract state
Easy to verify is_sorted(sort(x)) Complex algorithms with cheap checkers
Commutativity f(a, b) == f(b, a) Binary and set operations
Associativity f(f(a,b), c) == f(a, f(b,c)) Combining operations
Identity f(x, e) == x Operations with a neutral element

Strength ordering, weakest to strongest: no crash → type preservation → invariant → idempotence → roundtrip / oracle.

Assert the strongest property the code supports. "No crash" alone rarely justifies the dependency — if that is all you can find, either a small rearrangement exposes something stronger, or the honest report is that this code is a poor PBT candidate. Rule out the first before settling for the second.

The two ways a property test asserts nothing

  • Tautology. assert add(a, b) == a + b restates the implementation; no bug they share can fail it. Pick a property that constrains the function without recomputing it. Note the exception: f(x) == f(x) is a genuine determinism property when f is not obviously pure — serializers over dicts or sets, hashing, anything reading the clock.
  • Vacuity. assume() that filters out nearly every input passes without exercising anything, and self-contradictory assume() passes having run zero cases. Push constraints into the strategy so the generator produces valid inputs directly.

Where to look next

Load the one that matches the task in front of you:

Task File
Writing new tests, designing strategies references/generating.md
The code has no property to assert yet references/refactoring.md
Reviewing existing property tests references/reviewing.md
A property test just failed references/interpreting-failures.md
Library choice, Echidna and Medusa references/libraries.md

Introducing PBT to a project that lacks it

If the project already uses a PBT library, just write the tests in it. If it does not, adding one is a dependency decision that belongs to the user — offer it once with the specific property you would write, and take the answer either way.

PACKAGE TRANSPARENCY

Inspect before installing

Source: Trail of Bits · CC-BY-SA-4.0 · SHA-256 shown alongside the download.

17 files41381 ZIP bytes1 script/code file

License file included. A license and checksum are not a security certification. Review package instructions and scripts before running them.

View files and uncompressed sizes
Machine-readable installation guide →
CATALOG REVIEW NOTES

Know what you need before installing

Source and packaging checks recorded on 2026-10-04. These notes are not safety certification or measured task performance.

Requirements

Instruction-only property-testing workflow with five complete references. Use a suitable existing language-specific library; any new project dependency is a separate owner decision. No AI client, full skill invocation, upstream evaluation harness or smart-contract runtime was executed.

Costs, access & practical limits

Free CC BY-SA 4.0 instruction download with attribution, source/license links, change notices and ShareAlike. Complete legal text and publisher grant are included. No model API, paid resource or cloud service is used by inspection; external services can have separate prerequisites and costs.

View the recorded checks
  • All 13 original source files match pinned Git blobs and SHA-256
  • All nine skill files and five references preserved
  • All local skill/reference/README markdown links resolve
  • Complete CC BY-SA legal text and separate publisher grant preserved
  • AI evaluation harnesses and contract fixtures excluded
  • Source instructions and scoped synthetic testing are distinct from full AI skill evaluation

Upstream commit: 82fe8226252622fa807643bdca1710901198553a

Runtime status: not tested by this catalog. Configure your client and test the skill in your own environment.

LICENSE & ATTRIBUTION

CC BY-SA license & attribution

This instruction package retains the complete upstream legal text and the publisher's license grant. Keep attribution and license links, indicate changes, and follow ShareAlike when distributing adaptations. The download is not offered under MIT or Apache terms.

CC-BY-SA-4.0 ↗

Full CC BY-SA 4.0 legal text for the preserved skill instructions, references, README, metadata, asset and source context.

Attribution: Trail of Bits and the Skills repository contributors. Original credits and links are preserved.

Changes: All 13 selected upstream files retained byte for byte. Source-context names relocated. BB Skills adds source notes, review metadata, an original synthetic probe and its recorded evidence under CC BY-SA 4.0.

Included notice: property-based-testing/LICENSE.upstream.txt

Declared source ↗

CC-BY-SA-4.0 ↗

Separate upstream publisher grant and attribution.

Attribution: Trail of Bits and contributors; https://www.trailofbits.com/.

Changes: README unchanged; repository-level relative links refer to the fixed upstream repository.

Included notice: property-based-testing/source-context/README.repository.upstream.md

Declared source ↗

For the CC BY-SA material, retain attribution and license links, indicate changes, and apply the required ShareAlike terms to adaptations. These notices do not imply endorsement by the original creators.

SCENARIOS

Inputs, criteria and recorded outcomes

Records are supplied by the site administrator and bound to a specific package. They are not third-party safety certification. This page does not execute skills.

Native Hypothesis property and deliberate-defect fixture

Reported passed · v82fe82262526.bb1

View input and acceptance criteria

Input

In a separate local Python environment use synthetic bounded byte strings, Unicode scalar text, JSON objects and integer lists to check roundtrips, sorting preservation and canonical normalization. Challenge four deliberately wrong variants, record shrunk witnesses, check five malformed Base64 error paths and classify an excluded surrogate. Do not invoke an AI client, upstream harness, contract engine or production service.

Acceptance criteria

Four native property families record 202 calls each including explicit cases. Four deliberately wrong fixtures are rejected with witnesses; weak idempotence alone is demonstrated, five invalid encodings rejected and an out-of-domain surrogate classified. These 15 observations are bounded native checks, not a full skill or production security evaluation.

Recorded outcome

Native Hypothesis on four synthetic property families and four intentional negative controls, five invalid encodings and an out-of-domain surrogate. No AI client, whole skill, upstream harness, smart-contract engine, paid model or production input.

{
  "executed_at": "2026-10-04T08:41:09.266631+00:00",
  "python": "3.12.4",
  "platform": "Windows-11-10.0.26300-SP0",
  "hypothesis": "6.168.3",
  "probe_sha256": "3cd2309427870c92d528112f5de1f409062d53a088d1a4ead9a8811a27513f59",
  "cases": [
    {
      "name": "base64_roundtrip",
      "status": "passed",
      "examples_executed": 202,
      "distinct_inputs": 200,
      "domain": "bounded synthetic inputs; no production data"
    },
    {
      "name": "json_utf8_roundtrip",
      "status": "passed",
      "examples_executed": 202,
      "distinct_inputs": 197,
      "domain": "bounded synthetic inputs; no production data"
    },
    {
      "name": "sort_order_and_multiplicity",
      "status": "passed",
      "examples_executed": 202,
      "distinct_inputs": 201,
      "domain": "bounded synthetic inputs; no production data"
    },
    {
      "name": "NFC_idempotence_and_canonical_preservation",
      "status": "passed",
      "examples_executed": 202,
      "distinct_inputs": 201,
      "domain": "bounded synthetic inputs; no production data"
    },
    {
      "name": "negative_control_trailing_zero_loss",
      "status": "passed",
      "counterexample_hex": "00",
      "decoded_hex": "",
      "classification": "intentional fixture implementation defect"
    },
    {
      "name": "negative_control_sort_deduplication",
      "status": "passed",
      "counterexample": [
        0,
        0
      ],
      "actual": [
        0
      ],
      "classification": "intentional fixture implementation defect"
    },
    {
      "name": "negative_control_json_field_loss",
      "status": "passed",
      "counterexample": {
        "": 0
      },
      "actual": {},
      "classification": "intentional fixture implementation defect"
    },
    {
      "name": "negative_control_constant_normalizer",
      "status": "passed",
      "counterexample": "0",
      "actual": "",
      "classification": "intentional fixture implementation defect"
    },
    {
      "name": "weak_idempotence_accepts_constant",
      "status": "passed",
      "counterexample": "0",
      "meaning": "idempotence alone misses this preservation defect"
    },
    {
      "name": "strict_base64_invalid_3f",
      "status": "passed",
      "input_ascii": "?",
      "classification": "documented error path"
    },
    {
      "name": "strict_base64_invalid_41",
      "status": "passed",
      "input_ascii": "A",
      "classification": "documented error path"
    },
    {
      "name": "strict_base64_invalid_3d3d3d",
      "status": "passed",
      "input_ascii": "===",
      "classification": "documented error path"
    },
    {
      "name": "strict_base64_invalid_413d3d3d",
      "status": "passed",
      "input_ascii": "A===",
      "classification": "documented error path"
    },
    {
      "name": "strict_base64_invalid_25252525",
      "status": "passed",
      "input_ascii": "%%%%",
      "classification": "documented error path"
    },
    {
      "name": "surrogate_outside_scalar_text_domain",
      "status": "passed",
      "classification": "out-of-domain strategy example, not an in-domain encoder bug"
    }
  ],
  "observations": 15,
  "generated_families": 4,
  "property_examples": {
    "base64_roundtrip": 202,
    "json_utf8_roundtrip": 202,
    "sort_order_and_multiplicity": 202,
    "NFC_idempotence_and_canonical_preservation": 202
  },
  "native_fixture_tested": true,
  "runtime_tested": false,
  "scope": "Native Hypothesis on four synthetic property families and four intentional negative controls, five invalid encodings and an out-of-domain surrogate. No AI client, whole skill, upstream harness, smart-contract engine, paid model or production input."
}

Environment

Isolated local Windows fixture; Python 3.12.4; Hypothesis 6.168.3; sortedcontainers 2.4.0. Synthetic data only; no production project or database, network service, AI client or upstream harness. Probe 3cd2309427870c92d528112f5de1f409062d53a088d1a4ead9a8811a27513f59

Package SHA-256: 05c12075f2dd5e468a6892d2eb15ac5e6f1f6669aaa7089b65fcbef3fdab1740

Outcome recorded: 2026-10-04 08:41 UTC

Community reviews

★ New

Be the first to share your experience.

Sign in to leave a review →

Guides using this resource

All guides →
Practical guide

Review property-testing skills with real counterexamples

Review a property-testing skill by the guarantees its tests can falsify, the inputs it actually generates, and the failures it can explain. A large example count is useful…

By BB Skills · Read guide →

More to explore

View all ↗