Skip to main content

Adaptive fuzzing for property-based tests

Project description

HypoFuzz

Adaptive fuzzing of Hypothesis tests.

Property-based approaches help you to write better tests which find more bugs, but don't have great ways to exchange much more CPU time for more bugs. The goal of this project is to bring togther the best parts of fuzzing and PBT.

Motivation

You can run a traditional fuzzer like AFL on Hypothesis tests to get basic coverage guidance.

  • This does actually work, which is pretty cool
  • It's very slow though, and often fails to parse the bytes into an example
  • Installing, configuring, and connecting all the parts is a pain
  • Also assumes one fuzz target per core, which doesn't scale very far

Alternatively, you can just run Hypothesis with a large max_examples setting. This also works pretty well, but doesn't get the benefits of coverage guidance (i.e. avoiding the exponential scaling cliff by learning from feedback) and also occupies one fuzz target per core.

(turns out that you can emulate coverage guidance with hypothesis.target(), which appears to work well enough as a starting point)

(also Hypothesis used to have coverage guidance built in, but we took it back out because of performance and ecosystem integration problems - as a rule of thumb it's just not worth the trouble for less than a thousand inputs. see here.)

Features

  • Interleave execution of many test functions
  • Prioritise functions where we expect to make progress
  • Coverage-guided exploration of your system-under-test
  • Seamless python-native and CLI integrations
  • Web-based time-travel debugging with PyTrace (automatic if you pip install hypofuzz[pytrace])

Changelog

Patch notes can be found in CHANGELOG.md.

License

This is an active research project as part of my (Zac Hatfield-Dodds) PhD.

Unlike Hypothesis, it is not open source and I am not seeking external contributions.

As a complement to users of free, world-class PBT tools, I'm planning to sell licenses in order to fund ongoing work on both this project and Hypothesis itself. Please contact me if you are interested.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

hypofuzz-21.12.2.tar.gz (22.0 kB view hashes)

Uploaded Source

Built Distribution

hypofuzz-21.12.2-py3-none-any.whl (23.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page