Implement seeded domains by adding `.WithSeeds()` to domains. (#180)
Implement seeded domains by adding `.WithSeeds()` to domains.
Seeded domains occasionally return one of the seeds as a result of
`.Init()`.
NOTE: The seeds are accepted as user values and converted to corpus
values using
`.FromValue()`. As a consequence, setting the seeds won't work for
domains for
which `.FromValue()` fails (e.g., `MapOf`, `FlatMap`, and anything
derived from
them like `Arbitrary<absl::Duration>`). Moreover, currently
`.FromValue()`
doesn't validate the inputs, so it is possible to store a seed that
doesn't
meet the domain's constraints (e.g., `InRange(0, 10).WithSeeds({42})`
works).
This will be fixed separately.
The goal of the added unit tets is to almost always succeed when the
functionality is implemented and almost always fail when the
functionality is
missing. For most domains this usually means that it is sufficient to
check that
one or two seeds appear at least once among 1000 generated values, since
it is
unlikely that those seeds would have been generated otherwise. But this
is not
sufficient for domains with few elements. For example, for
`Arbitrary<bool>` we
are surely going to see `true` generated at least once regardless of
whether it
is a seed or not. For such domains I derived lower bounds for the number
of seed
appearances using the [Hoeffding's inequality][1] so that the tests are
robuts.
[1]: https://en.wikipedia.org/wiki/Binomial_distribution#Tail_bounds
Co-authored-by: Filip Niksic <fniksic@google.com>FuzzTest is a C++ testing framework for writing and executing fuzz tests, which are property-based tests executed using coverage-guided fuzzing under the hood. Fuzz tests are like regular unit tests, but more generic and more powerful. Instead of saying: “for this specific input, we expect this specific output”, we can say: “for these types of input, we expect this generic property to be true”. For example:
void MyApiAlwaysSucceedsOnPositiveIntegers(int i) { bool success = MyApi(i); EXPECT_TRUE(success); } FUZZ_TEST(MyApiTest, MyApiAlwaysSucceedsOnPositiveIntegers) .WithDomains(/*i:*/fuzztest::Positive<int>());
It is our latest fuzz testing technology and the successor of previously used fuzzing tools, such as libFuzzer. It allows you to write powerful fuzz tests more easily than with previously used fuzz targets. You can use it together with GoogleTest, or other unit testing frameworks, allowing you to write fuzz test side by side with regular unit tests, and just as easily.
It is a first-of-its-kind tool that bridges the gap between fuzzing and property-based testing, as it is both:
FuzzTest is for everyone who writes C++ code. (Currently, only C++ is supported.) Fuzz testing is a proven testing technique that has found tens of thousands of bugs. With the FuzzTest framework writing these tests becomes a breeze. Because fuzz tests are more generic, they are more powerful than regular unit tests. They can find tricky edge cases automatically for us, edge cases that most likely we would never think of.
You can write fuzz tests as easily as you write unit tests using GoogleTest for example. Simply use the FUZZ_TEST macro like you would use GoogleTest's TEST macro.
At Google, FuzzTest is widely used and software engineers love it. It has replaced the old style of writing fuzz targets.
To get started, read the Quickstart with Bazel or Quickstart with CMake, then take a look at the Overview and the Codelab.
Once you have a high level understanding about fuzz tests, consider reading the rest of the documentation, including the:
If you have a question or encounter a bug, please file an issue on GitHub.