If your fuzz test requires setup/teardown logic that is too expensive to be embedded in your property function and executed on each fuzz test iteration, you can use a test fixture. In FuzzTest, any default-constructible class can serve as a test fixture. To instantiate it, use the macro FUZZ_TEST_F, which corresponds to FUZZ_TEST in the same way that TEST_F corresponds to TEST in GoogleTest. Furthermore, you can easily adapt and reuse the existing GoogleTest fixtures in your fuzz tests.
Most best practices for writing unit tests (https://abseil.io/tips/122) apply equally well when writing fuzz tests. In view of that, you should generally avoid fixtures and prefer standalone helper functions for your fuzz test's initialization. However, in some situations test fixtures are appropriate.
If your fuzz test requires expensive setup, e.g., initializing a server, doing it in the property function may be prohibitively expensive. The fuzzing engine calls the property function once per fuzz test iteration, and running the setup each time may slow down fuzzing to a point where it becomes ineffective.
If you want to add a fuzz test to an existing code base that uses test fixtures for unit tests, refactoring the code to eliminate fixtures may be impractical or even impossible.
Suppose we wanted to test that a certain HTTP server called EchoServer echoes back the string it receives. We could do this with a fuzz test that doesn't use fixtures.
// A standalone property function that tests the property of interest. void ReturnsTheSameString(const std::string& request) { // Initialize the server. EchoServer server; server.Start("localhost:9999"); // Test the property. std::string response; SendRequest("localhost:9999", request, &response); EXPECT_EQ(response, request); // Tear down the server. server.Stop(); } // Instantiate the fuzz test using the FUZZ_TEST macro with a chosen suite name // and the property function as the test name. FUZZ_TEST(EchoServerFuzzTest, ReturnsTheSameString) .WithDomains(fuzztest::Arbitrary<std::string>()) .WithSeeds({"Hello"});
The issue with this fuzz test is that the server initialization and teardown happen in each fuzz test iteration, that is, in each call to the property function. If starting and stopping the server takes non-trivial time, the fuzz test will be slow and ineffective.
To improve the test, we would like to do the initialization and teardown once and reuse the same server across all fuzz test iterations. We can do this using a test fixture:
// In FuzzTest, any default-constructible class can be a test fixture. class EchoServerFuzzTest { public: // The constructor initializes the fixture. EchoServerFuzzTest() { server_.Start("localhost:9999"); } // The destructor tears down the fixture. ~EchoServerFuzzTest() { server_.Stop(); } // The fuzz test's property function must be a public member of the fixture. void ReturnsTheSameString(const std::string& request) { std::string response; SendRequest("localhost:9999", request, &response); EXPECT_EQ(response, request); } private: EchoServer server_; }; // Instantiate the fuzz test using the FUZZ_TEST_F macro, with the fixture as // the suite name and the property function as the test name. FUZZ_TEST_F(EchoServerFuzzTest, ReturnsTheSameString) // The macro supports the FUZZ_TEST clauses for specifying domains and seeds. .WithDomains(fuzztest::Arbitrary<std::string>()) .WithSeeds({"Hello"});
FuzzTest will instantiate the fixture once and call the property function on the same object in each fuzz test iteration.
While persisting state across fuzz test iterations can be effective, it can also have unwanted consequences. Consider the following example.
class MyFuzzTest { public: void MyProperty(int n) { MyApi(n, flag_); flag_ = !flag_; } private: bool flag_ = false; }; FUZZ_TEST_F(MyFuzzTest, MyProperty);
By mutating flag_ and using it in the call to MyApi(), the test no longer depends only on the input generated by the fuzzer. This can potentially break the fuzzer's assumptions about how the inputs affect coverage and make fuzzing ineffective. Additionally, a crashing input found by the fuzzer may not be sufficient for reproducing the crash, since the crash also depends on a particular value of flag_.
BEST PRACTICE: Make sure that the property function resets the mutated fixture so that each fuzz test iteration starts in the same state.
Note: Only use GoogleTest fixtures if they are shared with existing unit tests, or if you need static per-fixture setup and teardown (via SetUpTestSuite and TearDownTestSuite). Otherwise, prefer fixtureless code with standalone initialization functions.
If you already use a GoogleTest fixture derived from ::testing::Test in your unit tests and you want to reuse it in fuzz tests, you first need to choose when you want the fixture to be instantiated and destroyed. GoogleTest fixtures are typically built around a guarantee that the same fixture object will never be reused for multiple tests or multiple runs of the same test. In a fuzz test, this guarantee requires instantiating the fixture once per fuzz test iteration, i.e., calling the property function each time on a fresh fixture object. However, for increased performance, it is possible to instantiate a fixture once per fuzz test (if the fixture supports such persistence) and reuse the same fixture object in all calls to the property function. As noted earlier, this comes with the risk of state mutation, which can lead to non-reproducible crashes, so design your fixture carefully.
Adapt a fixture called Fixture by extending ::fuzztest::PerIterationFixtureAdapter<Fixture>. This case should be the more common one, since it observes the GoogleTest invariant of not using a fixture object more than once.
As an example, suppose we have a function SumVec() that returns the sum of a vector of integers and we have the following fixture.
class SumVecTest : public testing::Test { public: SumVecTest() : vec_{1, 2, 3} {} protected: std::vector<int> vec_; };
A fuzz test that uses the fixture would look like the following:
// Adapts the fixture using the "per-iteration" semantics. class SumVecFuzzTest : public fuzztest::PerIterationFixtureAdapter<SumVecTest> { public: void SumsLastEntry(int last_entry) { int previous_sum = SumVec(vec_); vec_.push_back(last_entry); EXPECT_EQ(SumVec(vec_), previous_sum + last_entry); } }; FUZZ_TEST_F(SumVecFuzzTest, SumsLastEntry);
The “per-iteration” semantics is appropriate for this fixture because the test mutates vec_.
Adapt a fixture called Fixture by extending ::fuzztest::PerFuzzTestFixtureAdapter<Fixture>. This case should be the less common one: it relies on the fixture being easily resettable, which is not a typical concern when designing GoogleTest fixtures.
As an example, let's return to EchoServer. Suppose we already have a GoogleTest fixture that sets up the server.
class EchoServerTest : public testing::Test { public: EchoServerTest() { server_.Start("localhost:9999"); } ~EchoServerTest() override { server_.Stop(); } private: EchoServer server_; };
We would write a test using this fixture as in the following code:
// Adapts the fixture using the "per-fuzz-test" semantics. class EchoServerFuzzTest : public fuzztest::PerFuzzTestFixtureAdapter<EchoServerTest> { public: void ReturnsTheSameString(const std::string& request) { std::string response; SendRequest("localhost:9999", request, &response); EXPECT_EQ(response, request); } }; FUZZ_TEST_F(EchoServerFuzzTest, ReturnsTheSameString);
Here, the “per-fuzz-test” semantics is appropriate because the test doesn't mutate the server in any way that would be relevant for the test, and the server initialization is too expensive to be performed in each iteration.
When using non-GoogleTest fixtures, FuzzTest can make use of runner functions on the fixture to specify per-test or per-iteration setups in a more flexible way. FuzzTest provides the following runner interfaces for a fixture to inherit:
::fuzztest::FuzzTestRunnerFixture, which declares the runner function void FuzzTestRunner(absl::AnyInvocable<void() &&> run_test)
If the interface is inherited, instead of directly running the fuzz test, FuzzTest will call FuzzTestRunner, expecting that it will in turn:
run_test, through which FuzzTest will essentially pass the whole fuzzing loop with multiple invocations to the property function. The function run_test will return once FuzzTest is done with the test.::fuzztest::IterationRunnerFixture, which declares the runner function void FuzzTestIterationRunner(absl::AnyInvocable<void() &&> run_iteration)
If the interface is inherited, instead of directly running a fuzz test iteration, FuzzTest will call FuzzTestIterationRunner, expecting that it will in turn:
run_iteration, through which FuzzTest will invoke the property function once. The function run_iteration will return once the iteration is done.Note that the type absl::AnyInvocable<void() &&> ensures that the functions run_test and run_iteration can be called only once. To call them, you must first move them, for example std::move(run_test)(). Attempting to call these functions more than once will result in a runtime error.
The traditional SetUp/TearDown functions of a fixture can be trivially transformed into a runner function. For example, the EchoServerFuzzTest fixture could be defined as:
class EchoServerFuzzTest : public fuzztest::FuzzTestRunnerFixture { public: void FuzzTestRunner(absl::AnyInvocable<void() &&> run_test) override { server_.Start("localhost:9999"); std::move(run_test)(); server_.Stop(); } // The fuzz test's property function void ReturnsTheSameString(const std::string& request) { std::string response; SendRequest("localhost:9999", request, &response); EXPECT_EQ(response, request); } private: EchoServer server_; }; FUZZ_TEST_F(EchoServerFuzzTest, ReturnsTheSameString);
To create a new EchoServer instance for each fuzz test iteration, define FuzzTestIterationRunner instead:
class EchoServerFuzzTest : public fuzztest::IterationRunnerFixture { public: void FuzzTestIterationRunner(absl::AnyInvocable<void() &&> run_iteration) override { server_.Start("localhost:9999"); std::move(run_iteration)(); server_.Stop(); } ...
Although converting SetUp/TearDown to runner functions is trivial, the other way around may not even be possible. For example, suppose there is a fuzz test whose property function must run within an SUT created by an SUT method RunWithinSut(sut_ready_cb), which starts the SUT and calls the callback when the SUT is ready, and shuts down the SUT when the callback returns. Suppose additionally that RunWithinSut must be run in the main thread of the test, which is also the thread where FuzzTest runs the test fixture. Given these constraints, it is impossible to run such test with a SetUp/TearDown-style fixture without calling RunWithinSut for every iteration, which may be too heavy and impact the fuzzing performance. With runner functions, the fixture can be defined as:
class SutBasedFuzzTest : public fuzztest::FuzzTestRunnerFixture { public: void FuzzTestRunner(absl::AnyInvocable<void() &&> run_test) { sut_.RunWithinSut([&] { std::move(run_test)(); });`. } private: SUT sut_; }