doc/fixtures.md - third_party/github/google/fuzztest - Git at Google

 # Test Fixtures

 If your fuzz test requires setup/teardown logic that is too expensive to be
 embedded in your [property function](fuzz-test-macro.md#the-property-function)
 and executed on each fuzz test iteration, you can use a
 test fixture.
 In FuzzTest, any default-constructible class can serve as a test fixture.
 To instantiate it, use the macro
 FUZZ_TEST_F,
 which corresponds to [`FUZZ_TEST`](fuzz-test-macro.md) in the same way that
 `TEST_F` corresponds to `TEST` in GoogleTest. Furthermore, you can easily adapt
 and reuse the existing
 [GoogleTest fixtures](https://google.github.io/googletest/primer.html#same-data-multiple-tests)
 in your fuzz tests.

 [TOC]

 ## When to use fixtures in fuzz tests

 Most best practices for writing unit tests
 (https://abseil.io/tips/122)
 apply equally well when writing fuzz tests. In view of that, you should
 generally avoid fixtures and prefer standalone helper functions for your fuzz
 test's initialization. However, in some situations test fixtures are
 appropriate.

 1.  If your fuzz test requires expensive setup, e.g., initializing a server,
     doing it in the property function may be prohibitively expensive. The
     fuzzing engine calls the property function once per fuzz test iteration, and
     running the setup each time may slow down fuzzing to a point where it
     becomes ineffective.

 2.  If you want to add a fuzz test to an existing code base that uses test
     fixtures for unit tests, refactoring the code to eliminate fixtures may be
     impractical or even impossible.

 ## Using a fixture for expensive setup

 Suppose we wanted to test that a certain HTTP server called EchoServer echoes
 back the string it receives. We could do this with a fuzz test that doesn't use
 fixtures.

 ```c++
 // A standalone property function that tests the property of interest.
 void ReturnsTheSameString(const std::string& request) {
   // Initialize the server.
   EchoServer server;
   server.Start("localhost:9999");

   // Test the property.
   std::string response;
   SendRequest("localhost:9999", request, &response);
   EXPECT_EQ(response, request);

   // Tear down the server.
   server.Stop();
 }

 // Instantiate the fuzz test using the FUZZ_TEST macro with a chosen suite name
 // and the property function as the test name.
 FUZZ_TEST(EchoServerFuzzTest, ReturnsTheSameString)
     .WithDomains(fuzztest::Arbitrary<std::string>())
     .WithSeeds({"Hello"});
 ```

 The issue with this fuzz test is that the server initialization and teardown
 happen in each fuzz test iteration, that is, in each call to the property
 function. If starting and stopping the server takes non-trivial time, the fuzz
 test will be slow and ineffective.

 To improve the test, we would like to do the initialization and teardown once
 and reuse the same server across all fuzz test iterations. We can do this using
 a test fixture:

 ```c++
 // In FuzzTest, any default-constructible class can be a test fixture.
 class EchoServerFuzzTest {
  public:
   // The constructor initializes the fixture.
   EchoServerFuzzTest() { server_.Start("localhost:9999"); }

   // The destructor tears down the fixture.
   ~EchoServerFuzzTest() { server_.Stop(); }

   // The fuzz test's property function must be a public member of the fixture.
   void ReturnsTheSameString(const std::string& request) {
     std::string response;
     SendRequest("localhost:9999", request, &response);
     EXPECT_EQ(response, request);
   }

  private:
   EchoServer server_;
 };

 // Instantiate the fuzz test using the FUZZ_TEST_F macro, with the fixture as
 // the suite name and the property function as the test name.
 FUZZ_TEST_F(EchoServerFuzzTest, ReturnsTheSameString)
     // The macro supports the FUZZ_TEST clauses for specifying domains and seeds.
     .WithDomains(fuzztest::Arbitrary<std::string>())
     .WithSeeds({"Hello"});
 ```

 FuzzTest will instantiate the fixture once and call the property function
 on the same object in each fuzz test iteration.

 ### Note on mutable test fixtures

 While persisting state across fuzz test iterations can be effective, it can also
 have unwanted consequences. Consider the following example.

 ```c++
 class MyFuzzTest {
  public:
   void MyProperty(int n) {
     MyApi(n, flag_);
     flag_ = !flag_;
   }

  private:
   bool flag_ = false;
 };
 FUZZ_TEST_F(MyFuzzTest, MyProperty);
 ```

 By mutating `flag_` and using it in the call to `MyApi()`, the test no longer
 depends only on the input generated by the fuzzer. This can potentially break
 the fuzzer's assumptions about how the inputs affect coverage and make fuzzing
 ineffective. Additionally, a crashing input found by the fuzzer may not be
 sufficient for reproducing the crash, since the crash also depends on a
 particular value of `flag_`.

 BEST PRACTICE: Make sure that the property function resets the mutated fixture
 so that each fuzz test iteration starts in the same state.

 ## Using existing GoogleTest fixtures in fuzz tests

 Note: Only use GoogleTest fixtures if they are shared with existing unit tests,
 or if you need static per-fixture setup and teardown (via `SetUpTestSuite` and
 `TearDownTestSuite`). Otherwise, prefer fixtureless code with
 [standalone initialization functions](https://abseil.io/tips/122).

 If you already use a GoogleTest fixture derived from `::testing::Test` in your
 unit tests and you want to reuse it in fuzz tests, you first need to choose when
 you want the fixture to be instantiated and destroyed. GoogleTest fixtures are
 typically built around a guarantee that the same fixture object will never be
 reused for multiple tests or multiple runs of the same test. In a fuzz test,
 this guarantee requires instantiating the fixture *once per fuzz test
 iteration*, i.e., calling the property function each time on a fresh fixture
 object. However, for increased performance, it is possible to instantiate a
 fixture *once per fuzz test* (if the fixture supports such persistence) and
 reuse the same fixture object in all calls to the property function. As noted
 earlier, this comes with the risk of state mutation, which can lead to
 non-reproducible crashes, so design your fixture carefully.

 ### Semantics 1: Fixture instantiated once per fuzz test iteration

 Adapt a fixture called `Fixture` by extending
 `::fuzztest::PerIterationFixtureAdapter<Fixture>`. This case should be the more
 common one, since it observes the GoogleTest invariant of not using a fixture
 object more than once.

 As an example, suppose we have a function `SumVec()` that returns the sum of a
 vector of integers and we have the following fixture.

 ```c++
 class SumVecTest : public testing::Test {
  public:
   SumVecTest() : vec_{1, 2, 3} {}

  protected:
   std::vector<int> vec_;
 };
 ```

 A fuzz test that uses the fixture would look like the following:

 ```c++
 // Adapts the fixture using the "per-iteration" semantics.
 class SumVecFuzzTest : public fuzztest::PerIterationFixtureAdapter<SumVecTest> {
  public:
   void SumsLastEntry(int last_entry) {
     int previous_sum = SumVec(vec_);
     vec_.push_back(last_entry);
     EXPECT_EQ(SumVec(vec_), previous_sum + last_entry);
   }
 };
 FUZZ_TEST_F(SumVecFuzzTest, SumsLastEntry);
 ```

 The "per-iteration" semantics is appropriate for this fixture because the test
 mutates `vec_`.

 ### Semantics 2: Fixture instantiated once per fuzz test

 Adapt a fixture called `Fixture` by extending
 `::fuzztest::PerFuzzTestFixtureAdapter<Fixture>`. This case should be the less
 common one: it relies on the fixture being easily resettable, which is not a
 typical concern when designing GoogleTest fixtures.

 As an example, let's return to EchoServer. Suppose we already have a GoogleTest
 fixture that sets up the server.

 ```c++
 class EchoServerTest : public testing::Test {
  public:
   EchoServerTest() { server_.Start("localhost:9999"); }
   ~EchoServerTest() override { server_.Stop(); }

  private:
   EchoServer server_;
 };
 ```

 We would write a test using this fixture as in the following code:

 ```c++
 // Adapts the fixture using the "per-fuzz-test" semantics.
 class EchoServerFuzzTest
     : public fuzztest::PerFuzzTestFixtureAdapter<EchoServerTest> {
  public:
   void ReturnsTheSameString(const std::string& request) {
     std::string response;
     SendRequest("localhost:9999", request, &response);
     EXPECT_EQ(response, request);
   }
 };
 FUZZ_TEST_F(EchoServerFuzzTest, ReturnsTheSameString);
 ```

 Here, the "per-fuzz-test" semantics is appropriate because the test doesn't
 mutate the server in any way that would be relevant for the test, and the server
 initialization is too expensive to be performed in each iteration.

 ## Advanced fixtures with runner functions

 When using non-GoogleTest fixtures, FuzzTest can make use of *runner functions*
 on the fixture to specify per-test or per-iteration setups in a more flexible
 way. FuzzTest provides the following runner interfaces for a fixture to inherit:

 -   `::fuzztest::FuzzTestRunnerFixture`, which declares the runner function
     `void FuzzTestRunner(absl::AnyInvocable<void() &&> run_test)`

     If the interface is inherited, instead of directly running the fuzz test,
     FuzzTest will call `FuzzTestRunner`, expecting that it will in turn:

     1.  Set up everything needed for running the test.
     2.  Invoke `run_test`, through which FuzzTest will essentially pass the
         whole fuzzing loop with multiple invocations to the property function.
         The function `run_test` will return once FuzzTest is done with the test.
     3.  Tear down everything no longer needed after the test.

 -   `::fuzztest::IterationRunnerFixture`, which declares the runner function
     `void FuzzTestIterationRunner(absl::AnyInvocable<void() &&> run_iteration)`

     If the interface is inherited, instead of directly running a fuzz test
     iteration, FuzzTest will call `FuzzTestIterationRunner`, expecting that it
     will in turn:

     1.  Set up everything needed for running a test iteration.
     2.  Invoke `run_iteration`, through which FuzzTest will invoke the property
         function once. The function `run_iteration` will return once the
         iteration is done.
     3.  Tear down everything no longer needed after the test iteration.

 Note that the type `absl::AnyInvocable<void() &&>` ensures that the functions
 `run_test` and `run_iteration` can be called only once. To call them, you must
 first move them, for example `std::move(run_test)()`. Attempting to call these
 functions more than once will result in a runtime error.

 The traditional SetUp/TearDown functions of a fixture can be trivially
 transformed into a runner function. For example, the `EchoServerFuzzTest`
 fixture could be defined as:

 ```c++
 class EchoServerFuzzTest : public fuzztest::FuzzTestRunnerFixture {
  public:

   void FuzzTestRunner(absl::AnyInvocable<void() &&> run_test) override {
    server_.Start("localhost:9999");
    std::move(run_test)();
    server_.Stop();
   }

   // The fuzz test's property function
   void ReturnsTheSameString(const std::string& request) {
     std::string response;
     SendRequest("localhost:9999", request, &response);
     EXPECT_EQ(response, request);
   }

  private:
   EchoServer server_;
 };

 FUZZ_TEST_F(EchoServerFuzzTest, ReturnsTheSameString);
 ```

 To create a new `EchoServer` instance for each fuzz test iteration, define
 `FuzzTestIterationRunner` instead:

 ```c++
 class EchoServerFuzzTest : public fuzztest::IterationRunnerFixture {
  public:

   void FuzzTestIterationRunner(absl::AnyInvocable<void() &&> run_iteration) override {
    server_.Start("localhost:9999");
    std::move(run_iteration)();
    server_.Stop();
   }

   ...
 ```

 Although converting SetUp/TearDown to runner functions is trivial, the other way
 around may not even be possible. For example, suppose there is a fuzz test whose
 property function must run within an SUT created by an SUT method
 `RunWithinSut(sut_ready_cb)`, which starts the SUT and calls the callback when
 the SUT is ready, and shuts down the SUT when the callback returns. Suppose
 additionally that `RunWithinSut` must be run in the main thread of the test,
 which is also the thread where FuzzTest runs the test fixture. Given these
 constraints, it is impossible to run such test with a SetUp/TearDown-style
 fixture without calling `RunWithinSut` for every iteration, which may be too
 heavy and impact the fuzzing performance. With runner functions, the fixture can
 be defined as:

 ```c++
 class SutBasedFuzzTest : public fuzztest::FuzzTestRunnerFixture {
  public:

   void FuzzTestRunner(absl::AnyInvocable<void() &&> run_test) {
    sut_.RunWithinSut([&] { std::move(run_test)(); });`.
   }

  private:
   SUT sut_;
 }
 ```
	# Test Fixtures

	If your fuzz test requires setup/teardown logic that is too expensive to be
	embedded in your [property function](fuzz-test-macro.md#the-property-function)
	and executed on each fuzz test iteration, you can use a
	test fixture.
	In FuzzTest, any default-constructible class can serve as a test fixture.
	To instantiate it, use the macro
	FUZZ_TEST_F,
	which corresponds to [`FUZZ_TEST`](fuzz-test-macro.md) in the same way that
	`TEST_F` corresponds to `TEST` in GoogleTest. Furthermore, you can easily adapt
	and reuse the existing
	[GoogleTest fixtures](https://google.github.io/googletest/primer.html#same-data-multiple-tests)
	in your fuzz tests.

	[TOC]

	## When to use fixtures in fuzz tests

	Most best practices for writing unit tests
	(https://abseil.io/tips/122)
	apply equally well when writing fuzz tests. In view of that, you should
	generally avoid fixtures and prefer standalone helper functions for your fuzz
	test's initialization. However, in some situations test fixtures are
	appropriate.

	1. If your fuzz test requires expensive setup, e.g., initializing a server,
	doing it in the property function may be prohibitively expensive. The
	fuzzing engine calls the property function once per fuzz test iteration, and
	running the setup each time may slow down fuzzing to a point where it
	becomes ineffective.

	2. If you want to add a fuzz test to an existing code base that uses test
	fixtures for unit tests, refactoring the code to eliminate fixtures may be
	impractical or even impossible.

	## Using a fixture for expensive setup

	Suppose we wanted to test that a certain HTTP server called EchoServer echoes
	back the string it receives. We could do this with a fuzz test that doesn't use
	fixtures.

	```c++
	// A standalone property function that tests the property of interest.
	void ReturnsTheSameString(const std::string& request) {
	// Initialize the server.
	EchoServer server;
	server.Start("localhost:9999");

	// Test the property.
	std::string response;
	SendRequest("localhost:9999", request, &response);
	EXPECT_EQ(response, request);

	// Tear down the server.
	server.Stop();
	}

	// Instantiate the fuzz test using the FUZZ_TEST macro with a chosen suite name
	// and the property function as the test name.
	FUZZ_TEST(EchoServerFuzzTest, ReturnsTheSameString)
	.WithDomains(fuzztest::Arbitrary<std::string>())
	.WithSeeds({"Hello"});
	```

	The issue with this fuzz test is that the server initialization and teardown
	happen in each fuzz test iteration, that is, in each call to the property
	function. If starting and stopping the server takes non-trivial time, the fuzz
	test will be slow and ineffective.

	To improve the test, we would like to do the initialization and teardown once
	and reuse the same server across all fuzz test iterations. We can do this using
	a test fixture:

	```c++
	// In FuzzTest, any default-constructible class can be a test fixture.
	class EchoServerFuzzTest {
	public:
	// The constructor initializes the fixture.
	EchoServerFuzzTest() { server_.Start("localhost:9999"); }

	// The destructor tears down the fixture.
	~EchoServerFuzzTest() { server_.Stop(); }

	// The fuzz test's property function must be a public member of the fixture.
	void ReturnsTheSameString(const std::string& request) {
	std::string response;
	SendRequest("localhost:9999", request, &response);
	EXPECT_EQ(response, request);
	}

	private:
	EchoServer server_;
	};

	// Instantiate the fuzz test using the FUZZ_TEST_F macro, with the fixture as
	// the suite name and the property function as the test name.
	FUZZ_TEST_F(EchoServerFuzzTest, ReturnsTheSameString)
	// The macro supports the FUZZ_TEST clauses for specifying domains and seeds.
	.WithDomains(fuzztest::Arbitrary<std::string>())
	.WithSeeds({"Hello"});
	```

	FuzzTest will instantiate the fixture once and call the property function
	on the same object in each fuzz test iteration.

	### Note on mutable test fixtures

	While persisting state across fuzz test iterations can be effective, it can also
	have unwanted consequences. Consider the following example.

	```c++
	class MyFuzzTest {
	public:
	void MyProperty(int n) {
	MyApi(n, flag_);
	flag_ = !flag_;
	}

	private:
	bool flag_ = false;
	};
	FUZZ_TEST_F(MyFuzzTest, MyProperty);
	```

	By mutating `flag_` and using it in the call to `MyApi()`, the test no longer
	depends only on the input generated by the fuzzer. This can potentially break
	the fuzzer's assumptions about how the inputs affect coverage and make fuzzing
	ineffective. Additionally, a crashing input found by the fuzzer may not be
	sufficient for reproducing the crash, since the crash also depends on a
	particular value of `flag_`.

	BEST PRACTICE: Make sure that the property function resets the mutated fixture
	so that each fuzz test iteration starts in the same state.

	## Using existing GoogleTest fixtures in fuzz tests

	Note: Only use GoogleTest fixtures if they are shared with existing unit tests,
	or if you need static per-fixture setup and teardown (via `SetUpTestSuite` and
	`TearDownTestSuite`). Otherwise, prefer fixtureless code with
	[standalone initialization functions](https://abseil.io/tips/122).

	If you already use a GoogleTest fixture derived from `::testing::Test` in your
	unit tests and you want to reuse it in fuzz tests, you first need to choose when
	you want the fixture to be instantiated and destroyed. GoogleTest fixtures are
	typically built around a guarantee that the same fixture object will never be
	reused for multiple tests or multiple runs of the same test. In a fuzz test,
	this guarantee requires instantiating the fixture *once per fuzz test
	iteration*, i.e., calling the property function each time on a fresh fixture
	object. However, for increased performance, it is possible to instantiate a
	fixture once per fuzz test (if the fixture supports such persistence) and
	reuse the same fixture object in all calls to the property function. As noted
	earlier, this comes with the risk of state mutation, which can lead to
	non-reproducible crashes, so design your fixture carefully.

	### Semantics 1: Fixture instantiated once per fuzz test iteration

	Adapt a fixture called `Fixture` by extending
	`::fuzztest::PerIterationFixtureAdapter<Fixture>`. This case should be the more
	common one, since it observes the GoogleTest invariant of not using a fixture
	object more than once.

	As an example, suppose we have a function `SumVec()` that returns the sum of a
	vector of integers and we have the following fixture.

	```c++
	class SumVecTest : public testing::Test {
	public:
	SumVecTest() : vec_{1, 2, 3} {}

	protected:
	std::vector<int> vec_;
	};
	```

	A fuzz test that uses the fixture would look like the following:

	```c++
	// Adapts the fixture using the "per-iteration" semantics.
	class SumVecFuzzTest : public fuzztest::PerIterationFixtureAdapter<SumVecTest> {
	public:
	void SumsLastEntry(int last_entry) {
	int previous_sum = SumVec(vec_);
	vec_.push_back(last_entry);
	EXPECT_EQ(SumVec(vec_), previous_sum + last_entry);
	}
	};
	FUZZ_TEST_F(SumVecFuzzTest, SumsLastEntry);
	```

	The "per-iteration" semantics is appropriate for this fixture because the test
	mutates `vec_`.

	### Semantics 2: Fixture instantiated once per fuzz test

	Adapt a fixture called `Fixture` by extending
	`::fuzztest::PerFuzzTestFixtureAdapter<Fixture>`. This case should be the less
	common one: it relies on the fixture being easily resettable, which is not a
	typical concern when designing GoogleTest fixtures.

	As an example, let's return to EchoServer. Suppose we already have a GoogleTest
	fixture that sets up the server.

	```c++
	class EchoServerTest : public testing::Test {
	public:
	EchoServerTest() { server_.Start("localhost:9999"); }
	~EchoServerTest() override { server_.Stop(); }

	private:
	EchoServer server_;
	};
	```

	We would write a test using this fixture as in the following code:

	```c++
	// Adapts the fixture using the "per-fuzz-test" semantics.
	class EchoServerFuzzTest
	: public fuzztest::PerFuzzTestFixtureAdapter<EchoServerTest> {
	public:
	void ReturnsTheSameString(const std::string& request) {
	std::string response;
	SendRequest("localhost:9999", request, &response);
	EXPECT_EQ(response, request);
	}
	};
	FUZZ_TEST_F(EchoServerFuzzTest, ReturnsTheSameString);
	```

	Here, the "per-fuzz-test" semantics is appropriate because the test doesn't
	mutate the server in any way that would be relevant for the test, and the server
	initialization is too expensive to be performed in each iteration.

	## Advanced fixtures with runner functions

	When using non-GoogleTest fixtures, FuzzTest can make use of runner functions
	on the fixture to specify per-test or per-iteration setups in a more flexible
	way. FuzzTest provides the following runner interfaces for a fixture to inherit:

	- `::fuzztest::FuzzTestRunnerFixture`, which declares the runner function
	`void FuzzTestRunner(absl::AnyInvocable<void() &&> run_test)`

	If the interface is inherited, instead of directly running the fuzz test,
	FuzzTest will call `FuzzTestRunner`, expecting that it will in turn:

	1. Set up everything needed for running the test.
	2. Invoke `run_test`, through which FuzzTest will essentially pass the
	whole fuzzing loop with multiple invocations to the property function.
	The function `run_test` will return once FuzzTest is done with the test.
	3. Tear down everything no longer needed after the test.

	- `::fuzztest::IterationRunnerFixture`, which declares the runner function
	`void FuzzTestIterationRunner(absl::AnyInvocable<void() &&> run_iteration)`

	If the interface is inherited, instead of directly running a fuzz test
	iteration, FuzzTest will call `FuzzTestIterationRunner`, expecting that it
	will in turn:

	1. Set up everything needed for running a test iteration.
	2. Invoke `run_iteration`, through which FuzzTest will invoke the property
	function once. The function `run_iteration` will return once the
	iteration is done.
	3. Tear down everything no longer needed after the test iteration.

	Note that the type `absl::AnyInvocable<void() &&>` ensures that the functions
	`run_test` and `run_iteration` can be called only once. To call them, you must
	first move them, for example `std::move(run_test)()`. Attempting to call these
	functions more than once will result in a runtime error.

	The traditional SetUp/TearDown functions of a fixture can be trivially
	transformed into a runner function. For example, the `EchoServerFuzzTest`
	fixture could be defined as:

	```c++
	class EchoServerFuzzTest : public fuzztest::FuzzTestRunnerFixture {
	public:

	void FuzzTestRunner(absl::AnyInvocable<void() &&> run_test) override {
	server_.Start("localhost:9999");
	std::move(run_test)();
	server_.Stop();
	}

	// The fuzz test's property function
	void ReturnsTheSameString(const std::string& request) {
	std::string response;
	SendRequest("localhost:9999", request, &response);
	EXPECT_EQ(response, request);
	}

	private:
	EchoServer server_;
	};

	FUZZ_TEST_F(EchoServerFuzzTest, ReturnsTheSameString);
	```

	To create a new `EchoServer` instance for each fuzz test iteration, define
	`FuzzTestIterationRunner` instead:

	```c++
	class EchoServerFuzzTest : public fuzztest::IterationRunnerFixture {
	public:

	void FuzzTestIterationRunner(absl::AnyInvocable<void() &&> run_iteration) override {
	server_.Start("localhost:9999");
	std::move(run_iteration)();
	server_.Stop();
	}

	...
	```

	Although converting SetUp/TearDown to runner functions is trivial, the other way
	around may not even be possible. For example, suppose there is a fuzz test whose
	property function must run within an SUT created by an SUT method
	`RunWithinSut(sut_ready_cb)`, which starts the SUT and calls the callback when
	the SUT is ready, and shuts down the SUT when the callback returns. Suppose
	additionally that `RunWithinSut` must be run in the main thread of the test,
	which is also the thread where FuzzTest runs the test fixture. Given these
	constraints, it is impossible to run such test with a SetUp/TearDown-style
	fixture without calling `RunWithinSut` for every iteration, which may be too
	heavy and impact the fuzzing performance. With runner functions, the fixture can
	be defined as:

	```c++
	class SutBasedFuzzTest : public fuzztest::FuzzTestRunnerFixture {
	public:

	void FuzzTestRunner(absl::AnyInvocable<void() &&> run_test) {
	sut_.RunWithinSut([&] { std::move(run_test)(); });`.
	}

	private:
	SUT sut_;
	}
	```