pw_perf_test/docs.rst - pigweed/pigweed - Git at Google

 .. _module-pw_perf_test:

 ============
 pw_perf_test
 ============

 .. pigweed-module::
    :name: pw_perf_test

    - **Simple**: Automatically manages boilerplate like iterations and durations.
    - **Easy**: Uses an intuitive API that resembles GoogleTest.
    - **Reusable**: Integrates with modules like ``pw_log`` that you already use.

 Pigweed's perf test module provides an easy way to measure performance on
 any test setup!

 ---------------
 Getting started
 ---------------
 You can add a simple performance test using the follow steps:

 Configure your toolchain
 ========================
 If necessary, configure your toolchain for performance testing:

 .. note:: Currently, ``pw_perf_test`` provides build integration with Bazel and
    GN. Performance tests can be built in CMake, but must be built as regular
    executables.

 .. tab-set::

    .. tab-item:: Bazel
       :sync: bazel

        - ``pw_perf_test_timer_backend``: Sets the backend used to measure
          durations. Options include:

          - ``@pigweed//pw_perf_test:chrono_timer``: Uses
            ``pw_chrono::SystemClock`` to measure time.
          - ``@pigweed//pw_perf_test:arm_cortex_timer``: Uses cycle count
            registers available on ARM-Cortex to measure time.

        - Currently, only the logging event handler is supported for Bazel.

    .. tab-item:: GN
       :sync: gn

        - ``pw_perf_test_TIMER_INTERFACE_BACKEND``: Sets the backend used to
          measure durations. Options include:

          - ``"$dir_pw_perf_test:chrono_timer"``: Uses
            ``pw_chrono::SystemClock`` to measure time.
          - ``"$dir_pw_perf_test:arm_cortex_timer"``: Uses cycle count
            registers available on ARM-Cortex to measure time.

        - ``pw_perf_test_MAIN_FUNCTION``: Indicates the GN target that provides
          a ``main`` function that sets the event handler and runs tests. The
          default is ``"$dir_pw_perf_test:logging_main"``.

 Write a test function
 =====================
 Write a test function that exercises the behavior you wish to benchmark. For
 this example, we will simulate doing work with:

 .. literalinclude:: examples/example_perf_test.cc
    :language: cpp
    :linenos:
    :start-after: [pw_perf_test_examples-simulate_work]
    :end-before: [pw_perf_test_examples-simulate_work]

 Creating a performance test is as simple as using the ``PW_PERF_TEST_SIMPLE``
 macro to name the function and optionally provide arguments to it:

 .. literalinclude:: examples/example_perf_test.cc
    :language: cpp
    :linenos:
    :start-after: [pw_perf_test_examples-simple_example]
    :end-before: [pw_perf_test_examples-simple_example]

 If you need to do additional setup as part of your test, you can use the
 ``PW_PERF_TEST`` macro, which provides an explicit ``pw::perf_test::State``
 reference. the behavior to be benchmarked should be put in a loop that checks
 ``State::KeepRunning()``:

 .. literalinclude:: examples/example_perf_test.cc
    :language: cpp
    :linenos:
    :start-after: [pw_perf_test_examples-full_example]
    :end-before: [pw_perf_test_examples-full_example]

 You can even use lambdas in place of standalone functions:

 .. literalinclude:: examples/example_perf_test.cc
    :language: cpp
    :linenos:
    :start-after: [pw_perf_test_examples-lambda_example]
    :end-before: [pw_perf_test_examples-lambda_example]

 .. _module-pw_perf_test-pw_perf_test:

 Build Your Test
 ===============
 .. tab-set::

    .. tab-item:: Bazel
       :sync: bazel

       Add your performance test to the build using the ``pw_cc_perf_test``
       rule from ``//pw_build:pigweed.bzl``.

       **Arguments**

       * All ``native.cc_binary`` arguments are supported.

       **Example**

       .. code-block::

          load("//pw_build:pigweed.bzl", "pw_cc_perf_test")

          pw_cc_perf_test(
              name = "my_perf_test",
              srcs = ["my_perf_test.cc"],
          )

    .. tab-item:: GN
       :sync: gn

       Add your performance test to the build using the ``pw_perf_test``
       template. This template creates two sub-targets.

       * ``<target_name>.lib``: The test sources without a main function.
       * ``<target_name>``: The test suite binary, linked against
         ``pw_perf_test_MAIN_FUNCTION``.

       **Arguments**

       * All ``pw_executable`` arguments are supported.
       * ``enable_if``: Boolean indicating whether the test should be built. If
         false, replaces the test with an empty target. Defaults to true.

       **Example**

       .. code-block::

          import("$dir_pw_perf_test/perf_test.gni")

          pw_perf_test("my_perf_test") {
            sources = [ "my_perf_test.cc" ]
            enable_if = device_has_1m_flash
          }

 Run your test
 =============
 .. tab-set::

    .. tab-item:: GN
       :sync: gn

       To run perf tests from GN, locate the associated binaries from the ``out``
       directory and run/flash them manually.

    .. tab-item:: Bazel
       :sync: bazel

       Use the default Bazel run command: ``bazel run //path/to:target``.

 -------------
 API reference
 -------------

 Macros
 ======

 .. doxygendefine:: PW_PERF_TEST

 .. doxygendefine:: PW_PERF_TEST_SIMPLE

 EventHandler
 ============

 .. doxygenclass:: pw::perf_test::EventHandler
    :members:

 ------
 Design
 ------

 ``pw_perf_test`` uses a ``Framework`` singleton similar to that of
 ``pw_unit_test``. This singleton is statically created, and tests declared using
 macros such as ``PW_PERF_TEST`` will automatically register themselves with it.

 A provided ``main`` function interacts with the ``Framework`` by calling
 ``pw::perf_test::RunAllTests`` and providing an ``EventHandler``. For each
 registered test, the ``Framework`` creates a ``State`` object and passes it to
 the test function.

 The State object tracks the number of iterations. It expects the test function
 to include a loop with the condition of ``State::KeepRunning``. This loop
 should include the behavior being banchmarked, e.g.

 .. code-block:: cpp

    while (state.KeepRunning()) {
      // Code to be benchmarked.
    }

 In particular, ``State::KeepRunning`` should be called exactly once before the
 first iteration, as in a ``for`` or ``while`` loop. The ``State`` object will
 use the timer facade to measure the elapsed duration between successive calls to
 ``State::KeepRunning``.

 Additionally, the ``State`` object receives a reference to the ``EventHandler``
 from the ``Framework``, and uses this to report both test progress and
 performance measurements.

 Timers
 ======
 Currently, Pigweed provides two implementations of the timer interface.
 Consumers may provide additional implementations and use them as a backend for
 the timer facade.

 Chrono Timer
 ------------
 This timer depends :ref:`module-pw_chrono` and will only measure performance in
 terms of nanoseconds. It is the default for performance tests on host.

 Cycle Count Timer
 -----------------
 On ARM Cortex devices, clock cycles may more accurately measure the actual
 performance of a benchmark.

 This implementation is OS-agnostic, as it directly accesses CPU registers.
 It enables the `DWT register`_ through the `DEMCR register`_. While this
 provides cycle counts directly from the CPU, it notably overflows if the
 duration of a test exceeding 2^32 clock cycles. At 100 MHz, this is
 approximately 43 seconds.

 .. warning::
   The interface only measures raw clock cycles and does not take into account
   other possible sources of pollution such as LSUs, Sleeps and other registers.
   `Read more on the DWT methods of counting instructions.`__

 .. __: `DWT methods`_

 EventHandlers
 =============
 Currently, Pigweed provides one implementation of ``EventHandler``. Consumers
 may provide additional implementations and use them by providing a dedicated
 ``main`` function that passes the handler to ``pw::perf_test::RunAllTests``.

 LoggingEventHandler
 -------------------
 The default method of running performance tests uses a ``LoggingEventHandler``.
 This event handler only logs the test results to the console and nothing more.
 It was chosen as the default method due to its portability and to cut down on
 the time it would take to implement other printing log handlers. Make sure to
 set a ``pw_log`` backend.

 -------
 Roadmap
 -------
 - `CMake support <https://g-issues.pigweed.dev/issues/309637691>`_
 - `Unified framework <https://g-issues.pigweed.dev/issues/309639171>`_.

 .. _DWT register: https://developer.arm.com/documentation/ddi0337/e/System-Debug/DWT?lang=en
 .. _DEMCR register: https://developer.arm.com/documentation/ddi0337/e/CEGHJDCF
 .. _DWT methods: https://developer.arm.com/documentation/ka001499/1-0/
	.. _module-pw_perf_test:

	============
	pw_perf_test
	============

	.. pigweed-module::
	:name: pw_perf_test

	- Simple: Automatically manages boilerplate like iterations and durations.
	- Easy: Uses an intuitive API that resembles GoogleTest.
	- Reusable: Integrates with modules like ``pw_log`` that you already use.

	Pigweed's perf test module provides an easy way to measure performance on
	any test setup!

	---------------
	Getting started
	---------------
	You can add a simple performance test using the follow steps:

	Configure your toolchain
	========================
	If necessary, configure your toolchain for performance testing:

	.. note:: Currently, ``pw_perf_test`` provides build integration with Bazel and
	GN. Performance tests can be built in CMake, but must be built as regular
	executables.

	.. tab-set::

	.. tab-item:: Bazel
	:sync: bazel

	- ``pw_perf_test_timer_backend``: Sets the backend used to measure
	durations. Options include:

	- ``@pigweed//pw_perf_test:chrono_timer``: Uses
	``pw_chrono::SystemClock`` to measure time.
	- ``@pigweed//pw_perf_test:arm_cortex_timer``: Uses cycle count
	registers available on ARM-Cortex to measure time.

	- Currently, only the logging event handler is supported for Bazel.

	.. tab-item:: GN
	:sync: gn

	- ``pw_perf_test_TIMER_INTERFACE_BACKEND``: Sets the backend used to
	measure durations. Options include:

	- ``"$dir_pw_perf_test:chrono_timer"``: Uses
	``pw_chrono::SystemClock`` to measure time.
	- ``"$dir_pw_perf_test:arm_cortex_timer"``: Uses cycle count
	registers available on ARM-Cortex to measure time.

	- ``pw_perf_test_MAIN_FUNCTION``: Indicates the GN target that provides
	a ``main`` function that sets the event handler and runs tests. The
	default is ``"$dir_pw_perf_test:logging_main"``.

	Write a test function
	=====================
	Write a test function that exercises the behavior you wish to benchmark. For
	this example, we will simulate doing work with:

	.. literalinclude:: examples/example_perf_test.cc
	:language: cpp
	:linenos:
	:start-after: [pw_perf_test_examples-simulate_work]
	:end-before: [pw_perf_test_examples-simulate_work]

	Creating a performance test is as simple as using the ``PW_PERF_TEST_SIMPLE``
	macro to name the function and optionally provide arguments to it:

	.. literalinclude:: examples/example_perf_test.cc
	:language: cpp
	:linenos:
	:start-after: [pw_perf_test_examples-simple_example]
	:end-before: [pw_perf_test_examples-simple_example]

	If you need to do additional setup as part of your test, you can use the
	``PW_PERF_TEST`` macro, which provides an explicit ``pw::perf_test::State``
	reference. the behavior to be benchmarked should be put in a loop that checks
	``State::KeepRunning()``:

	.. literalinclude:: examples/example_perf_test.cc
	:language: cpp
	:linenos:
	:start-after: [pw_perf_test_examples-full_example]
	:end-before: [pw_perf_test_examples-full_example]

	You can even use lambdas in place of standalone functions:

	.. literalinclude:: examples/example_perf_test.cc
	:language: cpp
	:linenos:
	:start-after: [pw_perf_test_examples-lambda_example]
	:end-before: [pw_perf_test_examples-lambda_example]

	.. _module-pw_perf_test-pw_perf_test:

	Build Your Test
	===============
	.. tab-set::

	.. tab-item:: Bazel
	:sync: bazel

	Add your performance test to the build using the ``pw_cc_perf_test``
	rule from ``//pw_build:pigweed.bzl``.

	Arguments

	* All ``native.cc_binary`` arguments are supported.

	Example

	.. code-block::

	load("//pw_build:pigweed.bzl", "pw_cc_perf_test")

	pw_cc_perf_test(
	name = "my_perf_test",
	srcs = ["my_perf_test.cc"],
	)

	.. tab-item:: GN
	:sync: gn

	Add your performance test to the build using the ``pw_perf_test``
	template. This template creates two sub-targets.

	* ``<target_name>.lib``: The test sources without a main function.
	* ``<target_name>``: The test suite binary, linked against
	``pw_perf_test_MAIN_FUNCTION``.

	Arguments

	* All ``pw_executable`` arguments are supported.
	* ``enable_if``: Boolean indicating whether the test should be built. If
	false, replaces the test with an empty target. Defaults to true.

	Example

	.. code-block::

	import("$dir_pw_perf_test/perf_test.gni")

	pw_perf_test("my_perf_test") {
	sources = [ "my_perf_test.cc" ]
	enable_if = device_has_1m_flash
	}

	Run your test
	=============
	.. tab-set::

	.. tab-item:: GN
	:sync: gn

	To run perf tests from GN, locate the associated binaries from the ``out``
	directory and run/flash them manually.

	.. tab-item:: Bazel
	:sync: bazel

	Use the default Bazel run command: ``bazel run //path/to:target``.

	-------------
	API reference
	-------------

	Macros
	======

	.. doxygendefine:: PW_PERF_TEST

	.. doxygendefine:: PW_PERF_TEST_SIMPLE

	EventHandler
	============

	.. doxygenclass:: pw::perf_test::EventHandler
	:members:

	------
	Design
	------

	``pw_perf_test`` uses a ``Framework`` singleton similar to that of
	``pw_unit_test``. This singleton is statically created, and tests declared using
	macros such as ``PW_PERF_TEST`` will automatically register themselves with it.

	A provided ``main`` function interacts with the ``Framework`` by calling
	``pw::perf_test::RunAllTests`` and providing an ``EventHandler``. For each
	registered test, the ``Framework`` creates a ``State`` object and passes it to
	the test function.

	The State object tracks the number of iterations. It expects the test function
	to include a loop with the condition of ``State::KeepRunning``. This loop
	should include the behavior being banchmarked, e.g.

	.. code-block:: cpp

	while (state.KeepRunning()) {
	// Code to be benchmarked.
	}

	In particular, ``State::KeepRunning`` should be called exactly once before the
	first iteration, as in a ``for`` or ``while`` loop. The ``State`` object will
	use the timer facade to measure the elapsed duration between successive calls to
	``State::KeepRunning``.

	Additionally, the ``State`` object receives a reference to the ``EventHandler``
	from the ``Framework``, and uses this to report both test progress and
	performance measurements.

	Timers
	======
	Currently, Pigweed provides two implementations of the timer interface.
	Consumers may provide additional implementations and use them as a backend for
	the timer facade.

	Chrono Timer
	------------
	This timer depends :ref:`module-pw_chrono` and will only measure performance in
	terms of nanoseconds. It is the default for performance tests on host.

	Cycle Count Timer
	-----------------
	On ARM Cortex devices, clock cycles may more accurately measure the actual
	performance of a benchmark.

	This implementation is OS-agnostic, as it directly accesses CPU registers.
	It enables the `DWT register`_ through the `DEMCR register`_. While this
	provides cycle counts directly from the CPU, it notably overflows if the
	duration of a test exceeding 2^32 clock cycles. At 100 MHz, this is
	approximately 43 seconds.

	.. warning::
	The interface only measures raw clock cycles and does not take into account
	other possible sources of pollution such as LSUs, Sleeps and other registers.
	`Read more on the DWT methods of counting instructions.`__

	.. __: `DWT methods`_

	EventHandlers
	=============
	Currently, Pigweed provides one implementation of ``EventHandler``. Consumers
	may provide additional implementations and use them by providing a dedicated
	``main`` function that passes the handler to ``pw::perf_test::RunAllTests``.

	LoggingEventHandler
	-------------------
	The default method of running performance tests uses a ``LoggingEventHandler``.
	This event handler only logs the test results to the console and nothing more.
	It was chosen as the default method due to its portability and to cut down on
	the time it would take to implement other printing log handlers. Make sure to
	set a ``pw_log`` backend.

	-------
	Roadmap
	-------
	- `CMake support <https://g-issues.pigweed.dev/issues/309637691>`_
	- `Unified framework <https://g-issues.pigweed.dev/issues/309639171>`_.

	.. _DWT register: https://developer.arm.com/documentation/ddi0337/e/System-Debug/DWT?lang=en
	.. _DEMCR register: https://developer.arm.com/documentation/ddi0337/e/CEGHJDCF
	.. _DWT methods: https://developer.arm.com/documentation/ka001499/1-0/