docs/automated_analysis.rst - pigweed/pigweed - Git at Google

 .. _docs-automated-analysis:

 ==================
 Automated analysis
 ==================

 The correctness and style of Pigweed's source code is continuously verified
 using a suite of automated tools. We also make it easy to use the same tools
 to verify the code of projects using Pigweed.

 -------
 Summary
 -------
 On presubmit or in CI we verify Pigweed using:

 * pylint
 * mypy
 * clang-tidy
 * AddressSanitizer (asan)
 * ThreadSanitizer (tsan)
 * UndefinedBehaviorSanitizer (ubsan)
 * OSS-Fuzz

 The rest of this document discusses these tools and their configuration in
 greater detail, and how to use them in your own project.

 --------------
 Analysis tools
 --------------

 Static analysis
 ===============

 PyLint
 ------
 `PyLint`_ is a customizable Python linter. Pigweed complies with almost all
 the default checks; see `.pylintrc`_ for details. PyLint detects problems such
 as overly broad catch statements, unused arguments/variables, and mutable
 default parameter values.

 For upstream Pigweed, PyLint can be run with ``ninja python.lint.pylint`` or
 ``ninja python.lint``.  It's also included in a variety of presubmit steps,
 like ``static_analysis`` and ``python_checks.gn_python_check``.  See the
 `Enabling analysis for your project`_ section to learn how to run PyLint on
 your Pigweed-based project.

 .. _PyLint: https://pylint.org/
 .. _.pylintrc: https://cs.pigweed.dev/pigweed/+/main:.pylintrc

 Mypy
 ----
 Python 3 allows for `type annotations`_ for variables, function arguments, and
 return values. Most, but not all, of Pigweed's Python code has type
 annotations, and these annotations have caught real bugs in code that didn't
 yet have unit tests. `Mypy`_ is an analysis tool that enforces these
 annotations.

 Mypy helps find bugs like when a string is passed into a function that expects
 a list of strings---since both are iterables this bug might otherwise be hard
 to track down.

 Mypy can be run with ``ninja python.lint.mypy`` or ``ninja python.lint``. It's
 also included in a variety of presubmit steps, like ``static_analysis`` and
 ``python_checks.gn_python_check``.

 .. _type annotations: https://docs.python.org/3/library/typing.html
 .. _Mypy: http://mypy-lang.org/

 clang-tidy
 ----------
 `clang-tidy`_ is a C++ "linter" and static analysis tool. It identifies
 bug-prone patterns (e.g., use after move), non-idiomatic usage (e.g., creating
 ``std::unique_ptr`` with ``new`` rather than ``std::make_unique``), and
 performance issues (e.g., unnecessary copies of loop variables).

 While powerful, clang-tidy defines a very large number of checks, many of which
 are special-purpose (e.g., only applicable to FPGA HLS code, or code using the
 `Abseil`_ library) or have high false positive rates. Pigweed enables over 50
 checks which are relevant to an embedded C/C++ library and have good
 signal-to-noise ratios. The full list of Pigweed's checks is in `.clang-tidy`_.

 We do not currently enable the `Clang Static Analyzers`_ because they suffer
 from false positives, and their findings are time-consuming to manually verify.

 clang-tidy can be run with ``ninja static_analysis`` or ``pw presubmit --step
 static_analysis``. Note that as a static analysis tool, clang-tidy will not
 produce any runnable binaries: it simply analyzes the source files.

 .. _clang-tidy: https://clang.llvm.org/extra/clang-tidy/
 .. _Abseil: https://abseil.io/
 .. _.clang-tidy: https://cs.pigweed.dev/pigweed/+/main:.clang-tidy
 .. _Clang Static Analyzers: https://clang-analyzer.llvm.org/available_checks.html


 Clang sanitizers
 ================
 We run all of Pigweed's unit tests with the additional instrumentation
 described in this section. For more detail about these sanitizers, see the
 `Github documentation`_.

 * asan: `AddressSanitizer`_ detects memory errors such as out-of-bounds access
   and use-after-free.
 * tsan: `ThreadSanitizer`_ detects data races.
 * ubsan: `UndefinedBehaviorSanitizer`_ is a fast undefined behavior detector.
   We use the default ``-fsanitize=undefined`` option.

 .. note::
    Pigweed does not currently support `MemorySanitizer`_ (msan). See
    :bug:`234876100` for details.

 The exact configurations we use for these sanitizers are in
 `pw_toolchain/host_clang/BUILD.gn <https://cs.pigweed.dev/pigweed/+/main:pw_toolchain/host_clang/BUILD.gn>`_.
 You can see the current status of the sanitizer builds in the `Pigweed CI
 console`_, as ``pigweed-linux-san-*``.

 Unlike clang-tidy, the clang sanitizers are runtime instrumentation: the
 instrumented binary needs to be run for issues to be detected.

 .. _Github documentation: https://github.com/google/sanitizers
 .. _AddressSanitizer: https://clang.llvm.org/docs/AddressSanitizer.html
 .. _MemorySanitizer: https://clang.llvm.org/docs/MemorySanitizer.html
 .. _Pigweed CI console: https://ci.chromium.org/p/pigweed/g/pigweed/console
 .. _ThreadSanitizer: https://clang.llvm.org/docs/ThreadSanitizer.html
 .. _UndefinedBehaviorSanitizer: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html


 Fuzzers
 =======
 `Fuzz testing`_ detects errors in software by providing it with randomly
 generated inputs.  We use `OSS-fuzz`_ to continuously uncover potential
 vulnerabilities in Pigweed.  `Dashboard with Pigweed's latest results`_. See
 the :ref:`module-pw_fuzzer` module documentation for more details.

 .. _Dashboard with Pigweed's latest results: https://oss-fuzz-build-logs.storage.googleapis.com/index.html#pigweed
 .. _Fuzz testing: https://en.wikipedia.org/wiki/Fuzzing
 .. _OSS-fuzz: https://github.com/google/oss-fuzz

 .. _Enabling analysis for your project:

 ----------------------------------
 Enabling analysis for your project
 ----------------------------------

 GN
 ==

 PyLint and Mypy
 ---------------
 PyLint and Mypy can be configured to run every time your project is built by
 adding ``python.lint`` to your default build group. (You can also add one or both
 individually using ``python.lint.mypy`` and ``python.lint.pylint``.) Likewise,
 these can be added to individual presubmit steps (`examples`_). You can also
 directly include the `python_checks.gn_python_lint`_ presubmit step.

 .. _examples: https://cs.opensource.google/search?q=file:pigweed_presubmit.py%20%22python.lint%22&sq=&ss=pigweed%2Fpigweed
 .. _python_checks.gn_python_lint: https://cs.pigweed.dev/pigweed/+/main:pw_presubmit/py/pw_presubmit/python_checks.py?q=file:python_checks.py%20gn_python_lint&ss=pigweed%2Fpigweed

 clang-tidy
 ----------
 `pw_toolchain/static_analysis_toolchain.gni`_ provides the
 ``pw_static_analysis_toolchain`` template that can be used to create a build
 group performing static analysis. See :ref:`module-pw_toolchain` documentation
 for more details. This group can then be added as a presubmit step using
 pw_presubmit.

 You can place a ``.clang-tidy`` file at the root of your repository to control
 which checks are executed. See the `clang documentation`_ for a discussion of how
 the tool chooses which ``.clang-tidy`` files to apply when run on a particular
 source file.

 .. _pw_toolchain/static_analysis_toolchain.gni: https://cs.pigweed.dev/pigweed/+/main:pw_toolchain/static_analysis_toolchain.gni
 .. _clang documentation: https://clang.llvm.org/extra/clang-tidy/

 Clang sanitizers
 ----------------
 There are two ways to enable sanitizers for your build.

 GN args on debug toolchains
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 If you are already building your tests with one of the following toolchains (or
 a toolchain derived from one of them):

 * ``pw_toolchain_host_clang.debug``
 * ``pw_toolchain_host_clang.speed_optimized``
 * ``pw_toolchain_host_clang.size_optimized``

 you can enable the clang sanitizers simply by setting the gn arg
 ``pw_toolchain_SANITIZERS`` to the desired subset of
 ``["address", "thread", "undefined"]``.

 Example
 .......
 If your project defines a toolchain ``host_clang_debug`` that is derived from
 one of the above toolchains, and you'd like to run the ``pw_executable`` target
 ``sample_binary`` defined in the ``BUILD.gn`` file in ``examples/sample`` with
 asan, you would run,

 .. code-block:: bash

    gn gen out --args='pw_toolchain_SANITIZERS=["address"]'
    ninja -C out host_clang_debug/obj/example/sample/bin/sample_binary
    out/host_clang_debug/obj/example/sample/bin/sample_binary

 Sanitizer toolchains
 ^^^^^^^^^^^^^^^^^^^^
 Otherwise, instead of using ``gn args`` you can build your tests with the
 appropriate toolchain from the following list (or a toolchain derived from one
 of them):

 * ``pw_toolchain_host_clang.asan``
 * ``pw_toolchain_host_clang.ubsan``
 * ``pw_toolchain_host_clang.tsan``

 See the :ref:`module-pw_toolchain` module documentation for more
 about Pigweed toolchains.

 Bazel
 =====

 .. _docs-automated-analysis-clang-sanitizers:

 Clang sanitizers
 ----------------
 If you're using Pigweed's own host toolchain configuration, you can enable
 AddressSanitizer by building with the appropriate flag:

 .. code-block:: sh

    bazelisk build --@pigweed//pw_toolchain/host_clang:asan //...

 If you're building your own toolchain, you can add
 ``@pigweed//pw_toolchain_bazel/flag_sets:asan`` to it.

 Fuzzers
 =======
 See the :ref:`module-pw_fuzzer` module documentation.
	.. _docs-automated-analysis:

	==================
	Automated analysis
	==================

	The correctness and style of Pigweed's source code is continuously verified
	using a suite of automated tools. We also make it easy to use the same tools
	to verify the code of projects using Pigweed.

	-------
	Summary
	-------
	On presubmit or in CI we verify Pigweed using:

	* pylint
	* mypy
	* clang-tidy
	* AddressSanitizer (asan)
	* ThreadSanitizer (tsan)
	* UndefinedBehaviorSanitizer (ubsan)
	* OSS-Fuzz

	The rest of this document discusses these tools and their configuration in
	greater detail, and how to use them in your own project.

	--------------
	Analysis tools
	--------------

	Static analysis
	===============

	PyLint
	------
	`PyLint`_ is a customizable Python linter. Pigweed complies with almost all
	the default checks; see `.pylintrc`_ for details. PyLint detects problems such
	as overly broad catch statements, unused arguments/variables, and mutable
	default parameter values.

	For upstream Pigweed, PyLint can be run with ``ninja python.lint.pylint`` or
	``ninja python.lint``. It's also included in a variety of presubmit steps,
	like ``static_analysis`` and ``python_checks.gn_python_check``. See the
	`Enabling analysis for your project`_ section to learn how to run PyLint on
	your Pigweed-based project.

	.. _PyLint: https://pylint.org/
	.. _.pylintrc: https://cs.pigweed.dev/pigweed/+/main:.pylintrc

	Mypy
	----
	Python 3 allows for `type annotations`_ for variables, function arguments, and
	return values. Most, but not all, of Pigweed's Python code has type
	annotations, and these annotations have caught real bugs in code that didn't
	yet have unit tests. `Mypy`_ is an analysis tool that enforces these
	annotations.

	Mypy helps find bugs like when a string is passed into a function that expects
	a list of strings---since both are iterables this bug might otherwise be hard
	to track down.

	Mypy can be run with ``ninja python.lint.mypy`` or ``ninja python.lint``. It's
	also included in a variety of presubmit steps, like ``static_analysis`` and
	``python_checks.gn_python_check``.

	.. _type annotations: https://docs.python.org/3/library/typing.html
	.. _Mypy: http://mypy-lang.org/

	clang-tidy
	----------
	`clang-tidy`_ is a C++ "linter" and static analysis tool. It identifies
	bug-prone patterns (e.g., use after move), non-idiomatic usage (e.g., creating
	``std::unique_ptr`` with ``new`` rather than ``std::make_unique``), and
	performance issues (e.g., unnecessary copies of loop variables).

	While powerful, clang-tidy defines a very large number of checks, many of which
	are special-purpose (e.g., only applicable to FPGA HLS code, or code using the
	`Abseil`_ library) or have high false positive rates. Pigweed enables over 50
	checks which are relevant to an embedded C/C++ library and have good
	signal-to-noise ratios. The full list of Pigweed's checks is in `.clang-tidy`_.

	We do not currently enable the `Clang Static Analyzers`_ because they suffer
	from false positives, and their findings are time-consuming to manually verify.

	clang-tidy can be run with ``ninja static_analysis`` or ``pw presubmit --step
	static_analysis``. Note that as a static analysis tool, clang-tidy will not
	produce any runnable binaries: it simply analyzes the source files.

	.. _clang-tidy: https://clang.llvm.org/extra/clang-tidy/
	.. _Abseil: https://abseil.io/
	.. _.clang-tidy: https://cs.pigweed.dev/pigweed/+/main:.clang-tidy
	.. _Clang Static Analyzers: https://clang-analyzer.llvm.org/available_checks.html


	Clang sanitizers
	================
	We run all of Pigweed's unit tests with the additional instrumentation
	described in this section. For more detail about these sanitizers, see the
	`Github documentation`_.

	* asan: `AddressSanitizer`_ detects memory errors such as out-of-bounds access
	and use-after-free.
	* tsan: `ThreadSanitizer`_ detects data races.
	* ubsan: `UndefinedBehaviorSanitizer`_ is a fast undefined behavior detector.
	We use the default ``-fsanitize=undefined`` option.

	.. note::
	Pigweed does not currently support `MemorySanitizer`_ (msan). See
	:bug:`234876100` for details.

	The exact configurations we use for these sanitizers are in
	`pw_toolchain/host_clang/BUILD.gn <https://cs.pigweed.dev/pigweed/+/main:pw_toolchain/host_clang/BUILD.gn>`_.
	You can see the current status of the sanitizer builds in the `Pigweed CI
	console`_, as ``pigweed-linux-san-*``.

	Unlike clang-tidy, the clang sanitizers are runtime instrumentation: the
	instrumented binary needs to be run for issues to be detected.

	.. _Github documentation: https://github.com/google/sanitizers
	.. _AddressSanitizer: https://clang.llvm.org/docs/AddressSanitizer.html
	.. _MemorySanitizer: https://clang.llvm.org/docs/MemorySanitizer.html
	.. _Pigweed CI console: https://ci.chromium.org/p/pigweed/g/pigweed/console
	.. _ThreadSanitizer: https://clang.llvm.org/docs/ThreadSanitizer.html
	.. _UndefinedBehaviorSanitizer: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html


	Fuzzers
	=======
	`Fuzz testing`_ detects errors in software by providing it with randomly
	generated inputs. We use `OSS-fuzz`_ to continuously uncover potential
	vulnerabilities in Pigweed. `Dashboard with Pigweed's latest results`_. See
	the :ref:`module-pw_fuzzer` module documentation for more details.

	.. _Dashboard with Pigweed's latest results: https://oss-fuzz-build-logs.storage.googleapis.com/index.html#pigweed
	.. _Fuzz testing: https://en.wikipedia.org/wiki/Fuzzing
	.. _OSS-fuzz: https://github.com/google/oss-fuzz

	.. _Enabling analysis for your project:

	----------------------------------
	Enabling analysis for your project
	----------------------------------

	GN
	==

	PyLint and Mypy
	---------------
	PyLint and Mypy can be configured to run every time your project is built by
	adding ``python.lint`` to your default build group. (You can also add one or both
	individually using ``python.lint.mypy`` and ``python.lint.pylint``.) Likewise,
	these can be added to individual presubmit steps (`examples`_). You can also
	directly include the `python_checks.gn_python_lint`_ presubmit step.

	.. _examples: https://cs.opensource.google/search?q=file:pigweed_presubmit.py%20%22python.lint%22&sq=&ss=pigweed%2Fpigweed
	.. _python_checks.gn_python_lint: https://cs.pigweed.dev/pigweed/+/main:pw_presubmit/py/pw_presubmit/python_checks.py?q=file:python_checks.py%20gn_python_lint&ss=pigweed%2Fpigweed

	clang-tidy
	----------
	`pw_toolchain/static_analysis_toolchain.gni`_ provides the
	``pw_static_analysis_toolchain`` template that can be used to create a build
	group performing static analysis. See :ref:`module-pw_toolchain` documentation
	for more details. This group can then be added as a presubmit step using
	pw_presubmit.

	You can place a ``.clang-tidy`` file at the root of your repository to control
	which checks are executed. See the `clang documentation`_ for a discussion of how
	the tool chooses which ``.clang-tidy`` files to apply when run on a particular
	source file.

	.. _pw_toolchain/static_analysis_toolchain.gni: https://cs.pigweed.dev/pigweed/+/main:pw_toolchain/static_analysis_toolchain.gni
	.. _clang documentation: https://clang.llvm.org/extra/clang-tidy/

	Clang sanitizers
	----------------
	There are two ways to enable sanitizers for your build.

	GN args on debug toolchains
	^^^^^^^^^^^^^^^^^^^^^^^^^^^
	If you are already building your tests with one of the following toolchains (or
	a toolchain derived from one of them):

	* ``pw_toolchain_host_clang.debug``
	* ``pw_toolchain_host_clang.speed_optimized``
	* ``pw_toolchain_host_clang.size_optimized``

	you can enable the clang sanitizers simply by setting the gn arg
	``pw_toolchain_SANITIZERS`` to the desired subset of
	``["address", "thread", "undefined"]``.

	Example
	.......
	If your project defines a toolchain ``host_clang_debug`` that is derived from
	one of the above toolchains, and you'd like to run the ``pw_executable`` target
	``sample_binary`` defined in the ``BUILD.gn`` file in ``examples/sample`` with
	asan, you would run,

	.. code-block:: bash

	gn gen out --args='pw_toolchain_SANITIZERS=["address"]'
	ninja -C out host_clang_debug/obj/example/sample/bin/sample_binary
	out/host_clang_debug/obj/example/sample/bin/sample_binary

	Sanitizer toolchains
	^^^^^^^^^^^^^^^^^^^^
	Otherwise, instead of using ``gn args`` you can build your tests with the
	appropriate toolchain from the following list (or a toolchain derived from one
	of them):

	* ``pw_toolchain_host_clang.asan``
	* ``pw_toolchain_host_clang.ubsan``
	* ``pw_toolchain_host_clang.tsan``

	See the :ref:`module-pw_toolchain` module documentation for more
	about Pigweed toolchains.

	Bazel
	=====

	.. _docs-automated-analysis-clang-sanitizers:

	Clang sanitizers
	----------------
	If you're using Pigweed's own host toolchain configuration, you can enable
	AddressSanitizer by building with the appropriate flag:

	.. code-block:: sh

	bazelisk build --@pigweed//pw_toolchain/host_clang:asan //...

	If you're building your own toolchain, you can add
	``@pigweed//pw_toolchain_bazel/flag_sets:asan`` to it.

	Fuzzers
	=======
	See the :ref:`module-pw_fuzzer` module documentation.