pw_presubmit/docs.rst - pigweed/pigweed - Git at Google

 .. _module-pw_presubmit:

 ============
 pw_presubmit
 ============
 The presubmit module provides Python tools for running presubmit checks and
 checking and fixing code format. It also includes the presubmit check script for
 the Pigweed repository, ``pigweed_presubmit.py``.

 Presubmit checks are essential tools, but they take work to set up, and
 projects don’t always get around to it. The ``pw_presubmit`` module provides
 tools for setting up high quality presubmit checks for any project. We use this
 framework to run Pigweed’s presubmit on our workstations and in our automated
 building tools.

 The ``pw_presubmit`` module also includes ``pw format``, a tool that provides a
 unified interface for automatically formatting code in a variety of languages.
 With ``pw format``, you can format C, C++, Python, GN, and Go code according to
 configurations defined by your project. ``pw format`` leverages existing tools
 like ``clang-format``, and it’s simple to add support for new languages.

 .. image:: docs/pw_presubmit_demo.gif
    :alt: ``pw format`` demo
    :align: left

 The ``pw_presubmit`` package includes presubmit checks that can be used with any
 project. These checks include:

 * Check code format of several languages including C, C++, and Python
 * Initialize a Python environment
 * Run all Python tests
 * Run pylint
 * Run mypy
 * Ensure source files are included in the GN and Bazel builds
 * Build and run all tests with GN
 * Build and run all tests with Bazel
 * Ensure all header files contain ``#pragma once``

 -------------
 Compatibility
 -------------
 Python 3

 -------------------------------------------
 Creating a presubmit check for your project
 -------------------------------------------
 Creating a presubmit check for a project using ``pw_presubmit`` is simple, but
 requires some customization. Projects must define their own presubmit check
 Python script that uses the ``pw_presubmit`` package.

 A project's presubmit script can be registered as a
 :ref:`pw_cli <module-pw_cli>` plugin, so that it can be run as ``pw
 presubmit``.

 Setting up the command-line interface
 -------------------------------------
 The ``pw_presubmit.cli`` module sets up the command-line interface for a
 presubmit script. This defines a standard set of arguments for invoking
 presubmit checks. Its use is optional, but recommended.

 pw_presubmit.cli
 ~~~~~~~~~~~~~~~~
 .. automodule:: pw_presubmit.cli
    :members: add_arguments, run

 Presubmit checks
 ----------------
 A presubmit check is defined as a function or other callable. The function must
 accept one argument: a ``PresubmitContext``, which provides the paths on which
 to run. Presubmit checks communicate failure by raising an exception.

 Presubmit checks may use the ``filter_paths`` decorator to automatically filter
 the paths list for file types they care about.

 Either of these functions could be used as presubmit checks:

 .. code-block:: python

   @pw_presubmit.filter_paths(endswith='.py')
   def file_contains_ni(ctx: PresubmitContext):
       for path in ctx.paths:
           with open(path) as file:
               contents = file.read()
               if 'ni' not in contents and 'nee' not in contents:
                   raise PresumitFailure('Files must say "ni"!', path=path)

   def run_the_build(_):
       subprocess.run(['make', 'release'], check=True)

 Presubmit checks functions are grouped into "programs" -- a named series of
 checks. Projects may find it helpful to have programs for different purposes,
 such as a quick program for local use and a full program for automated use. The
 :ref:`example script <example-script>` uses ``pw_presubmit.Programs`` to define
 ``quick`` and ``full`` programs.

 Existing Presubmit Checks
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 A small number of presubmit checks are made available through ``pw_presubmit``
 modules.

 Code Formatting
 ===============
 Formatting checks for a variety of languages are available from
 ``pw_presubmit.format_code``. These include C/C++, Java, Go, Python, GN, and
 others. All of these checks can be included by adding
 ``pw_presubmit.format_code.presubmit_checks()`` to a presubmit program. These
 all use language-specific formatters like clang-format or yapf.

 These will suggest fixes using ``pw format --fix``.

 #pragma once
 ============
 There's a ``pragma_once`` check that confirms the first non-comment line of
 C/C++ headers is ``#pragma once``. This is enabled by adding
 ``pw_presubmit.pragma_once`` to a presubmit program.

 Banned Words
 ============
 .. banned-words: disable

 The banned words check looks for words that are typical of non-inclusive code,
 like using "master" and "slave" in place of "primary" and "secondary" or
 "sanity check" in place of "consistency check".

 .. banned-words: enable

 These checks can be disabled for individual lines with "banned-words: ignore"
 on the line in question or the line above it, or for entire blocks by using
 "banned-words: disable" before the block and "banned-words: enable" after the
 block.

 .. In case things get moved around in the previous paragraphs the enable line
 .. is repeated here: banned-words: enable.

 pw_presubmit
 ~~~~~~~~~~~~
 .. automodule:: pw_presubmit
    :members: filter_paths, call, PresubmitFailure, Programs

 .. _example-script:

 Example
 -------
 A simple example presubmit check script follows. This can be copied-and-pasted
 to serve as a starting point for a project's presubmit check script.

 See ``pigweed_presubmit.py`` for a more complex presubmit check script example.

 .. code-block:: python

   """Example presubmit check script."""

   import argparse
   import logging
   import os
   from pathlib import Path
   import re
   import sys
   from typing import List, Pattern

   try:
       import pw_cli.log
   except ImportError:
       print('ERROR: Activate the environment before running presubmits!',
             file=sys.stderr)
       sys.exit(2)

   import pw_presubmit
   from pw_presubmit import banned_words, build, cli, environment, format_code
   from pw_presubmit import git_repo, python_checks, filter_paths
   from pw_presubmit import PresubmitContext
   from pw_presubmit.install_hook import install_hook

   # Set up variables for key project paths.
   PROJECT_ROOT = Path(os.environ['MY_PROJECT_ROOT'])
   PIGWEED_ROOT = PROJECT_ROOT / 'pigweed'

   #
   # Initialization
   #
   def init_cipd(ctx: PresubmitContext):
       environment.init_cipd(PIGWEED_ROOT, ctx.output_dir)


   def init_virtualenv(ctx: PresubmitContext):
       environment.init_virtualenv(PIGWEED_ROOT,
                                   ctx.output_dir,
                                   setup_py_roots=[PROJECT_ROOT])


   # Rerun the build if files with these extensions change.
   _BUILD_EXTENSIONS = frozenset(
       ['.rst', '.gn', '.gni', *format_code.C_FORMAT.extensions])


   #
   # Presubmit checks
   #
   def release_build(ctx: PresubmitContext):
       build.gn_gen(PROJECT_ROOT, ctx.output_dir, build_type='release')
       build.ninja(ctx.output_dir)


   def host_tests(ctx: PresubmitContext):
       build.gn_gen(PROJECT_ROOT, ctx.output_dir, run_host_tests='true')
       build.ninja(ctx.output_dir)


   # Avoid running some checks on certain paths.
   PATH_EXCLUSIONS = (
       re.compile(r'^external/'),
       re.compile(r'^vendor/'),
   )


   # Use the upstream pragma_once check, but apply a different set of path
   # filters with @filter_paths.
   @filter_paths(endswith='.h', exclude=PATH_EXCLUSIONS)
   def pragma_once(ctx: PresubmitContext):
       pw_presubmit.pragma_once(ctx)


   #
   # Presubmit check programs
   #
   QUICK = (
       # List some presubmit checks to run
       pragma_once,
       host_tests,
       # Use the upstream formatting checks, with custom path filters applied.
       format_code.presubmit_checks(exclude=PATH_EXCLUSIONS),
       # Include the upstream inclusive language check.
       banned_words.banned_words,
   )

   FULL = (
       QUICK,  # Add all checks from the 'quick' program
       release_build,
       # Use the upstream Python checks, with custom path filters applied.
       python_checks.all_checks(exclude=PATH_EXCLUSIONS),
   )

   PROGRAMS = pw_presubmit.Programs(quick=QUICK, full=FULL)


   def run(install: bool, **presubmit_args) -> int:
       """Process the --install argument then invoke pw_presubmit."""

       # Install the presubmit Git pre-push hook, if requested.
       if install:
           install_hook(__file__, 'pre-push', ['--base', 'HEAD~'],
                        git_repo.root())
           return 0

       return cli.run(root=PROJECT_ROOT, **presubmit_args)


   def main() -> int:
       """Run the presubmit checks for this repository."""
       parser = argparse.ArgumentParser(description=__doc__)
       cli.add_arguments(parser, PROGRAMS, 'quick')

       # Define an option for installing a Git pre-push hook for this script.
       parser.add_argument(
           '--install',
           action='store_true',
           help='Install the presubmit as a Git pre-push hook and exit.')

       return run(**vars(parser.parse_args()))

   if __name__ == '__main__':
       pw_cli.log.install(logging.INFO)
       sys.exit(main())

 ---------------------
 Code formatting tools
 ---------------------
 The ``pw_presubmit.format_code`` module formats supported source files using
 external code format tools. The file ``format_code.py`` can be invoked directly
 from the command line or from ``pw`` as ``pw format``.
	.. _module-pw_presubmit:

	============
	pw_presubmit
	============
	The presubmit module provides Python tools for running presubmit checks and
	checking and fixing code format. It also includes the presubmit check script for
	the Pigweed repository, ``pigweed_presubmit.py``.

	Presubmit checks are essential tools, but they take work to set up, and
	projects don’t always get around to it. The ``pw_presubmit`` module provides
	tools for setting up high quality presubmit checks for any project. We use this
	framework to run Pigweed’s presubmit on our workstations and in our automated
	building tools.

	The ``pw_presubmit`` module also includes ``pw format``, a tool that provides a
	unified interface for automatically formatting code in a variety of languages.
	With ``pw format``, you can format C, C++, Python, GN, and Go code according to
	configurations defined by your project. ``pw format`` leverages existing tools
	like ``clang-format``, and it’s simple to add support for new languages.

	.. image:: docs/pw_presubmit_demo.gif
	:alt: ``pw format`` demo
	:align: left

	The ``pw_presubmit`` package includes presubmit checks that can be used with any
	project. These checks include:

	* Check code format of several languages including C, C++, and Python
	* Initialize a Python environment
	* Run all Python tests
	* Run pylint
	* Run mypy
	* Ensure source files are included in the GN and Bazel builds
	* Build and run all tests with GN
	* Build and run all tests with Bazel
	* Ensure all header files contain ``#pragma once``

	-------------
	Compatibility
	-------------
	Python 3

	-------------------------------------------
	Creating a presubmit check for your project
	-------------------------------------------
	Creating a presubmit check for a project using ``pw_presubmit`` is simple, but
	requires some customization. Projects must define their own presubmit check
	Python script that uses the ``pw_presubmit`` package.

	A project's presubmit script can be registered as a
	:ref:`pw_cli <module-pw_cli>` plugin, so that it can be run as ``pw
	presubmit``.

	Setting up the command-line interface
	-------------------------------------
	The ``pw_presubmit.cli`` module sets up the command-line interface for a
	presubmit script. This defines a standard set of arguments for invoking
	presubmit checks. Its use is optional, but recommended.

	pw_presubmit.cli
	~~~~~~~~~~~~~~~~
	.. automodule:: pw_presubmit.cli
	:members: add_arguments, run

	Presubmit checks
	----------------
	A presubmit check is defined as a function or other callable. The function must
	accept one argument: a ``PresubmitContext``, which provides the paths on which
	to run. Presubmit checks communicate failure by raising an exception.

	Presubmit checks may use the ``filter_paths`` decorator to automatically filter
	the paths list for file types they care about.

	Either of these functions could be used as presubmit checks:

	.. code-block:: python

	@pw_presubmit.filter_paths(endswith='.py')
	def file_contains_ni(ctx: PresubmitContext):
	for path in ctx.paths:
	with open(path) as file:
	contents = file.read()
	if 'ni' not in contents and 'nee' not in contents:
	raise PresumitFailure('Files must say "ni"!', path=path)

	def run_the_build(_):
	subprocess.run(['make', 'release'], check=True)

	Presubmit checks functions are grouped into "programs" -- a named series of
	checks. Projects may find it helpful to have programs for different purposes,
	such as a quick program for local use and a full program for automated use. The
	:ref:`example script <example-script>` uses ``pw_presubmit.Programs`` to define
	``quick`` and ``full`` programs.

	Existing Presubmit Checks
	~~~~~~~~~~~~~~~~~~~~~~~~~
	A small number of presubmit checks are made available through ``pw_presubmit``
	modules.

	Code Formatting
	===============
	Formatting checks for a variety of languages are available from
	``pw_presubmit.format_code``. These include C/C++, Java, Go, Python, GN, and
	others. All of these checks can be included by adding
	``pw_presubmit.format_code.presubmit_checks()`` to a presubmit program. These
	all use language-specific formatters like clang-format or yapf.

	These will suggest fixes using ``pw format --fix``.

	#pragma once
	============
	There's a ``pragma_once`` check that confirms the first non-comment line of
	C/C++ headers is ``#pragma once``. This is enabled by adding
	``pw_presubmit.pragma_once`` to a presubmit program.

	Banned Words
	============
	.. banned-words: disable

	The banned words check looks for words that are typical of non-inclusive code,
	like using "master" and "slave" in place of "primary" and "secondary" or
	"sanity check" in place of "consistency check".

	.. banned-words: enable

	These checks can be disabled for individual lines with "banned-words: ignore"
	on the line in question or the line above it, or for entire blocks by using
	"banned-words: disable" before the block and "banned-words: enable" after the
	block.

	.. In case things get moved around in the previous paragraphs the enable line
	.. is repeated here: banned-words: enable.

	pw_presubmit
	~~~~~~~~~~~~
	.. automodule:: pw_presubmit
	:members: filter_paths, call, PresubmitFailure, Programs

	.. _example-script:

	Example
	-------
	A simple example presubmit check script follows. This can be copied-and-pasted
	to serve as a starting point for a project's presubmit check script.

	See ``pigweed_presubmit.py`` for a more complex presubmit check script example.

	.. code-block:: python

	"""Example presubmit check script."""

	import argparse
	import logging
	import os
	from pathlib import Path
	import re
	import sys
	from typing import List, Pattern

	try:
	import pw_cli.log
	except ImportError:
	print('ERROR: Activate the environment before running presubmits!',
	file=sys.stderr)
	sys.exit(2)

	import pw_presubmit
	from pw_presubmit import banned_words, build, cli, environment, format_code
	from pw_presubmit import git_repo, python_checks, filter_paths
	from pw_presubmit import PresubmitContext
	from pw_presubmit.install_hook import install_hook

	# Set up variables for key project paths.
	PROJECT_ROOT = Path(os.environ['MY_PROJECT_ROOT'])
	PIGWEED_ROOT = PROJECT_ROOT / 'pigweed'

	#
	# Initialization
	#
	def init_cipd(ctx: PresubmitContext):
	environment.init_cipd(PIGWEED_ROOT, ctx.output_dir)


	def init_virtualenv(ctx: PresubmitContext):
	environment.init_virtualenv(PIGWEED_ROOT,
	ctx.output_dir,
	setup_py_roots=[PROJECT_ROOT])


	# Rerun the build if files with these extensions change.
	_BUILD_EXTENSIONS = frozenset(
	['.rst', '.gn', '.gni', *format_code.C_FORMAT.extensions])


	#
	# Presubmit checks
	#
	def release_build(ctx: PresubmitContext):
	build.gn_gen(PROJECT_ROOT, ctx.output_dir, build_type='release')
	build.ninja(ctx.output_dir)


	def host_tests(ctx: PresubmitContext):
	build.gn_gen(PROJECT_ROOT, ctx.output_dir, run_host_tests='true')
	build.ninja(ctx.output_dir)


	# Avoid running some checks on certain paths.
	PATH_EXCLUSIONS = (
	re.compile(r'^external/'),
	re.compile(r'^vendor/'),
	)


	# Use the upstream pragma_once check, but apply a different set of path
	# filters with @filter_paths.
	@filter_paths(endswith='.h', exclude=PATH_EXCLUSIONS)
	def pragma_once(ctx: PresubmitContext):
	pw_presubmit.pragma_once(ctx)


	#
	# Presubmit check programs
	#
	QUICK = (
	# List some presubmit checks to run
	pragma_once,
	host_tests,
	# Use the upstream formatting checks, with custom path filters applied.
	format_code.presubmit_checks(exclude=PATH_EXCLUSIONS),
	# Include the upstream inclusive language check.
	banned_words.banned_words,
	)

	FULL = (
	QUICK, # Add all checks from the 'quick' program
	release_build,
	# Use the upstream Python checks, with custom path filters applied.
	python_checks.all_checks(exclude=PATH_EXCLUSIONS),
	)

	PROGRAMS = pw_presubmit.Programs(quick=QUICK, full=FULL)


	def run(install: bool, **presubmit_args) -> int:
	"""Process the --install argument then invoke pw_presubmit."""

	# Install the presubmit Git pre-push hook, if requested.
	if install:
	install_hook(__file__, 'pre-push', ['--base', 'HEAD~'],
	git_repo.root())
	return 0

	return cli.run(root=PROJECT_ROOT, **presubmit_args)


	def main() -> int:
	"""Run the presubmit checks for this repository."""
	parser = argparse.ArgumentParser(description=__doc__)
	cli.add_arguments(parser, PROGRAMS, 'quick')

	# Define an option for installing a Git pre-push hook for this script.
	parser.add_argument(
	'--install',
	action='store_true',
	help='Install the presubmit as a Git pre-push hook and exit.')

	return run(**vars(parser.parse_args()))

	if __name__ == '__main__':
	pw_cli.log.install(logging.INFO)
	sys.exit(main())

	---------------------
	Code formatting tools
	---------------------
	The ``pw_presubmit.format_code`` module formats supported source files using
	external code format tools. The file ``format_code.py`` can be invoked directly
	from the command line or from ``pw`` as ``pw format``.