pw_env_setup/docs.rst - pigweed/pigweed - Git at Google

 .. _module-pw_env_setup:

 ------------
 pw_env_setup
 ------------
 A classic problem in the embedded space is reducing the time from git clone
 to having a binary executing on a device. The issue is that an entire suite
 of tools is needed for non-trivial production embedded projects. For example:

  - A C++ compiler for your target device, and also for your host
  - A build system or three; for example, GN, Ninja, CMake, Bazel
  - A code formatting program like clang-format
  - A debugger like OpenOCD to flash and debug your embedded device
  - A known Python version with known modules installed for scripting
  - A Go compiler for the Go-based command line tools

 ...and so on

 In the server space, container solutions like Docker or Podman solve this;
 however, in our experience container solutions are a mixed bag for embedded
 systems development where one frequently needs access to native system
 resources like USB devices, or must operate on Windows.

 ``pw_env_setup`` is our compromise solution for this problem that works on Mac,
 Windows, and Linux. It leverages the Chrome packaging system `CIPD`_ to
 bootstrap a Python installation, which in turn inflates a virtual
 environment. The tooling is installed into your workspace, and makes no
 changes to your system. This tooling is designed to be reused by any
 project.

 Users interact with  ``pw_env_setup`` with two commands: ``. bootstrap.sh`` and
 ``. activate.sh``. The bootstrap command always pulls down the current versions
 of CIPD packages and sets up the Python virtual environment. The activate
 command reinitializes a previously configured environment, and if none is found,
 runs bootstrap.

 .. _CIPD: https://github.com/luci/luci-go/tree/master/cipd

 .. note::
   On Windows the scripts used to set up the environment are ``bootstrap.bat``
   and ``activate.bat``. For simplicity they will be referred to with the ``.sh``
   endings unless the distinction is relevant.

 By default packages will be installed in a ``.environment`` folder within the
 checkout root, and CIPD will cache files in ``$HOME/.cipd-cache-dir``. These
 paths can be overridden by setting ``PW_ENVIRONMENT_ROOT`` and
 ``CIPD_CACHE_DIR``, respectively.

 .. warning::
   At this time ``pw_env_setup`` works for us, but isn’t well tested. We don’t
   suggest relying on it just yet. However, we are interested in experience
   reports; if you give it a try, please `send us a note`_ about your
   experience.

 .. _send us a note: pigweed@googlegroups.com

 ==================================
 Using pw_env_setup in your project
 ==================================

 Downstream Projects Using Pigweed's Packages
 ********************************************

 Projects using Pigweed can leverage ``pw_env_setup`` to install Pigweed's
 dependencies or their own dependencies. Projects that only want to use Pigweed's
 dependencies without modifying them can just source Pigweed's ``bootstrap.sh``
 and ``activate.sh`` scripts.

 An example of what your project's `bootstrap.sh` could look like is below. This
 assumes `bootstrap.sh` is at the top level of your repository.

 .. code-block:: bash

   # Do not include a "#!" line, this must be sourced and not executed.

   # This assumes the user is sourcing this file from it's parent directory. See
   # below for a more flexible way to handle this.
   PROJ_SETUP_SCRIPT_PATH="$(pwd)/bootstrap.sh"

   export PROJ_ROOT="$(_python_abspath "$(dirname "$PROJ_SETUP_SCRIPT_PATH")")"

   # You may wish to check if the user is attempting to execute this script
   # instead of sourcing it. See below for an example of how to handle that
   # situation.

   # Source Pigweed's bootstrap script.
   # Using '.' instead of 'source' for dash compatibility. Since users don't use
   # dash directly, using 'source' in documentation so users don't get confused
   # and try to `./bootstrap.sh`.
   . "$PROJ_ROOT/third_party/pigweed/$(basename "$PROJ_SETUP_SCRIPT_PATH")"

 User-Friendliness
 -----------------

 You may wish to allow sourcing `bootstrap.sh` from a different directory. In
 that case you'll need the following at the top of `bootstrap.sh`.

 .. code-block:: bash

   _python_abspath () {
     python -c "import os.path; print(os.path.abspath('$@'))"
   }

   # Use this code from Pigweed's bootstrap to find the path to this script when
   # sourced. This should work with common shells. PW_CHECKOUT_ROOT is only used in
   # presubmit tests with strange setups, and can be omitted if you're not using
   # Pigweed's automated testing infrastructure.
   if test -n "$PW_CHECKOUT_ROOT"; then
     PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$PW_CHECKOUT_ROOT/bootstrap.sh")"
     unset PW_CHECKOUT_ROOT
   # Shell: bash.
   elif test -n "$BASH"; then
     PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$BASH_SOURCE")"
   # Shell: zsh.
   elif test -n "$ZSH_NAME"; then
     PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "${(%):-%N}")"
   # Shell: dash.
   elif test ${0##*/} = dash; then
     PROJ_SETUP_SCRIPT_PATH="$(_python_abspath \
       "$(lsof -p $$ -Fn0 | tail -1 | sed 's#^[^/]*##;')")"
   # If everything else fails, try $0. It could work.
   else
     PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$0")"
   fi

 You may also wish to check if the user is attempting to execute `bootstrap.sh`
 instead of sourcing it. Executing `bootstrap.sh` would download everything
 required for the environment, but cannot modify the environment of the parent
 process. To check for this add the following.

 .. code-block:: bash

   # Check if this file is being executed or sourced.
   _pw_sourced=0
   # If not running in Pigweed's automated testing infrastructure the
   # SWARMING_BOT_ID check is unnecessary.
   if [ -n "$SWARMING_BOT_ID" ]; then
     # If set we're running on swarming and don't need this check.
     _pw_sourced=1
   elif [ -n "$ZSH_EVAL_CONTEXT" ]; then
     case $ZSH_EVAL_CONTEXT in *:file) _pw_sourced=1;; esac
   elif [ -n "$KSH_VERSION" ]; then
     [ "$(cd $(dirname -- $0) && pwd -P)/$(basename -- $0)" != \
       "$(cd $(dirname -- ${.sh.file}) && pwd -P)/$(basename -- ${.sh.file})" ] \
       && _pw_sourced=1
   elif [ -n "$BASH_VERSION" ]; then
     (return 0 2>/dev/null) && _pw_sourced=1
   else  # All other shells: examine $0 for known shell binary filenames
     # Detects `sh` and `dash`; add additional shell filenames as needed.
     case ${0##*/} in sh|dash) _pw_sourced=1;; esac
   fi

   if [ "$_pw_sourced" -eq 0 ]; then
     _S_NAME=$(basename "$PROJ_SETUP_SCRIPT_PATH" .sh)
     echo "Error: Attempting to $_S_NAME in a subshell"
     echo "  Since $_S_NAME.sh modifies your shell's environment variables, it"
     echo "  must be sourced rather than executed. In particular, "
     echo "  'bash $_S_NAME.sh' will not work since the modified environment "
     echo "  will get destroyed at the end of the script. Instead, source the "
     echo "  script's contents in your shell:"
     echo ""
     echo "    \$ source $_S_NAME.sh"
     exit 1
   fi

 Downstream Projects Using Different Packages
 ********************************************

 Projects depending on Pigweed but using additional or different packages should
 copy Pigweed's ``bootstrap.sh`` and update the call to ``env_setup.py``. Search
 for "downstream" for other places that may require changes, like setting the
 ``PW_ROOT`` environment variable. Relevant arguments to ``env_setup.py`` are
 listed here.

 ``--use-pigweed-defaults``
   Use Pigweed default values in addition to the other switches.

 ``--cipd-package-file path/to/packages.json``
   CIPD package file. JSON file consisting of a list of dictionaries with "path"
   and "tags" keys, where "tags" is a list of strings.

 ``--virtualenv-requierements path/to/requirements.txt``
   Pip requirements file. Compiled with pip-compile.

 ``--virtualenv-setup-py-root path/to/directory``
   Directory in which to recursively search for ``setup.py`` files.

 ``--cargo-package-file path/to/packages.txt``
   Rust cargo packages to install. Lines with package name and version separated
   by a space. Has no effect without ``--enable-cargo``.

 ``--enable-cargo``
   Enable cargo package installation.

 An example of the changed env_setup.py line is below.

 .. code-block:: bash

   "$ROOT/third_party/pigweed/pw_env_setup/py/pw_env_setup/env_setup.py" \
     --shell-file "$SETUP_SH" \
     --install-dir "$_PW_ACTUAL_ENVIRONMENT_ROOT" \
     --use-pigweed-defaults \
     --cipd-package-file "$ROOT/path/to/cipd.json" \
     --virtualenv-setup-py-root "$ROOT"

 Projects wanting some of the Pigweed environment packages but not all of them
 should not use ``--use-pigweed-defaults`` and must manually add the references
 to Pigweed default packages through the other arguments. The arguments below
 are identical to using ``--use-pigweed-defaults``.

 .. code-block:: bash

   --cipd-package-file
   "$PW_ROOT/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json"
   --cipd-package-file
   "$PW_ROOT/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json"
   --virtualenv-requirements
   "$PW_ROOT/pw_env_setup/py/pw_env_setup/virtualenv_setup/requirements.txt"
   --virtualenv-setup-py-root
   "$PW_ROOT"
   --cargo-package-file
   "$PW_ROOT/pw_env_setup/py/pw_env_setup/cargo_setup/packages.txt"

 Implementation
 **************

 The environment is set up by installing CIPD and Python packages in
 ``PW_ENVIRONMENT_ROOT`` or ``<checkout>/.environment``, and saving modifications
 to environment variables in setup scripts in those directories. To support
 multiple operating systems this is done in an operating system-agnostic manner
 and then written into operating system-specific files to be sourced now and in
 the future when running ``activate.sh`` instead of ``bootstrap.sh``. In the
 future these could be extended to C shell and PowerShell. A logical mapping of
 high-level commands to system-specific initialization files is shown below.

 .. image:: doc_resources/pw_env_setup_output.png
    :alt: Mapping of high-level commands to system-specific commands.
    :align: left
	.. _module-pw_env_setup:

	------------
	pw_env_setup
	------------
	A classic problem in the embedded space is reducing the time from git clone
	to having a binary executing on a device. The issue is that an entire suite
	of tools is needed for non-trivial production embedded projects. For example:

	- A C++ compiler for your target device, and also for your host
	- A build system or three; for example, GN, Ninja, CMake, Bazel
	- A code formatting program like clang-format
	- A debugger like OpenOCD to flash and debug your embedded device
	- A known Python version with known modules installed for scripting
	- A Go compiler for the Go-based command line tools

	...and so on

	In the server space, container solutions like Docker or Podman solve this;
	however, in our experience container solutions are a mixed bag for embedded
	systems development where one frequently needs access to native system
	resources like USB devices, or must operate on Windows.

	``pw_env_setup`` is our compromise solution for this problem that works on Mac,
	Windows, and Linux. It leverages the Chrome packaging system `CIPD`_ to
	bootstrap a Python installation, which in turn inflates a virtual
	environment. The tooling is installed into your workspace, and makes no
	changes to your system. This tooling is designed to be reused by any
	project.

	Users interact with ``pw_env_setup`` with two commands: ``. bootstrap.sh`` and
	``. activate.sh``. The bootstrap command always pulls down the current versions
	of CIPD packages and sets up the Python virtual environment. The activate
	command reinitializes a previously configured environment, and if none is found,
	runs bootstrap.

	.. _CIPD: https://github.com/luci/luci-go/tree/master/cipd

	.. note::
	On Windows the scripts used to set up the environment are ``bootstrap.bat``
	and ``activate.bat``. For simplicity they will be referred to with the ``.sh``
	endings unless the distinction is relevant.

	By default packages will be installed in a ``.environment`` folder within the
	checkout root, and CIPD will cache files in ``$HOME/.cipd-cache-dir``. These
	paths can be overridden by setting ``PW_ENVIRONMENT_ROOT`` and
	``CIPD_CACHE_DIR``, respectively.

	.. warning::
	At this time ``pw_env_setup`` works for us, but isn’t well tested. We don’t
	suggest relying on it just yet. However, we are interested in experience
	reports; if you give it a try, please `send us a note`_ about your
	experience.

	.. _send us a note: pigweed@googlegroups.com

	==================================
	Using pw_env_setup in your project
	==================================

	Downstream Projects Using Pigweed's Packages
	********************************************

	Projects using Pigweed can leverage ``pw_env_setup`` to install Pigweed's
	dependencies or their own dependencies. Projects that only want to use Pigweed's
	dependencies without modifying them can just source Pigweed's ``bootstrap.sh``
	and ``activate.sh`` scripts.

	An example of what your project's `bootstrap.sh` could look like is below. This
	assumes `bootstrap.sh` is at the top level of your repository.

	.. code-block:: bash

	# Do not include a "#!" line, this must be sourced and not executed.

	# This assumes the user is sourcing this file from it's parent directory. See
	# below for a more flexible way to handle this.
	PROJ_SETUP_SCRIPT_PATH="$(pwd)/bootstrap.sh"

	export PROJ_ROOT="$(_python_abspath "$(dirname "$PROJ_SETUP_SCRIPT_PATH")")"

	# You may wish to check if the user is attempting to execute this script
	# instead of sourcing it. See below for an example of how to handle that
	# situation.

	# Source Pigweed's bootstrap script.
	# Using '.' instead of 'source' for dash compatibility. Since users don't use
	# dash directly, using 'source' in documentation so users don't get confused
	# and try to `./bootstrap.sh`.
	. "$PROJ_ROOT/third_party/pigweed/$(basename "$PROJ_SETUP_SCRIPT_PATH")"

	User-Friendliness
	-----------------

	You may wish to allow sourcing `bootstrap.sh` from a different directory. In
	that case you'll need the following at the top of `bootstrap.sh`.

	.. code-block:: bash

	_python_abspath () {
	python -c "import os.path; print(os.path.abspath('$@'))"
	}

	# Use this code from Pigweed's bootstrap to find the path to this script when
	# sourced. This should work with common shells. PW_CHECKOUT_ROOT is only used in
	# presubmit tests with strange setups, and can be omitted if you're not using
	# Pigweed's automated testing infrastructure.
	if test -n "$PW_CHECKOUT_ROOT"; then
	PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$PW_CHECKOUT_ROOT/bootstrap.sh")"
	unset PW_CHECKOUT_ROOT
	# Shell: bash.
	elif test -n "$BASH"; then
	PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$BASH_SOURCE")"
	# Shell: zsh.
	elif test -n "$ZSH_NAME"; then
	PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "${(%):-%N}")"
	# Shell: dash.
	elif test ${0##*/} = dash; then
	PROJ_SETUP_SCRIPT_PATH="$(_python_abspath \
	"$(lsof -p $$ -Fn0 \| tail -1 \| sed 's#^[^/]*##;')")"
	# If everything else fails, try $0. It could work.
	else
	PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$0")"
	fi

	You may also wish to check if the user is attempting to execute `bootstrap.sh`
	instead of sourcing it. Executing `bootstrap.sh` would download everything
	required for the environment, but cannot modify the environment of the parent
	process. To check for this add the following.

	.. code-block:: bash

	# Check if this file is being executed or sourced.
	_pw_sourced=0
	# If not running in Pigweed's automated testing infrastructure the
	# SWARMING_BOT_ID check is unnecessary.
	if [ -n "$SWARMING_BOT_ID" ]; then
	# If set we're running on swarming and don't need this check.
	_pw_sourced=1
	elif [ -n "$ZSH_EVAL_CONTEXT" ]; then
	case $ZSH_EVAL_CONTEXT in *:file) _pw_sourced=1;; esac
	elif [ -n "$KSH_VERSION" ]; then
	[ "$(cd $(dirname -- $0) && pwd -P)/$(basename -- $0)" != \
	"$(cd $(dirname -- ${.sh.file}) && pwd -P)/$(basename -- ${.sh.file})" ] \
	&& _pw_sourced=1
	elif [ -n "$BASH_VERSION" ]; then
	(return 0 2>/dev/null) && _pw_sourced=1
	else # All other shells: examine $0 for known shell binary filenames
	# Detects `sh` and `dash`; add additional shell filenames as needed.
	case ${0##*/} in sh\|dash) _pw_sourced=1;; esac
	fi

	if [ "$_pw_sourced" -eq 0 ]; then
	_S_NAME=$(basename "$PROJ_SETUP_SCRIPT_PATH" .sh)
	echo "Error: Attempting to $_S_NAME in a subshell"
	echo " Since $_S_NAME.sh modifies your shell's environment variables, it"
	echo " must be sourced rather than executed. In particular, "
	echo " 'bash $_S_NAME.sh' will not work since the modified environment "
	echo " will get destroyed at the end of the script. Instead, source the "
	echo " script's contents in your shell:"
	echo ""
	echo " \$ source $_S_NAME.sh"
	exit 1
	fi

	Downstream Projects Using Different Packages
	********************************************

	Projects depending on Pigweed but using additional or different packages should
	copy Pigweed's ``bootstrap.sh`` and update the call to ``env_setup.py``. Search
	for "downstream" for other places that may require changes, like setting the
	``PW_ROOT`` environment variable. Relevant arguments to ``env_setup.py`` are
	listed here.

	``--use-pigweed-defaults``
	Use Pigweed default values in addition to the other switches.

	``--cipd-package-file path/to/packages.json``
	CIPD package file. JSON file consisting of a list of dictionaries with "path"
	and "tags" keys, where "tags" is a list of strings.

	``--virtualenv-requierements path/to/requirements.txt``
	Pip requirements file. Compiled with pip-compile.

	``--virtualenv-setup-py-root path/to/directory``
	Directory in which to recursively search for ``setup.py`` files.

	``--cargo-package-file path/to/packages.txt``
	Rust cargo packages to install. Lines with package name and version separated
	by a space. Has no effect without ``--enable-cargo``.

	``--enable-cargo``
	Enable cargo package installation.

	An example of the changed env_setup.py line is below.

	.. code-block:: bash

	"$ROOT/third_party/pigweed/pw_env_setup/py/pw_env_setup/env_setup.py" \
	--shell-file "$SETUP_SH" \
	--install-dir "$_PW_ACTUAL_ENVIRONMENT_ROOT" \
	--use-pigweed-defaults \
	--cipd-package-file "$ROOT/path/to/cipd.json" \
	--virtualenv-setup-py-root "$ROOT"

	Projects wanting some of the Pigweed environment packages but not all of them
	should not use ``--use-pigweed-defaults`` and must manually add the references
	to Pigweed default packages through the other arguments. The arguments below
	are identical to using ``--use-pigweed-defaults``.

	.. code-block:: bash

	--cipd-package-file
	"$PW_ROOT/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json"
	--cipd-package-file
	"$PW_ROOT/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json"
	--virtualenv-requirements
	"$PW_ROOT/pw_env_setup/py/pw_env_setup/virtualenv_setup/requirements.txt"
	--virtualenv-setup-py-root
	"$PW_ROOT"
	--cargo-package-file
	"$PW_ROOT/pw_env_setup/py/pw_env_setup/cargo_setup/packages.txt"

	Implementation
	**************

	The environment is set up by installing CIPD and Python packages in
	``PW_ENVIRONMENT_ROOT`` or ``<checkout>/.environment``, and saving modifications
	to environment variables in setup scripts in those directories. To support
	multiple operating systems this is done in an operating system-agnostic manner
	and then written into operating system-specific files to be sourced now and in
	the future when running ``activate.sh`` instead of ``bootstrap.sh``. In the
	future these could be extended to C shell and PowerShell. A logical mapping of
	high-level commands to system-specific initialization files is shown below.

	.. image:: doc_resources/pw_env_setup_output.png
	:alt: Mapping of high-level commands to system-specific commands.
	:align: left