blob: 70307b9407deb9ecfe48920d90b477f1c33bfa95 [file] [log] [blame]
.. _module-pw_env_setup:
A classic problem in the embedded space is reducing the time from git clone
to having a binary executing on a device. The issue is that an entire suite
of tools is needed for non-trivial production embedded projects. For example:
- A C++ compiler for your target device, and also for your host
- A build system or three; for example, GN, Ninja, CMake, Bazel
- A code formatting program like clang-format
- A debugger like OpenOCD to flash and debug your embedded device
- A known Python version with known modules installed for scripting
- A Go compiler for the Go-based command line tools
...and so on
In the server space, container solutions like Docker or Podman solve this;
however, in our experience container solutions are a mixed bag for embedded
systems development where one frequently needs access to native system
resources like USB devices, or must operate on Windows.
``pw_env_setup`` is our compromise solution for this problem that works on Mac,
Windows, and Linux. It leverages the Chrome packaging system `CIPD`_ to
bootstrap a Python installation, which in turn inflates a virtual
environment. The tooling is installed into your workspace, and makes no
changes to your system. This tooling is designed to be reused by any
.. _CIPD:
Users interact with ``pw_env_setup`` with two commands: ``.`` and
``.``. The bootstrap command always pulls down the current versions
of CIPD packages and sets up the Python virtual environment. The activate
command reinitializes a previously configured environment, and if none is found,
runs bootstrap.
.. note::
On Windows the scripts used to set up the environment are ``bootstrap.bat``
and ``activate.bat``. For simplicity they will be referred to with the ``.sh``
endings unless the distinction is relevant.
.. warning::
At this time ``pw_env_setup`` works for us, but isnt well tested. We dont
suggest relying on it just yet. However, we are interested in experience
reports; if you give it a try, please `send us a note`_ about your
.. _send us a note:
Using pw_env_setup in your project
Downstream Projects Using Pigweed's Packages
Projects using Pigweed can leverage ``pw_env_setup`` to install Pigweed's
dependencies or their own dependencies. Projects that only want to use Pigweed's
dependencies without modifying them can just source Pigweed's ````
and ```` scripts.
An example of what your project's `` could look like is below. This
assumes `` is at the top level of your repository.
.. code-block:: bash
# Do not include a "#!" line, this must be sourced and not executed.
# This assumes the user is sourcing this file from it's parent directory. See
# below for a more flexible way to handle this.
export PW_PROJECT_ROOT="$(_python_abspath "$(dirname "$PROJ_SETUP_SCRIPT_PATH")")"
# You may wish to check if the user is attempting to execute this script
# instead of sourcing it. See below for an example of how to handle that
# situation.
# Source Pigweed's bootstrap utility script.
# Using '.' instead of 'source' for POSIX compatibility. Since users don't use
# dash directly, using 'source' in most documentation so users don't get
# confused and try to `./`.
. "$PW_PROJECT_ROOT/third_party/pigweed/pw_env_setup/"
pw_check_root "$PW_ROOT"
pw_bootstrap --args... # See below for details about args.
pw_finalize bootstrap "$SETUP_SH"
Bazel Usage
It is possible to pull in a CIPD dependency into Bazel using WORKSPACE rules
rather than using ``. e.g.
.. code:: python
# Must be called before cipd_repository
name = "bloaty",
path = "pigweed/third_party/bloaty-embedded/${os=linux,mac}-${arch=amd64}",
tag = "git_revision:2d87d204057b419f5290f8d38b61b9c2c5b4fb52-2",
From here it is possible to get access to the Bloaty binaries using the
following command.
.. code:: sh
bazel run @bloaty//:bloaty -- --help
You may wish to allow sourcing `` from a different directory. In
that case you'll need the following at the top of ``.
.. code-block:: bash
_python_abspath () {
python -c "import os.path; print(os.path.abspath('$@'))"
# Use this code from Pigweed's bootstrap to find the path to this script when
# sourced. This should work with common shells. PW_CHECKOUT_ROOT is only used in
# presubmit tests with strange setups, and can be omitted if you're not using
# Pigweed's automated testing infrastructure.
if test -n "$PW_CHECKOUT_ROOT"; then
# Shell: bash.
elif test -n "$BASH"; then
PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$BASH_SOURCE")"
# Shell: zsh.
elif test -n "$ZSH_NAME"; then
PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "${(%):-%N}")"
# Shell: dash.
elif test ${0##*/} = dash; then
PROJ_SETUP_SCRIPT_PATH="$(_python_abspath \
"$(lsof -p $$ -Fn0 | tail -1 | sed 's#^[^/]*##;')")"
# If everything else fails, try $0. It could work.
PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$0")"
You may also wish to check if the user is attempting to execute ``
instead of sourcing it. Executing `` would download everything
required for the environment, but cannot modify the environment of the parent
process. To check for this add the following.
.. code-block:: bash
# Check if this file is being executed or sourced.
# If not running in Pigweed's automated testing infrastructure the
# SWARMING_BOT_ID check is unnecessary.
if [ -n "$SWARMING_BOT_ID" ]; then
# If set we're running on swarming and don't need this check.
elif [ -n "$ZSH_EVAL_CONTEXT" ]; then
case $ZSH_EVAL_CONTEXT in *:file) _pw_sourced=1;; esac
elif [ -n "$KSH_VERSION" ]; then
[ "$(cd $(dirname -- $0) && pwd -P)/$(basename -- $0)" != \
"$(cd $(dirname -- ${.sh.file}) && pwd -P)/$(basename -- ${.sh.file})" ] \
&& _pw_sourced=1
elif [ -n "$BASH_VERSION" ]; then
(return 0 2>/dev/null) && _pw_sourced=1
else # All other shells: examine $0 for known shell binary filenames
# Detects `sh` and `dash`; add additional shell filenames as needed.
case ${0##*/} in sh|dash) _pw_sourced=1;; esac
_pw_eval_sourced "$_pw_sourced"
Downstream Projects Using Different Packages
Projects depending on Pigweed but using additional or different packages should
copy the Pigweed `sample project`'s ```` and ``config.json`` and
update the call to ``pw_bootstrap``. Search for "downstream" for other places
that may require changes, like setting the ``PW_ROOT`` and ``PW_PROJECT_ROOT``
environment variables. Explanations of parts of ``config.json`` are described
.. _sample project:
CIPD package file. JSON file consisting of a list of dictionaries with "path",
"platforms", and "tags" keys. An example is below.
.. code-block:: json
"path": "infra/3pp/tools/go/${platform}",
"platforms": [
"tags": [
Target for installing Python packages. Downstream projects will need to
create targets to install their packages or only use Pigweed Python packages.
The root directory of your GN build tree, relative to ``PW_PROJECT_ROOT``.
This is the directory your project's ``.gn`` file is located in. If you're
only installing Pigweed Python packages, use the location of the Pigweed
An example of a config file is below.
.. code-block:: json
"cipd_package_files": [
"virtualenv": {
"gn_root": ".",
"gn_targets": [
In case the CIPD packages need to be referenced from other scripts, variables
like ``PW_${BASENAME}_CIPD_INSTALL_DIR`` point to the CIPD install directories,
where ``${BASENAME}`` is "PIGWEED" for
"pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json" and "LUCI" for
"pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json". This example would
set the following environment variables.
Environment Variables
The following environment variables affect env setup behavior. Most users will
never need to set these.
Location of CIPD cache dir. Defaults to ``$HOME/.cipd-cache-dir``.
If set, skip running ``pw doctor`` at end of bootstrap/activate. Intended to
be used by automated tools but not interactively.
Python executable to be used, for example "python2" or "python3". Defaults to
Location to which packages are installed. Defaults to ``.environment`` folder
within the checkout root.
Disable the spinner during env setup. Intended to be used when the output is
being redirected to a log.
Disables all non-error output.
Non-Shell Environments
If using this outside of bash—for example directly from an IDE or CI
system—users can process the ``actions.json`` file that's generated in the
environment directory. It lists variables to set, clear, and modify. An
example ``actions.json`` is shown below. The "append" and "prepend" actions
are listed in the order they should be applied, so the
``<pigweed-root>/out/host/host_tools`` entry should be at the beginning of
``PATH`` and not in the middle somewhere.
.. code-block:: json
"modify": {
"PATH": {
"append": [],
"prepend": [
"remove": []
"set": {
"PW_PROJECT_ROOT": "<pigweed-root>",
"PW_ROOT": "<pigweed-root>",
"_PW_ACTUAL_ENVIRONMENT_ROOT": "<pigweed-root>/.environment",
"PW_CIPD_INSTALL_DIR": "<pigweed-root>/.environment/cipd",
"CIPD_CACHE_DIR": "/usr/local/google/home/mohrr/.cipd-cache-dir",
"PW_PIGWEED_CIPD_INSTALL_DIR": "<pigweed-root>/.environment/cipd/pigweed",
"PW_LUCI_CIPD_INSTALL_DIR": "<pigweed-root>/.environment/cipd/luci",
"VIRTUAL_ENV": "<pigweed-root>/.environment/pigweed-venv",
The environment is set up by installing CIPD and Python packages in
``PW_ENVIRONMENT_ROOT`` or ``<checkout>/.environment``, and saving modifications
to environment variables in setup scripts in those directories. To support
multiple operating systems this is done in an operating system-agnostic manner
and then written into operating system-specific files to be sourced now and in
the future when running ```` instead of ````. In the
future these could be extended to C shell and PowerShell. A logical mapping of
high-level commands to system-specific initialization files is shown below.
.. image:: doc_resources/pw_env_setup_output.png
:alt: Mapping of high-level commands to system-specific commands.
:align: left