doc/guides/west/repo-tool.rst

.. _west-multi-repo:

Basics
######

This page introduces west's basic concepts and built-in commands along with
references to further reading.

.. _west-workspace:

Introduction
************

West's built-in commands allow you to work with *projects* (Git
repositories) under a common *workspace* directory. You can create a
workspace using the :ref:`west-init` command.

If you've followed the upstream Zephyr getting started guide, your
workspace looks like this:

.. code-block:: none

   zephyrproject/                 # west topdir
   ├── .west/                     # marks the location of the topdir
   │   └── config                 # per-workspace local configuration file
   ├── zephyr/                    # .git/ repo    │ the manifest repository,
   │   ├── west.yml               # manifest file │ never modified by west
   │   └── [... other files ...]                  │ after creation
   ├── modules/
   │   └── lib/
   │       └── tinycbor/          # .git/ project
   ├── net-tools/                 # .git/ project
   └── [ ... other projects ...]

Above, :file:`zephyrproject` is the name of the workspace's top level
directory, or *topdir*. (The name :file:`zephyrproject` is just an example
-- it could be anything, like ``z``, ``my-zephyr-workspace``, etc.)

The topdir contains the :file:`.west` directory. When west needs to find
the topdir, it searches for :file:`.west`, and uses its parent directory.
The search starts from the current working directory (and starts again from
the location in the :envvar:`ZEPHYR_BASE` environment variable as a
fallback if that fails). The file :file:`.west/config` is the workspace's
:ref:`local configuration file <west-config>`.

Every west workspace contains exactly one *manifest repository*, which is a
Git repository containing a *manifest file*. The location of the manifest
repository is given by the :ref:`manifest.path configuration option
<west-config-index>` in the local configuration file.

For upstream Zephyr, :file:`zephyr` is the manifest repository, but you can
configure west to use any Git repository in the workspace as the manifest
repository. The only requirement is that it contains a valid manifest file.
See :ref:`west-topologies` for information on other options, and
:ref:`west-manifests` for details on the manifest file format.

The manifest file is a YAML file that defines *projects*, which are the
additional Git repositories in the workspace managed by west. The manifest
file is named :file:`west.yml` by default; this can be overridden using the
``manifest.file`` local configuration option.

You use the :ref:`west-update` command to update the workspace's projects
based on the contents of the manifest file. The manifest file controls
things like where the project should be placed within the workspace, what
URL to clone it from if it's missing, and what Git revision should be
checked out locally.

Projects can be located anywhere inside the workspace, but they may not
"escape" it. Project repositories need not be located in subdirectories of
the manifest repository or as immediate subdirectories of the topdir.
However, projects must have paths inside the workspace. (You may replace a
project's repository directory within the workspace with a symbolic link to
elsewhere on your computer, but west will not do this for you.)

A workspace can contain additional Git repositories or other files and
directories not managed by west. West basically ignores anything in the
workspace except :file:`.west`, the manifest repository, and the projects
specified in the manifest file.

For upstream Zephyr, ``tinycbor`` and ``net-tools`` are projects. They are
specified in the manifest file :file:`zephyr/west.yml`. This file specifies
that ``tinycbor`` is located in the :file:`modules/lib/tinycbor` directory
beneath the workspace topdir. By default, the Zephyr :ref:`build system
<build_overview>` uses west to get the locations of all the projects in the
workspace, so any code they contain can be used as :ref:`modules`.

Finally, any repository managed by a west workspace (either the manifest
repository or any project repository) can define :ref:`west-extensions`.
Extensions are extra commands not built into west that you can run when
using that workspace.

The zephyr repository uses this feature to provide Zephyr-specific commands
like :ref:`west build <west-building>`. Defining these as extensions keeps
west's core agnostic to the specifics of any workspace's Zephyr version,
etc.

.. _west-struct:

Structure
*********

West's code is distributed via PyPI in a Python package named ``west``.
This distribution includes a launcher executable, which is also named
``west`` (or ``west.exe`` on Windows).

When west is installed, the launcher is placed by :file:`pip3` somewhere in
the user's filesystem (exactly where depends on the operating system, but
should be on the ``PATH`` :ref:`environment variable <env_vars>`). This
launcher is the command-line entry point to running both built-in commmands
like ``west init``, ``west update``, along with any extensions discovered
in the workspace.

In addition to its command-line interface, you can also use west's Python
APIs directly. See :ref:`west-apis` for details.

.. _west-manifest-rev:

The ``manifest-rev`` branch
***************************

West creates and controls a Git branch named ``manifest-rev`` in each
project. This branch points to the revision that the manifest file
specified for the project at the time :ref:`west-update` was last run.
Other workspace management commands may use ``manifest-rev`` as a reference
point for the upstream revision as of this latest update. Among other
purposes, the ``manifest-rev`` branch allows the manifest file to use SHAs
as project revisions.

Although ``manifest-rev`` is a normal Git branch, west will recreate and/or
reset it on the next update. For this reason, it is **dangerous**
to check it out or otherwise modify it yourself. For instance, any commits
you manually add to this branch may be lost the next time you run ``west
update``. Instead, check out a local branch with another name, and either
rebase it on top of a new ``manifest-rev``, or merge ``manifest-rev`` into
it.

.. note::

   West does not create a ``manifest-rev`` branch in the manifest repository,
   since west does not manage the manifest repository's branches or revisions.

The ``refs/west/*`` Git refs
****************************

West also reserves all Git refs that begin with ``refs/west/`` (such as
``refs/west/foo``) for itself in local project repositories. Unlike
``manifest-rev``, these refs are not regular branches. West's behavior here is
an implementation detail; users should not rely on these refs existence or
behavior.

.. _west-built-in-cmds:

Built-in Commands
*****************

This section gives an overview of west's built-in commands.

Some commands are related to Git commands with the same name, but operate
on the entire workspace. For example, ``west diff`` shows local changes in
multiple Git repositories in the workspace.

Some commands take projects as arguments. These arguments can be project
names as specified in the manifest file, or (as a fallback) paths to them
on the local file system. Omitting project arguments to commands which
accept them (such as ``west list``, ``west forall``, etc.) usually defaults
to using all projects in the manifest file plus the manifest repository
itself.

The following documentation does not exhaustively describe all commands.
For additional help, run ``west <command> -h`` (e.g. ``west init -h``).

.. _west-init:

``west init``
=============

This command creates a west workspace. It can be used in two ways:

1. Cloning a new manifest repository from a remote URL
2. Creating a workspace around an existing local manifest repository

**Option 1**: to clone a new manifest repository from a remote URL, use:

.. code-block:: none

   west init [-m URL] [--mr REVISION] [--mf FILE] [directory]

The new workspace is created in the given :file:`directory`, creating a new
:file:`.west` inside this directory. You can give the manifest URL using
the ``-m`` switch, the initial revision to check out using ``--mr``, and
the location of the manifest file within the repository using ``--mf``.

For example, running:

.. code-block:: shell

   west init -m https://github.com/zephyrproject-rtos/zephyr --mr v1.14.0 zp

would clone the upstream official zephyr repository into :file:`zp/zephyr`,
and check out the ``v1.14.0`` release. This command creates
:file:`zp/.west`, and set the ``manifest.path`` :ref:`configuration option
<west-config>` to ``zephyr`` to record the location of the manifest
repository in the workspace. The default manifest file location is used.

The ``-m`` option defaults to
``https://github.com/zephyrproject-rtos/zephyr``. The ``--mr`` option
defaults to ``master``. The ``--mf`` option defaults to ``west.yml``. If no
``directory`` is given, the current working directory is used.

**Option 2**: to create a workspace around an existing local manifest
repository, use:

.. code-block:: none

   west init -l [--mf FILE] directory

This creates :file:`.west` **next to** :file:`directory` in the file
system, and sets ``manifest.path`` to ``directory``.

As above, ``--mf`` defaults to ``west.yml``.

The ``west init`` command does not clone any of the projects defined in the
manifest file, regardless of whether ``-l`` is given. To do that, use
``west update``.

.. _west-update:

``west update``
---------------

This command clones and updates the projects specified in the :term:`west
manifest` file.

.. code-block:: none

   west update [-f {always,smart}] [-k] [-r]
               [--group-filter FILTER] [--stats] [PROJECT ...]

**Which projects are updated:**

By default, this command parses the manifest file, usually
:file:`west.yml`, and updates each project specified there.
If your manifest uses :ref:`project groups <west-manifest-groups>`, then
only the active projects are updated.

To operate on a subset of projects only, give ``PROJECT`` argument(s). Each
``PROJECT`` is either a project name as given in the manifest file, or a
path that points to the project within the workspace. If you specify
projects explicitly, they are updated regardless of whether they are active.

**Project update procedure:**

For each project that is updated, this command:

#. Initializes a local Git repository for the project in the workspace, if
   it does not already exist
#. Inspects the project's ``revision`` field in the manifest, and fetches
   it from the remote if it is not already available locally
#. Sets the project's :ref:`manifest-rev <west-manifest-rev>` branch to the
   commit specified by the revision in the previous step.
#. Checks out ``manifest-rev`` in the local working copy as a `detached
   HEADs <https://git-scm.com/docs/git-checkout#_detached_head>`_
#. If the manifest file specifies a :ref:`submodules
   <west-manifest-submodules>` key for the project, recursively updates
   the project's submodules as described below.

To avoid unnecessary fetches, ``west update`` will not fetch project
``revision`` values which are Git SHAs or tags that are already available
locally. This is the behavior when the ``-f`` (``--fetch``) option has its
default value, ``smart``. To force this command to fetch from project remotes
even if the revisions appear to be available locally, either use ``-f always``
or set the ``update.fetch`` :ref:`configuration option <west-config>` to
``always``. SHAs may be given as unique prefixes as long as they are acceptable
to Git [#fetchall]_.

If the project ``revision`` is a Git ref that is not a tag nor a SHA (i.e.
if the project is tracking a branch), ``west update`` always fetches,
regardless of ``-f`` and ``update.fetch``.

Some branch names might look like short SHAs, like ``deadbeef``. You can
always disambiguate this situation by prefixing the ``revision`` value with
``refs/heads/``, e.g. ``revision: refs/heads/deadbeef``.

For safety, ``west update`` uses ``git checkout --detach`` to check out a
detached ``HEAD`` at the manifest revision for each updated project,
leaving behind any branches which were already checked out. This is
typically a safe operation that will not modify any of your local branches.

However, if you had added some local commits onto a previously detached
``HEAD`` checked out by west, then git will warn you that you've left
behind some commits which are no longer referred to by any branch. These
may be garbage-collected and lost at some point in the future. To avoid
this if you have local commits in the project, make sure you have a local
branch checked out before running ``west update``.

If you would rather rebase any locally checked out branches instead, use
the ``-r`` (``--rebase``) option.

If you would like ``west update`` to keep local branches checked out as
long as they point to commits that are descendants of the new
``manifest-rev``, use the ``-k`` (``--keep-descendants``) option.

.. note::

   ``west update --rebase`` will fail in projects that have git conflicts
   between your branch and new commits brought in by the manifest. You
   should immediately resolve these conflicts as you usually do with
   ``git``, or you can use ``git -C <project_path> rebase --abort`` to
   ignore incoming changes for the moment.

   With a clean working tree, a plain ``west update`` never fails
   because it does not try to hold on to your commits and simply
   leaves them aside.

   ``west update --keep-descendants`` offers an intermediate option that
   never fails either but does not treat all projects the same:

   - in projects where your branch diverged from the incoming commits, it
     does not even try to rebase and leaves your branches behind just like a
     plain ``west update`` does;
   - in all other projects where no rebase or merge is needed it keeps
     your branches in place.

**One-time project group manipulation:**

The ``--group-filter`` option can be used to change which project groups
are enabled or disabled for the duration of a single ``west update`` command.
See :ref:`west-manifest-groups` for details on the project group feature.

The ``west update`` command behaves as if the ``--group-filter`` option's
value were appended to the ``manifest.group-filter``
:ref:`configuration option <west-config-index>`.

For example, running ``west update --group-filter=+foo,-bar`` would behave
the same way as if you had temporarily appended the string ``"+foo,-bar"``
to the value of ``manifest.group-filter``, run ``west update``, then restored
``manifest.group-filter`` to its original value.

Note that using the syntax ``--group-filter=VALUE`` instead of
``--group-filter VALUE`` avoids issues parsing command line options
if you just want to disable a single group, e.g. ``--group-filter=-bar``.

**Submodule update procedure:**

If a project in the manifest has a ``submodules`` key, the submodules are
updated as follows, depending on the value of the ``submodules`` key.

If the project has ``submodules: true``, west runs one of the following in
the project repository, depending on whether you run ``west update``
with the ``--rebase`` option or without it:

.. code-block::

   # without --rebase, e.g. "west update":
   git submodule update init --checkout --recursive

   # with --rebase, e.g. "west update --rebase":
   git submodule update init --rebase --recursive

Otherwise, the project has ``submodules: <list-of-submodules>``. In this
case, west runs one of the following in the project repository for each
submodule path in the list, depending on whether you run ``west update``
with the ``--rebase`` option or without it:

.. code-block::

   # without --rebase, e.g. "west update":
   git submodule update init --checkout --recursive <submodule-path>

   # with --rebase, e.g. "west update --rebase":
   git submodule update init --rebase --recursive <submodule-path>

.. _west-multi-repo-misc:

Other Repository Management Commands
====================================

West has a few more commands for managing the repositories in the
workspace, which are summarized here. Run ``west <command> -h`` for
detailed help.

- ``west list``: print a line of information about each project in the
  manifest, according to a format string
- ``west manifest``: manage the manifest file. See :ref:`west-manifest-cmd`.
- ``west diff``: run ``git diff`` in local project repositories
- ``west status``: run ``git status`` in local project repositories
- ``west forall``: run an arbitrary command in local project repositories

Additional Commands
===================

Finally, here is a summary of other built-in commands.

- ``west config``: get or set :ref:`configuration options <west-config>`
- ``west topdir``: print the top level directory of the west workspace
- ``west help``: get help about a command, or print information about all
  commands in the workspace, including :ref:`west-extensions`

.. _west-topologies:

Topologies supported
********************

The following are example source code topologies supported by west.

- T1: star topology, zephyr is the manifest repository
- T2: star topology, a Zephyr application is the manifest repository
- T3: forest topology, freestanding manifest repository

T1: Star topology, zephyr is the manifest repository
====================================================

- The zephyr repository acts as the central repository and specifies
  its :ref:`modules` in its :file:`west.yml`
- Analogy with existing mechanisms: Git submodules with zephyr as the
  super-project

This is the default. See :ref:`west-workspace` for how mainline Zephyr is an
example of this topology.

.. _west-t2:

T2: Star topology, application is the manifest repository
=========================================================

- Useful for those focused on a single application
- A repository containing a Zephyr application acts as the central repository
  and names other projects required to build it in its :file:`west.yml`. This
  includes the zephyr repository and any modules.
- Analogy with existing mechanisms: Git submodules with the application as
  the super-project, zephyr and other projects as submodules

A workspace using this topology looks like this:

.. code-block:: none

   west-workspace/
   │
   ├── application/         # .git/     │
   │   ├── CMakeLists.txt               │
   │   ├── prj.conf                     │  never modified by west
   │   ├── src/                         │
   │   │   └── main.c                   │
   │   └── west.yml         # main manifest with optional import(s) and override(s)
   │                                    │
   ├── modules/
   │   └── lib/
   │       └── tinycbor/    # .git/ project from either the main manifest or some import.
   │
   └── zephyr/              # .git/ project
       └── west.yml         # This can be partially imported with lower precedence or ignored.
                            # Only the 'manifest-rev' version can be imported.


Here is an example :file:`application/west.yml` which uses
:ref:`west-manifest-import`, available since west 0.7, to import Zephyr v2.2.0
and its modules into the application manifest file:

.. code-block:: yaml

   # Example T2 west.yml, using manifest imports.
   manifest:
     remotes:
       - name: zephyrproject-rtos
         url-base: https://github.com/zephyrproject-rtos
     projects:
       - name: zephyr
         remote: zephyrproject-rtos
         revision: v2.2.0
         import: true
     self:
       path: application

You can still selectively "override" individual Zephyr modules if you use
``import:`` in this way; see :ref:`west-manifest-ex1.3` for an example.

Another way to do the same thing is to copy/paste :file:`zephyr/west.yml`
to :file:`application/west.yml`, adding an entry for the zephyr
project itself, like this:

.. code-block:: yaml

   # Equivalent to the above, but with manually maintained Zephyr modules.
   manifest:
     remotes:
       - name: zephyrproject-rtos
         url-base: https://github.com/zephyrproject-rtos
     defaults:
       remote: zephyrproject-rtos
     projects:
       - name: zephyr
         revision: v2.2.0
         west-commands: scripts/west-commands.yml
       - name: net-tools
         revision: some-sha-goes-here
         path: tools/net-tools
       # ... other Zephyr modules go here ...
     self:
       path: application

(The ``west-commands`` is there for :ref:`west-build-flash-debug` and other
Zephyr-specific :ref:`west-extensions`. It's not necessary when using
``import``.)

The main advantage to using ``import`` is not having to track the revisions of
imported projects separately. In the above example, using ``import`` means
Zephyr's :ref:`module <modules>` versions are automatically determined from the
:file:`zephyr/west.yml` revision, instead of having to be copy/pasted (and
maintained) on their own.

T3: Forest topology
===================

- Useful for those supporting multiple independent applications or downstream
  distributions with no "central" repository
- A dedicated manifest repository which contains no Zephyr source code,
  and specifies a list of projects all at the same "level"
- Analogy with existing mechanisms: Google repo-based source distribution

A workspace using this topology looks like this:

.. code-block:: none

   west-workspace/
   ├── app1/               # .git/ project
   │   ├── CMakeLists.txt
   │   ├── prj.conf
   │   └── src/
   │       └── main.c
   ├── app2/               # .git/ project
   │   ├── CMakeLists.txt
   │   ├── prj.conf
   │   └── src/
   │       └── main.c
   ├── manifest-repo/      # .git/ never modified by west
   │   └── west.yml        # main manifest with optional import(s) and override(s)
   ├── modules/
   │   └── lib/
   │       └── tinycbor/   # .git/ project from either the main manifest or
   │                       #       frome some import
   │
   └── zephyr/             # .git/ project
       └── west.yml        # This can be partially imported with lower precedence or ignored.
                           # Only the 'manifest-rev' version can be imported.


Here is an example T3 :file:`manifest-repo/west.yml` which uses
:ref:`west-manifest-import`, available since west 0.7, to import Zephyr
v2.2.0 and its modules, then add the ``app1`` and ``app2`` projects:

.. code-block:: yaml

   manifest:
     remotes:
       - name: zephyrproject-rtos
         url-base: https://github.com/zephyrproject-rtos
       - name: your-git-server
         url-base: https://git.example.com/your-company
     defaults:
       remote: your-git-server
     projects:
       - name: zephyr
         remote: zephyrproject-rtos
         revision: v2.2.0
         import: true
       - name: app1
         revision: SOME_SHA_OR_BRANCH_OR_TAG
       - name: app2
         revision: ANOTHER_SHA_OR_BRANCH_OR_TAG
     self:
       path: manifest-repo

You can also do this "by hand" by copy/pasting :file:`zephyr/west.yml`
as shown :ref:`above <west-t2>` for the T2 topology, with the same caveats.

Private repositories
********************

You can use west to fetch from private repositories. There is nothing
west-specific about this.

The ``west update`` command essentially runs ``git fetch YOUR_PROJECT_URL``
when a project's ``manifest-rev`` branch must be updated to a newly fetched
commit. It's up to your environment to make sure the fetch succeeds.

You can either enter the password manually or use any of the `credential
helpers built in to Git`_. Since Git has credential storage built in, there is
no need for a west-specific feature.

The following sections cover common cases for running ``west update`` without
having to enter your password, as well as how to troubleshoot issues.

.. _credential helpers built in to Git:
   https://git-scm.com/docs/gitcredentials

Fetching via HTTPS
==================

On Windows when fetching from GitHub, recent versions of Git prompt you for
your GitHub password in a graphical window once, then store it for future use
(in a default installation). Passwordless fetching from GitHub should therefore
work "out of the box" on Windows after you have done it once.

In general, you can store your credentials on disk using the "store" git
credential helper. See the `git-credential-store`_ manual page for details.

To use this helper for all the repositories in your workspace, run:

.. code-block:: shell

   west forall -c "git config credential.helper store"

To use this helper on just the projects ``foo`` and ``bar``, run:

.. code-block:: shell

   west forall -c "git config credential.helper store" foo bar

To use this helper by default on your computer, run:

.. code-block:: shell

   git config --global credential.helper store

On GitHub, you can set up a `personal access token`_ to use in place of your
account password. (This may be required if your account has two-factor
authentication enabled, and may be preferable to storing your account password
in plain text even if two-factor authentication is disabed.)

If you don't want to store any credentials on the file system, you can store
them in memory temporarily using `git-credential-cache`_ instead.

.. _git-credential-store:
   https://git-scm.com/docs/git-credential-store#_examples
.. _git-credential-cache:
   https://git-scm.com/docs/git-credential-cache
.. _personal access token:
   https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token

Fetching via SSH
================

If your SSH key has no password, fetching should just work. If it does have a
password, you can avoid entering it manually every time using `ssh-agent`_.

On GitHub, see `Connecting to GitHub with SSH`_ for details on configuration
and key creation.

.. _ssh-agent:
   https://www.ssh.com/ssh/agent
.. _Connecting to GitHub with SSH:
   https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh

Troubleshooting
===============

One good way to troubleshoot fetching issues is to run ``west update`` in
verbose mode, like this:

.. code-block:: shell

   west -v update

The output includes Git commands run by west and their outputs. Look for
something like this:

.. code-block:: none

   === updating your_project (path/to/your/project):
   west.manifest: your_project: checking if cloned
   [...other west.manifest logs...]
   --- your_project: fetching, need revision SOME_SHA
   west.manifest: running 'git fetch ... https://github.com/your-username/your_project ...' in /some/directory

The ``git fetch`` command example in the last line above is what needs to
succeed. Go to ``/some/directory``, copy/paste and run the entire ``git fetch``
command, then debug from there using the documentation for your credential
storage helper.

If you can get the ``git fetch`` command to run successfully without prompting
for a password when you run it directly, you will be able to run ``west
update`` without entering your password in that same shell.

.. _PyPI:
   https://pypi.org/project/west/

.. _Zephyr issue #6770:
   https://github.com/zephyrproject-rtos/zephyr/issues/6770

.. _namespace package:
   https://www.python.org/dev/peps/pep-0420/

.. rubric:: Footnotes

.. [#fetchall]

   West may fetch all refs from the Git server when given a SHA as a revision.
   This is because some Git servers have historically not allowed fetching
   SHAs directly.