| .. _west-multi-repo: |
| |
| Multiple Repository Management |
| ############################## |
| |
| This page introduces basic concepts related to west and its multiple repository |
| management features, and gives an overview of the associated commands. See |
| :ref:`west-history` and `Zephyr issue #6770`_ for additional discussion, |
| rationale, and motivation. |
| |
| .. note:: |
| |
| West's multi-repo commands are meant to augment Git in minor ways for |
| multi-repo work, not to replace it. For tasks that only operate on one |
| repository, just use plain Git commands. |
| |
| .. _west-workspace: |
| |
| Introduction |
| ************ |
| |
| West's built-in commands allow you to work with projects composed of multiple |
| Git repositories installed under a common parent directory, which we call a |
| *west workspace* (before west 0.7, this was called a *west installation*). This |
| works similarly to `Git Submodules |
| <https://git-scm.com/book/en/v2/Git-Tools-Submodules>`_ and Google's `repo |
| <https://gerrit.googlesource.com/git-repo/>`_. |
| |
| A west workspace is the result of running the ``west init`` command, which |
| is described in more detail below. For upstream Zephyr, the workspace looks |
| like this: |
| |
| .. code-block:: none |
| |
| zephyrproject |
| ├── .west |
| │ └── config |
| ├── zephyr |
| │ ├── west.yml |
| │ └── [... other files ...] |
| ├── modules |
| │ └── lib |
| │ └── tinycbor |
| ├── net-tools |
| └── [ ... other projects ...] |
| |
| Above, :file:`zephyrproject` is the name of the west workspace's root |
| directory. This name is just an example -- it could be anything, like ``z``, |
| ``my-zephyr-workspace``, etc. The file :file:`.west/config` is the |
| workspace's :ref:`local configuration file <west-config>`. |
| |
| Every west workspace contains exactly one *manifest repository*, which is a |
| Git repository containing a file named :file:`west.yml`, which is the *west |
| manifest*. The location of the manifest repository is given by the |
| :ref:`manifest.path configuration option <west-config-index>` in the local |
| configuration file. The manifest file, along with west's configuration files, |
| controls the workspace's behavior. For upstream Zephyr, :file:`zephyr` is |
| the manifest repository, but you can configure west to use any Git repository |
| in the workspace as the manifest repository. The only requirement is that it |
| contains a valid manifest file. See :ref:`west-manifests` for more details on |
| what this means. |
| |
| Both of the :file:`tinycbor` and :file:`net-tools` directories are *projects* |
| managed by west, and configured in the manifest file. A west workspace can |
| contain arbitrarily many projects. As shown above, projects can be located |
| anywhere in the workspace. They don't have to be subdirectories of the |
| manifest directory, and they can be inside of arbitrary subdirectories inside |
| the workspace's root directory. By default, the Zephyr build system uses |
| west to get the locations of all the projects in the workspace, so any code |
| they contain can be used by applications. This behavior can be overridden using |
| the ``ZEPHYR_MODULES`` CMake variable; see :ref:`modules` for details. |
| |
| Finally, any repository managed by a west workspace can contain |
| :ref:`extension commands <west-extensions>`, which are extra west commands |
| provided by that project. This includes the manifest repository and any project |
| repository. |
| |
| Topologies supported |
| ******************** |
| |
| The following are example source code topologies supported by west. |
| |
| - T1: star topology, zephyr is the manifest repository |
| - T2: star topology, a Zephyr application is the manifest repository |
| - T3: forest topology, freestanding manifest repository |
| |
| T1: Star topology, zephyr is the manifest repository |
| ==================================================== |
| |
| - The zephyr repository acts as the central repository and specifies |
| its :ref:`modules` in its :file:`west.yml` |
| - Analogy with existing mechanisms: Git submodules with zephyr as the |
| super-project |
| |
| This is the default. See :ref:`west-workspace` for how mainline Zephyr is an |
| example of this topology. |
| |
| .. _west-t2: |
| |
| T2: Star topology, application is the manifest repository |
| ========================================================= |
| |
| - Useful for those focused on a single application |
| - A repository containing a Zephyr application acts as the central repository |
| and names other projects required to build it in its :file:`west.yml`. This |
| includes the zephyr repository and any modules. |
| - Analogy with existing mechanisms: Git submodules with the application as |
| the super-project, zephyr and other projects as submodules |
| |
| A workspace using this topology looks like this: |
| |
| .. code-block:: none |
| |
| west-workspace |
| ├── application |
| │ ├── CMakeLists.txt |
| │ ├── prj.conf |
| │ ├── src |
| │ │ └── main.c |
| │ └── west.yml |
| ├── modules |
| │ └── lib |
| │ └── tinycbor |
| └── zephyr |
| |
| Here is an example :file:`application/west.yml` which uses |
| :ref:`west-manifest-import`, available since west 0.7, to import Zephyr v2.2.0 |
| and its modules into the application manifest file: |
| |
| .. code-block:: yaml |
| |
| # Example T2 west.yml, using manifest imports. |
| manifest: |
| remotes: |
| - name: zephyrproject-rtos |
| url-base: https://github.com/zephyrproject-rtos |
| projects: |
| - name: zephyr |
| remote: zephyrproject-rtos |
| revision: v2.2.0 |
| import: true |
| self: |
| path: application |
| |
| You can still selectively "override" individual Zephyr modules if you use |
| ``import:`` in this way; see :ref:`west-manifest-ex1.3` for an example. |
| |
| Another way to do the same thing is to copy/paste :file:`zephyr/west.yml` |
| to :file:`application/west.yml`, adding an entry for the zephyr |
| project itself, like this: |
| |
| .. code-block:: yaml |
| |
| # Equivalent to the above, but with manually maintained Zephyr modules. |
| manifest: |
| remotes: |
| - name: zephyrproject-rtos |
| url-base: https://github.com/zephyrproject-rtos |
| defaults: |
| remote: zephyrproject-rtos |
| projects: |
| - name: zephyr |
| revision: v2.2.0 |
| west-commands: scripts/west-commands.yml |
| - name: net-tools |
| revision: some-sha-goes-here |
| path: tools/net-tools |
| # ... other Zephyr modules go here ... |
| self: |
| path: application |
| |
| (The ``west-commands`` is there for :ref:`west-build-flash-debug` and other |
| Zephyr-specific :ref:`west-extensions`. It's not necessary when using |
| ``import``.) |
| |
| The main advantage to using ``import`` is not having to track the revisions of |
| imported projects separately. In the above example, using ``import`` means |
| Zephyr's :ref:`module <modules>` versions are automatically determined from the |
| :file:`zephyr/west.yml` revision, instead of having to be copy/pasted (and |
| maintained) on their own. |
| |
| T3: Forest topology |
| =================== |
| |
| - Useful for those supporting multiple independent applications or downstream |
| distributions with no "central" repository |
| - A dedicated manifest repository which contains no Zephyr source code, |
| and specifies a list of projects all at the same "level" |
| - Analogy with existing mechanisms: Google repo-based source distribution |
| |
| A workspace using this topology looks like this: |
| |
| .. code-block:: none |
| |
| west-workspace |
| ├── app1 |
| │ ├── CMakeLists.txt |
| │ ├── prj.conf |
| │ └── src |
| │ └── main.c |
| ├── app2 |
| │ ├── CMakeLists.txt |
| │ ├── prj.conf |
| │ └── src |
| │ └── main.c |
| ├── manifest-repo |
| │ └── west.yml |
| ├── modules |
| │ └── lib |
| │ └── tinycbor |
| └── zephyr |
| |
| Here is an example T3 :file:`manifest-repo/west.yml` which uses |
| :ref:`west-manifest-import`, available since west 0.7, to import Zephyr |
| v2.2.0 and its modules, then add the ``app1`` and ``app2`` projects: |
| |
| .. code-block:: yaml |
| |
| manifest: |
| remotes: |
| - name: zephyrproject-rtos |
| url-base: https://github.com/zephyrproject-rtos |
| - name: your-git-server |
| url-base: https://git.example.com/your-company |
| defaults: |
| remote: your-git-server |
| projects: |
| - name: zephyr |
| remote: zephyrproject-rtos |
| revision: v2.2.0 |
| import: true |
| - name: app1 |
| revision: SOME_SHA_OR_BRANCH_OR_TAG |
| - name: app2 |
| revision: ANOTHER_SHA_OR_BRANCH_OR_TAG |
| self: |
| path: manifest-repo |
| |
| You can also do this "by hand" by copy/pasting :file:`zephyr/west.yml` |
| as shown :ref:`above <west-t2>` for the T2 topology, with the same caveats. |
| |
| .. _west-struct: |
| |
| West Structure |
| ************** |
| |
| West's code is distributed via PyPI in a Python package named ``west``. See |
| :ref:`west-apis` for API documentation. |
| |
| This distribution also includes a launcher executable, also named ``west``. |
| When west is installed, the launcher is placed by :file:`pip3` somewhere in the |
| user's filesystem (exactly where depends on the operating system, but should be |
| on the ``PATH`` :ref:`environment variable <env_vars>`). This launcher is the |
| command-line entry point. |
| |
| .. _west-manifest-rev: |
| |
| The ``manifest-rev`` branch |
| *************************** |
| |
| West creates a branch named ``manifest-rev`` in each project, pointing to the |
| commit the project's revision resolves to. The branch is updated whenever |
| project data is fetched by ``west update``. Other multi-repo commands also use |
| ``manifest-rev`` as a reference for the upstream revision as of the most recent |
| update. See :ref:`west-multi-repo-cmds`, below, for more information. |
| |
| ``manifest-rev`` is a normal Git branch, but if you delete or otherwise modify |
| it, west will recreate and/or reset it as if with ``git reset --hard`` on the |
| next update (though ``git update-ref`` is used internally). For this reason, it |
| is normally a **bad idea to modify it yourself**. ``manifest-rev`` was added to |
| allow SHAs as project revisions in the manifest, and to give a consistent |
| reference for the current upstream revision regardless of how the manifest |
| changes over time. |
| |
| .. note:: |
| |
| West does not create a ``manifest-rev`` branch in the manifest repository, |
| since west does not manage the manifest repository's branches or revisions. |
| |
| .. _west-multi-repo-cmds: |
| |
| Multi-Repo Commands |
| ******************* |
| |
| This section gives a quick overview of the multi-repo commands, split up by |
| functionality. Some commands loosely mimic the corresponding Git command, but |
| in a multi-repo context (e.g. ``west diff`` shows local changes on all |
| repositories). |
| |
| Project arguments can be the names of projects in the manifest, or (as |
| fallback) paths to them. Omitting project arguments to commands which accept a |
| list of projects (such as ``west list``, ``west forall``, etc.) usually |
| defaults to using all projects in the manifest file plus the manifest |
| repository itself. |
| |
| For help on individual commands, run ``west <command> -h`` (e.g. ``west diff |
| -h``). |
| |
| Main Commands |
| ============= |
| |
| The ``west init`` and ``west update`` multi-repo commands are the most |
| important to understand. |
| |
| - ``west init [-l] [-m URL] [--mr REVISION] [PATH]``: create a west |
| workspace in directory :file:`PATH` (i.e. :file:`.west` etc. will be |
| created there). If the ``PATH`` argument is not given, the current working |
| directory is used. This command does not clone any of the projects in the |
| manifest; that is done the next time ``west update`` is run. |
| |
| This command can be invoked in two ways: |
| |
| 1. If you already have a local clone of the zephyr repository and want to |
| create a west workspace around it, you can use the ``-l`` switch to |
| pass its path to west, as in: ``west init -l path/to/zephyr``. This is |
| the only reason to use ``-l``. |
| |
| 2. Otherwise, omit ``-l`` to create a new workspace from a remote manifest |
| repository. You can give the manifest URL using the ``-m`` switch, and its |
| revision using ``--mr``. For example, invoking west with: ``west init -m |
| https://github.com/zephyrproject-rtos/zephyr --mr v1.15.0`` would clone |
| the upstream official zephyr repository at the tagged release v1.15.0 |
| (``-m`` defaults to https://github.com/zephyrproject-rtos/zephyr, and |
| the ``-mr`` default is overridden to ``v1.15.0``). |
| |
| - ``west update [--fetch {always,smart}] [--rebase] [--keep-descendants] |
| [PROJECT ...]``: clone and update the specified projects based |
| on the current :term:`west manifest`. |
| |
| By default, this command: |
| |
| #. Parses the manifest file, :file:`west.yml` |
| #. Clones any project repositories that are not already present locally |
| #. Fetches any project revisions in the manifest file which are not already |
| pulled from the remote |
| #. Sets each project's :ref:`manifest-rev <west-manifest-rev>` branch to the |
| current manifest revision |
| #. Checks out those revisions in local working trees |
| |
| To operate on a subset of projects only, specify them using the ``PROJECT`` |
| positional arguments, which can be either project names as given in the |
| manifest file, or paths to the local project clones. |
| |
| To force this command to fetch from project remotes even if the revisions |
| appear to be available locally, either use ``--fetch always`` or set the |
| ``update.fetch`` :ref:`configuration option <west-config>` to ``"always"``. |
| |
| For safety, ``west update`` uses ``git checkout --detach`` to check out a |
| detached ``HEAD`` at the manifest revision for each updated project, leaving |
| behind any branches which were already checked out. This is typically a safe |
| operation that will not modify any of your local branches. See the help for |
| the ``--rebase`` / ``-r`` and ``--keep-descendants`` / ``-k`` options for |
| ways to influence this. |
| |
| .. _west-multi-repo-misc: |
| |
| Miscellaneous Commands |
| ====================== |
| |
| West has a few more commands for managing the multi-repo, which are briefly |
| discussed here. Run ``west <command> -h`` for detailed help. |
| |
| - ``west forall -c COMMAND [PROJECT ...]``: Runs the shell command ``COMMAND`` |
| within the top-level repository directory of each of the specified projects |
| (default: all cloned projects). If ``COMMAND`` consists of more than one |
| word, it must be quoted to prevent it from being split up by the shell. |
| |
| To run an arbitrary Git command in each project, use something like ``west |
| forall -c 'git <command> --options'``. Note that ``west forall`` can be used |
| to run any command, though, not just Git commands. |
| |
| - ``west help <command>``: this is equivalent to ``west <command> -h``. |
| |
| - ``west list [-f FORMAT] [PROJECT ...]``: Lists project information from the |
| manifest file, such as URL, revision, path, etc. The printed information can |
| be controlled using the ``-f`` option. |
| |
| - ``west manifest``: Manipulates manifest files. See :ref:`west-manifest-cmd`. |
| |
| - ``west manifest --validate``: Ensure the current manifest file is |
| well-formed. Print information about what's wrong and fail the process in |
| case of error. |
| |
| - ``west diff [PROJECT ...]``: Runs a multi-repo ``git diff`` |
| for the specified projects. |
| |
| - ``west status [PROJECT ...]``: Like ``west diff``, for |
| running ``git status``. |
| |
| - ``west topdir``: Prints the top directory of the west workspace. |
| |
| Private repositories |
| ******************** |
| |
| You can use west to fetch from private repositories. There is nothing |
| west-specific about this. |
| |
| The ``west update`` command essentially runs ``git fetch YOUR_PROJECT_URL`` |
| when a project's ``manifest-rev`` branch must be updated to a newly fetched |
| commit. It's up to your environment to make sure the fetch succeeds. |
| |
| You can either enter the password manually or use any of the `credential |
| helpers built in to Git`_. Since Git has credential storage built in, there is |
| no need for a west-specific feature. |
| |
| The following sections cover common cases for running ``west update`` without |
| having to enter your password, as well as how to troubleshoot issues. |
| |
| .. _credential helpers built in to Git: |
| https://git-scm.com/docs/gitcredentials |
| |
| Fetching via HTTPS |
| ================== |
| |
| On Windows when fetching from GitHub, recent versions of Git prompt you for |
| your GitHub password in a graphical window once, then store it for future use |
| (in a default installation). Passwordless fetching from GitHub should therefore |
| work "out of the box" on Windows after you have done it once. |
| |
| In general, you can store your credentials on disk using the "store" git |
| credential helper. See the `git-credential-store`_ manual page for details. |
| |
| To use this helper for all the repositories in your workspace, run: |
| |
| .. code-block:: shell |
| |
| west forall -c "git config credential.helper store" |
| |
| To use this helper on just the projects ``foo`` and ``bar``, run: |
| |
| .. code-block:: shell |
| |
| west forall -c "git config credential.helper store" foo bar |
| |
| To use this helper by default on your computer, run: |
| |
| .. code-block:: shell |
| |
| git config --global credential.helper store |
| |
| On GitHub, you can set up a `personal access token`_ to use in place of your |
| account password. (This may be required if your account has two-factor |
| authentication enabled, and may be preferable to storing your account password |
| in plain text even if two-factor authentication is disabed.) |
| |
| If you don't want to store any credentials on the file system, you can store |
| them in memory temporarily using `git-credential-cache`_ instead. |
| |
| .. _git-credential-store: |
| https://git-scm.com/docs/git-credential-store#_examples |
| .. _git-credential-cache: |
| https://git-scm.com/docs/git-credential-cache |
| .. _personal access token: |
| https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token |
| |
| Fetching via SSH |
| ================ |
| |
| If your SSH key has no password, fetching should just work. If it does have a |
| password, you can avoid entering it manually every time using `ssh-agent`_. |
| |
| On GitHub, see `Connecting to GitHub with SSH`_ for details on configuration |
| and key creation. |
| |
| .. _ssh-agent: |
| https://www.ssh.com/ssh/agent |
| .. _Connecting to GitHub with SSH: |
| https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh |
| |
| Troubleshooting |
| =============== |
| |
| One good way to troubleshoot fetching issues is to run ``west update`` in |
| verbose mode, like this: |
| |
| .. code-block:: shell |
| |
| west -v update |
| |
| The output includes Git commands run by west and their outputs. Look for |
| something like this: |
| |
| .. code-block:: none |
| |
| === updating your_project (path/to/your/project): |
| west.manifest: your_project: checking if cloned |
| [...other west.manifest logs...] |
| --- your_project: fetching, need revision SOME_SHA |
| west.manifest: running 'git fetch ... https://github.com/your-username/your_project ...' in /some/directory |
| |
| The ``git fetch`` command example in the last line above is what needs to |
| succeed. Go to ``/some/directory``, copy/paste and run the entire ``git fetch`` |
| command, then debug from there using the documentation for your credential |
| storage helper. |
| |
| If you can get the ``git fetch`` command to run successfully without prompting |
| for a password when you run it directly, you will be able to run ``west |
| update`` without entering your password in that same shell. |
| |
| .. _PyPI: |
| https://pypi.org/project/west/ |
| |
| .. _Zephyr issue #6770: |
| https://github.com/zephyrproject-rtos/zephyr/issues/6770 |
| |
| .. _namespace package: |
| https://www.python.org/dev/peps/pep-0420/ |