:::{default-domain} bzl :::
:::{seealso} For WORKSPACE instructions see here. :::
To add PyPI dependencies to your MODULE.bazel file, use the pip.parse extension and call it to create the central external repo and individual wheel external repos. Include the toolchain extension in the MODULE.bazel file as shown in the first bzlmod example above.
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip") pip.parse( hub_name = "my_deps", python_version = "3.13", requirements_lock = "//:requirements_lock_3_11.txt", ) use_repo(pip, "my_deps")
For more documentation, see the Bzlmod examples under the {gh-path}examples folder or the documentation for the {obj}@rules_python//python/extensions:pip.bzl extension.
:::note} We are using a host-platform compatible toolchain by default to setup pip dependencies. During the setup phase, we create some symlinks, which may be inefficient on Windows by default. In that case use the following .bazelrc options to improve performance if you have admin privileges:
startup --windows_enable_symlinks
This will enable symlinks on Windows and help with bootstrap performance of setting up the hermetic host python interpreter on this platform. Linux and OSX users should see no difference. :::
The {obj}pip.parse bzlmod extension by default uses the hermetic Python toolchain for the host platform, but you can customize the interpreter using {attr}pip.parse.python_interpreter and {attr}pip.parse.python_interpreter_target.
You can use the pip extension multiple times. This configuration will create multiple external repos that have no relation to one another and may result in downloading the same wheels numerous times.
As with any repository rule or extension, if you would like to ensure that pip_parse is re-executed to pick up a non-hermetic change to your environment (e.g., updating your system python interpreter), you can force it to re-execute by running bazel sync --only [pip_parse name].
(per-os-arch-requirements)=
In some cases, you may need to use different requirements files for different OS and architecture combinations. This is enabled via the requirements_by_platform attribute in the pip.parse extension and the {obj}pip.parse tag class. The keys of the dictionary are labels to the file, and the values are a list of comma-separated target (os, arch) tuples.
For example:
# ... requirements_by_platform = { "requirements_linux_x86_64.txt": "linux_x86_64", "requirements_osx.txt": "osx_*", "requirements_linux_exotic.txt": "linux_exotic", "requirements_some_platforms.txt": "linux_aarch64,windows_*", }, # For the list of standard platforms that the rules_python has toolchains for, default to # the following requirements file. requirements_lock = "requirements_lock.txt",
In case of duplicate platforms, rules_python will raise an error, as there has to be an unambiguous mapping of the requirement files to the (os, arch) tuples.
An alternative way is to use per-OS requirement attributes.
# ... requirements_windows = "requirements_windows.txt", requirements_darwin = "requirements_darwin.txt", # For the remaining platforms (which is basically only linux OS), use this file. requirements_lock = "requirements_lock.txt", )
:::{note} If you are using a universal lock file but want to restrict the list of platforms that the lock file will be evaluated against, consider using the aforementioned requirements_by_platform attribute and listing the platforms explicitly. :::
Historically, the {obj}pip_parse and {obj}pip.parse have only been downloading/building Python dependencies for the host platform that the bazel commands are executed on. Over the years, people started needing support for building containers, and usually, that involves fetching dependencies for a particular target platform that may be different from the host platform.
Multi-platform support for cross-building the wheels can be done in two ways:
experimental_index_url for the {bzl:obj}pip.parse bzlmod tag classpip.parse.download_only setting.:::{warning} This will not work for sdists with C extensions, but pure Python sdists may still work using the first approach. :::
download_only attributeLet's say you have two requirements files:
# requirements.linux_x86_64.txt --platform=manylinux_2_17_x86_64 --python-version=39 --implementation=cp --abi=cp39 foo==0.0.1 --hash=sha256:deadbeef bar==0.0.1 --hash=sha256:deadb00f
# requirements.osx_aarch64.txt contents --platform=macosx_10_9_arm64 --python-version=39 --implementation=cp --abi=cp39 foo==0.0.3 --hash=sha256:deadbaaf
With these 2 files your {bzl:obj}pip.parse could look like:
pip.parse( hub_name = "pip", python_version = "3.9", # Tell `pip` to ignore sdists download_only = True, requirements_by_platform = { "requirements.linux_x86_64.txt": "linux_x86_64", "requirements.osx_aarch64.txt": "osx_aarch64", }, )
With this, pip.parse will create a hub repository that is going to support only two platforms - cp39_osx_aarch64 and cp39_linux_x86_64 - and it will only use wheels and ignore any sdists that it may find on the PyPI- compatible indexes.
:::{warning} Because bazel is not aware what exactly is downloaded, the same wheel may be downloaded multiple times. :::
:::{note} This will only work for wheel-only setups, i.e., all of your dependencies need to have wheels available on the PyPI index that you use. :::
Requires-Dist resolution:::{note} Currently this is disabled by default, but you can turn it on using {envvar}RULES_PYTHON_ENABLE_PIPSTAR environment variable. :::
In order to understand what dependencies to pull for a particular package, rules_python parses the whl file METADATA. Packages can express dependencies via Requires-Dist, and they can add conditions using “environment markers”, which represent the Python version, OS, etc.
While the PyPI integration provides reasonable defaults to support most platforms and environment markers, the values it uses can be customized in case more esoteric configurations are needed.
To customize the values used, you need to do two things:
EnvMarkerInfo//python/config_settings:pip_env_marker_config flag to the target defined in (1).The keys and values should be compatible with the PyPA dependency specifiers specification. This is not strictly enforced, however, so you can return a subset of keys or additional keys, which become available during dependency evaluation.
(bazel-downloader)=
:::{warning} This is currently still experimental, and whilst it has been proven to work in quite a few environments, the APIs are still being finalized, and there may be changes to the APIs for this feature without much notice.
The issues that you can subscribe to for updates are:
2601357 :::The {obj}pip extension supports pulling information from PyPI (or a compatible mirror), and it will ensure that the bazel downloader is used for downloading the wheels.
This provides the following benefits:
pip for wheel fetching and results in much faster dependency fetching.To enable the feature specify {attr}pip.parse.experimental_index_url as shown in the {gh-path}examples/bzlmod/MODULE.bazel example.
Similar to uv, one can override the index that is used for a single package. By default, we first search in the index specified by {attr}pip.parse.experimental_index_url, then we iterate through the {attr}pip.parse.experimental_extra_index_urls unless there are overrides specified via {attr}pip.parse.experimental_index_url_overrides.
When using this feature during the pip extension evaluation you will see the accessed indexes similar to below:
Loading: 0 packages loaded Fetching module extension @@//python/extensions:pip.bzl%pip; Fetch package lists from PyPI index Fetching https://pypi.org/simple/jinja2/
This does not mean that rules_python is fetching the wheels eagerly; rather, it means that it is calling the PyPI server to get the Simple API response to get the list of all available source and wheel distributions. Once it has gotten all of the available distributions, it will select the right ones depending on the sha256 values in your requirements_lock.txt file. If sha256 hashes are not present in the requirements file, we will fall back to matching by version specified in the lock file.
Fetching the distribution information from the PyPI allows rules_python to know which whl should be used on which target platform and it will determine that by parsing the whl filename based on PEP600, PEP656 standards. This allows the user to configure the behaviour by using the following publicly available flags:
--@rules_python//python/config_settings:py_linux_libc for selecting the Linux libc variant.--@rules_python//python/config_settings:pip_whl for selecting whl distribution preference.--@rules_python//python/config_settings:pip_whl_osx_arch for selecting MacOS wheel preference.--@rules_python//python/config_settings:pip_whl_glibc_version for selecting the GLIBC version compatibility.--@rules_python//python/config_settings:pip_whl_muslc_version for selecting the musl version compatibility.--@rules_python//python/config_settings:pip_whl_osx_version for selecting MacOS version compatibility.(credential-helper)=
The Bazel downloader usage allows for the Bazel Credential Helper. Your Python artifact registry may provide a credential helper for you. Refer to your index's docs to see if one is provided.
The simplest form of a credential helper is a bash script that accepts an argument and spits out JSON to stdout. For a service like Google Artifact Registry that uses ‘Basic’ HTTP Auth and does not provide a credential helper that conforms to the spec, the script might look like:
#!/bin/bash # cred_helper.sh ARG=$1 # but we don't do anything with it as it's always "get" # formatting is optional echo '{' echo ' "headers": {' echo ' "Authorization": ["Basic dGVzdDoxMjPCow=="]' echo ' }' echo '}'
Configure Bazel to use this credential helper for your Python index example.com:
# .bazelrc build --credential_helper=example.com=/full/path/to/cred_helper.sh
Bazel will call this file like cred_helper.sh get and use the returned JSON to inject headers into whatever HTTP(S) request it performs against example.com.
See the Credential Helper Spec for more details.