docs: split PyPI docs up and add more (#2935)
Summary:
- Split the PyPI docs per topic.
- Move everything to its own folder.
- Separate the `bzlmod` and `WORKSPACE` documentation. Some of the
features are only available in `bzlmod` and since `bzlmod` is the
future having that as the default makes things a little easier.
- Fix a few warnings.
Fixes #2810.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index b087119..324801c 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -68,7 +68,7 @@
## Developer guide
For more more details, guidance, and tips for working with the code base,
-see [DEVELOPING.md](DEVELOPING.md)
+see [docs/devguide.md](./devguide)
## Formatting
diff --git a/MODULE.bazel b/MODULE.bazel
index fa24ed0..d3a9535 100644
--- a/MODULE.bazel
+++ b/MODULE.bazel
@@ -134,6 +134,7 @@
download_only = True,
experimental_index_url = "https://pypi.org/simple",
hub_name = "dev_pip",
+ parallel_download = False,
python_version = "3.11",
requirements_lock = "//docs:requirements.txt",
)
diff --git a/docs/BUILD.bazel b/docs/BUILD.bazel
index b3e5f52..852c4d4 100644
--- a/docs/BUILD.bazel
+++ b/docs/BUILD.bazel
@@ -120,7 +120,10 @@
"//python/private:rule_builders_bzl",
"//python/private/api:py_common_api_bzl",
"//python/private/pypi:config_settings_bzl",
+ "//python/private/pypi:env_marker_info_bzl",
"//python/private/pypi:pkg_aliases_bzl",
+ "//python/private/pypi:whl_config_setting_bzl",
+ "//python/private/pypi:whl_library_bzl",
"//python/uv:lock_bzl",
"//python/uv:uv_bzl",
"//python/uv:uv_toolchain_bzl",
diff --git a/docs/conf.py b/docs/conf.py
index 96bbdb5..1d9f526 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -91,6 +91,8 @@
"api/sphinxdocs/private/sphinx_docs_library": "/api/sphinxdocs/sphinxdocs/private/sphinx_docs_library.html",
"api/sphinxdocs/sphinx_docs_library": "/api/sphinxdocs/sphinxdocs/sphinx_docs_library.html",
"api/sphinxdocs/inventories/index": "/api/sphinxdocs/sphinxdocs/inventories/index.html",
+ "pip.html": "pypi/index.html",
+ "pypi-dependencies.html": "pypi/index.html",
}
# Adapted from the template code:
@@ -139,7 +141,9 @@
# --- Extlinks configuration
extlinks = {
+ "gh-issue": (f"https://github.com/bazel-contrib/rules_python/issues/%s", "#%s issue"),
"gh-path": (f"https://github.com/bazel-contrib/rules_python/tree/main/%s", "%s"),
+ "gh-pr": (f"https://github.com/bazel-contrib/rules_python/pulls/%s", "#%s PR"),
}
# --- MyST configuration
diff --git a/docs/getting-started.md b/docs/getting-started.md
index 60d5d5e..7e7b88a 100644
--- a/docs/getting-started.md
+++ b/docs/getting-started.md
@@ -8,13 +8,13 @@
For more details information about configuring `rules_python`, see:
* [Configuring the runtime](configuring-toolchains)
-* [Configuring third party dependencies (pip/pypi)](pypi-dependencies)
+* [Configuring third party dependencies (pip/pypi)](./pypi/index)
* [API docs](api/index)
-## Using bzlmod
+## Including dependencies
-The first step to using rules_python with bzlmod is to add the dependency to
-your MODULE.bazel file:
+The first step to using `rules_python` is to add the dependency to
+your `MODULE.bazel` file:
```starlark
# Update the version "0.0.0" to the release found here:
@@ -30,7 +30,7 @@
use_repo(pip, "pypi")
```
-## Using a WORKSPACE file
+### Using a WORKSPACE file
Using WORKSPACE is deprecated, but still supported, and a bit more involved than
using Bzlmod. Here is a simplified setup to download the prebuilt runtimes.
diff --git a/docs/index.md b/docs/index.md
index 4983a6a..82023f3 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -95,9 +95,8 @@
:hidden:
self
getting-started
-pypi-dependencies
+pypi/index
Toolchains <toolchains>
-pip
coverage
precompiling
gazelle
diff --git a/docs/pip.md b/docs/pip.md
deleted file mode 100644
index 43d8fc4..0000000
--- a/docs/pip.md
+++ /dev/null
@@ -1,4 +0,0 @@
-(pip-integration)=
-# Pip Integration
-
-See [PyPI dependencies](./pypi-dependencies).
diff --git a/docs/pypi-dependencies.md b/docs/pypi-dependencies.md
deleted file mode 100644
index b3ae7fe..0000000
--- a/docs/pypi-dependencies.md
+++ /dev/null
@@ -1,519 +0,0 @@
-:::{default-domain} bzl
-:::
-
-# Using dependencies from PyPI
-
-Using PyPI packages (aka "pip install") involves two main steps.
-
-1. [Generating requirements file](#generating-requirements-file)
-2. [Installing third party packages](#installing-third-party-packages)
-3. [Using third party packages as dependencies](#using-third-party-packages)
-
-{#generating-requirements-file}
-## Generating requirements file
-
-Generally, when working on a Python project, you'll have some dependencies that themselves have other dependencies. You might also specify dependency bounds instead of specific versions. So you'll need to generate a full list of all transitive dependencies and pinned versions for every dependency.
-
-Typically, you'd have your dependencies specified in `pyproject.toml` or `requirements.in` and generate the full pinned list of dependencies in `requirements_lock.txt`, which you can manage with the `compile_pip_requirements` Bazel rule:
-
-```starlark
-load("@rules_python//python:pip.bzl", "compile_pip_requirements")
-
-compile_pip_requirements(
- name = "requirements",
- src = "requirements.in",
- requirements_txt = "requirements_lock.txt",
-)
-```
-
-This rule generates two targets:
-- `bazel run [name].update` will regenerate the `requirements_txt` file
-- `bazel test [name]_test` will test that the `requirements_txt` file is up to date
-
-For more documentation, see the API docs under {obj}`@rules_python//python:pip.bzl`.
-
-Once you generate this fully specified list of requirements, you can install the requirements with the instructions in [Installing third party packages](#installing-third-party-packages).
-
-:::{warning}
-If you're specifying dependencies in `pyproject.toml`, make sure to include the `[build-system]` configuration, with pinned dependencies. `compile_pip_requirements` will use the build system specified to read your project's metadata, and you might see non-hermetic behavior if you don't pin the build system.
-
-Not specifying `[build-system]` at all will result in using a default `[build-system]` configuration, which uses unpinned versions ([ref](https://peps.python.org/pep-0518/#build-system-table)).
-:::
-
-{#installing-third-party-packages}
-## Installing third party packages
-
-### Using bzlmod
-
-To add pip dependencies to your `MODULE.bazel` file, use the `pip.parse`
-extension, and call it to create the central external repo and individual wheel
-external repos. Include in the `MODULE.bazel` the toolchain extension as shown
-in the first bzlmod example above.
-
-```starlark
-pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
-pip.parse(
- hub_name = "my_deps",
- python_version = "3.11",
- requirements_lock = "//:requirements_lock_3_11.txt",
-)
-use_repo(pip, "my_deps")
-```
-For more documentation, see the bzlmod examples under the {gh-path}`examples` folder or the documentation
-for the {obj}`@rules_python//python/extensions:pip.bzl` extension.
-
-```{note}
-We are using a host-platform compatible toolchain by default to setup pip dependencies.
-During the setup phase, we create some symlinks, which may be inefficient on Windows
-by default. In that case use the following `.bazelrc` options to improve performance if
-you have admin privileges:
-
- startup --windows_enable_symlinks
-
-This will enable symlinks on Windows and help with bootstrap performance of setting up the
-hermetic host python interpreter on this platform. Linux and OSX users should see no
-difference.
-```
-
-### Using a WORKSPACE file
-
-To add pip dependencies to your `WORKSPACE`, load the `pip_parse` function and
-call it to create the central external repo and individual wheel external repos.
-
-```starlark
-load("@rules_python//python:pip.bzl", "pip_parse")
-
-# Create a central repo that knows about the dependencies needed from
-# requirements_lock.txt.
-pip_parse(
- name = "my_deps",
- requirements_lock = "//path/to:requirements_lock.txt",
-)
-# Load the starlark macro, which will define your dependencies.
-load("@my_deps//:requirements.bzl", "install_deps")
-# Call it to define repos for your requirements.
-install_deps()
-```
-
-(vendoring-requirements)=
-#### Vendoring the requirements.bzl file
-
-In some cases you may not want to generate the requirements.bzl file as a repository rule
-while Bazel is fetching dependencies. For example, if you produce a reusable Bazel module
-such as a ruleset, you may want to include the requirements.bzl file rather than make your users
-install the WORKSPACE setup to generate it.
-See https://github.com/bazel-contrib/rules_python/issues/608
-
-This is the same workflow as Gazelle, which creates `go_repository` rules with
-[`update-repos`](https://github.com/bazelbuild/bazel-gazelle#update-repos)
-
-To do this, use the "write to source file" pattern documented in
-https://blog.aspect.dev/bazel-can-write-to-the-source-folder
-to put a copy of the generated requirements.bzl into your project.
-Then load the requirements.bzl file directly rather than from the generated repository.
-See the example in rules_python/examples/pip_parse_vendored.
-
-(per-os-arch-requirements)=
-### Requirements for a specific OS/Architecture
-
-In some cases you may need to use different requirements files for different OS, Arch combinations. This is enabled via the `requirements_by_platform` attribute in `pip.parse` extension and the `pip_parse` repository rule. The keys of the dictionary are labels to the file and the values are a list of comma separated target (os, arch) tuples.
-
-For example:
-```starlark
- # ...
- requirements_by_platform = {
- "requirements_linux_x86_64.txt": "linux_x86_64",
- "requirements_osx.txt": "osx_*",
- "requirements_linux_exotic.txt": "linux_exotic",
- "requirements_some_platforms.txt": "linux_aarch64,windows_*",
- },
- # For the list of standard platforms that the rules_python has toolchains for, default to
- # the following requirements file.
- requirements_lock = "requirements_lock.txt",
-```
-
-In case of duplicate platforms, `rules_python` will raise an error as there has
-to be unambiguous mapping of the requirement files to the (os, arch) tuples.
-
-An alternative way is to use per-OS requirement attributes.
-```starlark
- # ...
- requirements_windows = "requirements_windows.txt",
- requirements_darwin = "requirements_darwin.txt",
- # For the remaining platforms (which is basically only linux OS), use this file.
- requirements_lock = "requirements_lock.txt",
-)
-```
-
-### pip rules
-
-Note that since `pip_parse` and `pip.parse` are executed at evaluation time,
-Bazel has no information about the Python toolchain and cannot enforce that the
-interpreter used to invoke `pip` matches the interpreter used to run
-`py_binary` targets. By default, `pip_parse` uses the system command
-`"python3"`. To override this, pass in the `python_interpreter` attribute or
-`python_interpreter_target` attribute to `pip_parse`. The `pip.parse` `bzlmod` extension
-by default uses the hermetic python toolchain for the host platform.
-
-You can have multiple `pip_parse`s in the same workspace, or use the pip
-extension multiple times when using bzlmod. This configuration will create
-multiple external repos that have no relation to one another and may result in
-downloading the same wheels numerous times.
-
-As with any repository rule, if you would like to ensure that `pip_parse` is
-re-executed to pick up a non-hermetic change to your environment (e.g., updating
-your system `python` interpreter), you can force it to re-execute by running
-`bazel sync --only [pip_parse name]`.
-
-{#using-third-party-packages}
-## Using third party packages as dependencies
-
-Each extracted wheel repo contains a `py_library` target representing
-the wheel's contents. There are two ways to access this library. The
-first uses the `requirement()` function defined in the central
-repo's `//:requirements.bzl` file. This function maps a pip package
-name to a label:
-
-```starlark
-load("@my_deps//:requirements.bzl", "requirement")
-
-py_library(
- name = "mylib",
- srcs = ["mylib.py"],
- deps = [
- ":myotherlib",
- requirement("some_pip_dep"),
- requirement("another_pip_dep"),
- ]
-)
-```
-
-The reason `requirement()` exists is to insulate from
-changes to the underlying repository and label strings. However, those
-labels have become directly used, so aren't able to easily change regardless.
-
-On the other hand, using `requirement()` has several drawbacks; see
-[this issue][requirements-drawbacks] for an enumeration. If you don't
-want to use `requirement()`, you can use the library
-labels directly instead. For `pip_parse`, the labels are of the following form:
-
-```starlark
-@{name}//{package}
-```
-
-Here `name` is the `name` attribute that was passed to `pip_parse` and
-`package` is the pip package name with characters that are illegal in
-Bazel label names (e.g. `-`, `.`) replaced with `_`. If you need to
-update `name` from "old" to "new", then you can run the following
-buildozer command:
-
-```shell
-buildozer 'substitute deps @old//([^/]+) @new//${1}' //...:*
-```
-
-[requirements-drawbacks]: https://github.com/bazel-contrib/rules_python/issues/414
-
-### Entry points
-
-If you would like to access [entry points][whl_ep], see the `py_console_script_binary` rule documentation,
-which can help you create a `py_binary` target for a particular console script exposed by a package.
-
-[whl_ep]: https://packaging.python.org/specifications/entry-points/
-
-### 'Extras' dependencies
-
-Any 'extras' specified in the requirements lock file will be automatically added
-as transitive dependencies of the package. In the example above, you'd just put
-`requirement("useful_dep")` or `@pypi//useful_dep`.
-
-### Consuming Wheel Dists Directly
-
-If you need to depend on the wheel dists themselves, for instance, to pass them
-to some other packaging tool, you can get a handle to them with the
-`whl_requirement` macro. For example:
-
-```starlark
-load("@pypi//:requirements.bzl", "whl_requirement")
-
-filegroup(
- name = "whl_files",
- data = [
- # This is equivalent to "@pypi//boto3:whl"
- whl_requirement("boto3"),
- ]
-)
-```
-
-### Creating a filegroup of files within a whl
-
-The rule {obj}`whl_filegroup` exists as an easy way to extract the necessary files
-from a whl file without the need to modify the `BUILD.bazel` contents of the
-whl repositories generated via `pip_repository`. Use it similarly to the `filegroup`
-above. See the API docs for more information.
-
-(advance-topics)=
-## Advanced topics
-
-(circular-deps)=
-### Circular dependencies
-
-Sometimes PyPi packages contain dependency cycles -- for instance a particular
-version `sphinx` (this is no longer the case in the latest version as of
-2024-06-02) depends on `sphinxcontrib-serializinghtml`. When using them as
-`requirement()`s, ala
-
-```
-py_binary(
- name = "doctool",
- ...
- deps = [
- requirement("sphinx"),
- ],
-)
-```
-
-Bazel will protest because it doesn't support cycles in the build graph --
-
-```
-ERROR: .../external/pypi_sphinxcontrib_serializinghtml/BUILD.bazel:44:6: in alias rule @pypi_sphinxcontrib_serializinghtml//:pkg: cycle in dependency graph:
- //:doctool (...)
- @pypi//sphinxcontrib_serializinghtml:pkg (...)
-.-> @pypi_sphinxcontrib_serializinghtml//:pkg (...)
-| @pypi_sphinxcontrib_serializinghtml//:_pkg (...)
-| @pypi_sphinx//:pkg (...)
-| @pypi_sphinx//:_pkg (...)
-`-- @pypi_sphinxcontrib_serializinghtml//:pkg (...)
-```
-
-The `experimental_requirement_cycles` argument allows you to work around these
-issues by specifying groups of packages which form cycles. `pip_parse` will
-transparently fix the cycles for you and provide the cyclic dependencies
-simultaneously.
-
-```starlark
-pip_parse(
- ...
- experimental_requirement_cycles = {
- "sphinx": [
- "sphinx",
- "sphinxcontrib-serializinghtml",
- ]
- },
-)
-```
-
-`pip_parse` supports fixing multiple cycles simultaneously, however cycles must
-be distinct. `apache-airflow` for instance has dependency cycles with a number
-of its optional dependencies, which means those optional dependencies must all
-be a part of the `airflow` cycle. For instance --
-
-```starlark
-pip_parse(
- ...
- experimental_requirement_cycles = {
- "airflow": [
- "apache-airflow",
- "apache-airflow-providers-common-sql",
- "apache-airflow-providers-postgres",
- "apache-airflow-providers-sqlite",
- ]
- }
-)
-```
-
-Alternatively, one could resolve the cycle by removing one leg of it.
-
-For example while `apache-airflow-providers-sqlite` is "baked into" the Airflow
-package, `apache-airflow-providers-postgres` is not and is an optional feature.
-Rather than listing `apache-airflow[postgres]` in your `requirements.txt` which
-would expose a cycle via the extra, one could either _manually_ depend on
-`apache-airflow` and `apache-airflow-providers-postgres` separately as
-requirements. Bazel rules which need only `apache-airflow` can take it as a
-dependency, and rules which explicitly want to mix in
-`apache-airflow-providers-postgres` now can.
-
-Alternatively, one could use `rules_python`'s patching features to remove one
-leg of the dependency manually. For instance by making
-`apache-airflow-providers-postgres` not explicitly depend on `apache-airflow` or
-perhaps `apache-airflow-providers-common-sql`.
-
-
-### Multi-platform support
-
-Multi-platform support of cross-building the wheels can be done in two ways - either
-using {bzl:attr}`experimental_index_url` for the {bzl:obj}`pip.parse` bzlmod tag class
-or by using the {bzl:attr}`pip.parse.download_only` setting. In this section we
-are going to outline quickly how one can use the latter option.
-
-Let's say you have 2 requirements files:
-```
-# requirements.linux_x86_64.txt
---platform=manylinux_2_17_x86_64
---python-version=39
---implementation=cp
---abi=cp39
-
-foo==0.0.1 --hash=sha256:deadbeef
-bar==0.0.1 --hash=sha256:deadb00f
-```
-
-```
-# requirements.osx_aarch64.txt contents
---platform=macosx_10_9_arm64
---python-version=39
---implementation=cp
---abi=cp39
-
-foo==0.0.3 --hash=sha256:deadbaaf
-```
-
-With these 2 files your {bzl:obj}`pip.parse` could look like:
-```
-pip.parse(
- hub_name = "pip",
- python_version = "3.9",
- # Tell `pip` to ignore sdists
- download_only = True,
- requirements_by_platform = {
- "requirements.linux_x86_64.txt": "linux_x86_64",
- "requirements.osx_aarch64.txt": "osx_aarch64",
- },
-)
-```
-
-With this, the `pip.parse` will create a hub repository that is going to
-support only two platforms - `cp39_osx_aarch64` and `cp39_linux_x86_64` and it
-will only use `wheels` and ignore any sdists that it may find on the PyPI
-compatible indexes.
-
-```{note}
-This is only supported on `bzlmd`.
-```
-
-<!--
-
-TODO: uncomment this when analysis-phase dependency selection is available
-
-#### Customizing requirements resolution
-
-In Python packaging, packages can express dependencies with conditions
-using "environment markers", which represent the Python version, OS, etc.
-
-While the PyPI integration provides reasonable defaults to support most
-platforms and environment markers, the values it uses can be customized in case
-more esoteric configurations are needed.
-
-To customize the values used, you need to do two things:
-1. Define a target that returns {obj}`EnvMarkerInfo`
-2. Set the {obj}`//python/config_settings:pip_env_marker_config` flag to
- the target defined in (1).
-
-The keys and values should be compatible with the [PyPA dependency specifiers
-specification](https://packaging.python.org/en/latest/specifications/dependency-specifiers/).
-This is not strictly enforced, however, so you can return a subset of keys or
-additional keys, which become available during dependency evalution.
-
--->
-
-(bazel-downloader)=
-### Bazel downloader and multi-platform wheel hub repository.
-
-The `bzlmod` `pip.parse` call supports pulling information from `PyPI` (or a
-compatible mirror) and it will ensure that the [bazel
-downloader][bazel_downloader] is used for downloading the wheels. This allows
-the users to use the [credential helper](#credential-helper) to authenticate
-with the mirror and it also ensures that the distribution downloads are cached.
-It also avoids using `pip` altogether and results in much faster dependency
-fetching.
-
-This can be enabled by `experimental_index_url` and related flags as shown in
-the {gh-path}`examples/bzlmod/MODULE.bazel` example.
-
-When using this feature during the `pip` extension evaluation you will see the accessed indexes similar to below:
-```console
-Loading: 0 packages loaded
- currently loading: docs/
- Fetching module extension pip in @@//python/extensions:pip.bzl; starting
- Fetching https://pypi.org/simple/twine/
-```
-
-This does not mean that `rules_python` is fetching the wheels eagerly, but it
-rather means that it is calling the PyPI server to get the Simple API response
-to get the list of all available source and wheel distributions. Once it has
-got all of the available distributions, it will select the right ones depending
-on the `sha256` values in your `requirements_lock.txt` file. If `sha256` hashes
-are not present in the requirements file, we will fallback to matching by version
-specified in the lock file. The compatible distribution URLs will be then
-written to the `MODULE.bazel.lock` file. Currently users wishing to use the
-lock file with `rules_python` with this feature have to set an environment
-variable `RULES_PYTHON_OS_ARCH_LOCK_FILE=0` which will become default in the
-next release.
-
-Fetching the distribution information from the PyPI allows `rules_python` to
-know which `whl` should be used on which target platform and it will determine
-that by parsing the `whl` filename based on [PEP600], [PEP656] standards. This
-allows the user to configure the behaviour by using the following publicly
-available flags:
-* {obj}`--@rules_python//python/config_settings:py_linux_libc` for selecting the Linux libc variant.
-* {obj}`--@rules_python//python/config_settings:pip_whl` for selecting `whl` distribution preference.
-* {obj}`--@rules_python//python/config_settings:pip_whl_osx_arch` for selecting MacOS wheel preference.
-* {obj}`--@rules_python//python/config_settings:pip_whl_glibc_version` for selecting the GLIBC version compatibility.
-* {obj}`--@rules_python//python/config_settings:pip_whl_muslc_version` for selecting the musl version compatibility.
-* {obj}`--@rules_python//python/config_settings:pip_whl_osx_version` for selecting MacOS version compatibility.
-
-[bazel_downloader]: https://bazel.build/rules/lib/builtins/repository_ctx#download
-[pep600]: https://peps.python.org/pep-0600/
-[pep656]: https://peps.python.org/pep-0656/
-
-(credential-helper)=
-### Credential Helper
-
-The "use Bazel downloader for python wheels" experimental feature includes support for the Bazel
-[Credential Helper][cred-helper-design].
-
-Your python artifact registry may provide a credential helper for you. Refer to your index's docs
-to see if one is provided.
-
-See the [Credential Helper Spec][cred-helper-spec] for details.
-
-[cred-helper-design]: https://github.com/bazelbuild/proposals/blob/main/designs/2022-06-07-bazel-credential-helpers.md
-[cred-helper-spec]: https://github.com/EngFlow/credential-helper-spec/blob/main/spec.md
-
-
-#### Basic Example:
-
-The simplest form of a credential helper is a bash script that accepts an arg and spits out JSON to
-stdout. For a service like Google Artifact Registry that uses ['Basic' HTTP Auth][rfc7617] and does
-not provide a credential helper that conforms to the [spec][cred-helper-spec], the script might
-look like:
-
-```bash
-#!/bin/bash
-# cred_helper.sh
-ARG=$1 # but we don't do anything with it as it's always "get"
-
-# formatting is optional
-echo '{'
-echo ' "headers": {'
-echo ' "Authorization": ["Basic dGVzdDoxMjPCow=="]'
-echo ' }'
-echo '}'
-```
-
-Configure Bazel to use this credential helper for your python index `example.com`:
-
-```
-# .bazelrc
-build --credential_helper=example.com=/full/path/to/cred_helper.sh
-```
-
-Bazel will call this file like `cred_helper.sh get` and use the returned JSON to inject headers
-into whatever HTTP(S) request it performs against `example.com`.
-
-[rfc7617]: https://datatracker.ietf.org/doc/html/rfc7617
-
-<!--
-
-
-
--->
diff --git a/docs/pypi/circular-dependencies.md b/docs/pypi/circular-dependencies.md
new file mode 100644
index 0000000..d22f5b3
--- /dev/null
+++ b/docs/pypi/circular-dependencies.md
@@ -0,0 +1,82 @@
+:::{default-domain} bzl
+:::
+
+# Circular dependencies
+
+Sometimes PyPi packages contain dependency cycles -- for instance a particular
+version `sphinx` (this is no longer the case in the latest version as of
+2024-06-02) depends on `sphinxcontrib-serializinghtml`. When using them as
+`requirement()`s, ala
+
+```starlark
+py_binary(
+ name = "doctool",
+ ...
+ deps = [
+ requirement("sphinx"),
+ ],
+)
+```
+
+Bazel will protest because it doesn't support cycles in the build graph --
+
+```
+ERROR: .../external/pypi_sphinxcontrib_serializinghtml/BUILD.bazel:44:6: in alias rule @pypi_sphinxcontrib_serializinghtml//:pkg: cycle in dependency graph:
+ //:doctool (...)
+ @pypi//sphinxcontrib_serializinghtml:pkg (...)
+.-> @pypi_sphinxcontrib_serializinghtml//:pkg (...)
+| @pypi_sphinxcontrib_serializinghtml//:_pkg (...)
+| @pypi_sphinx//:pkg (...)
+| @pypi_sphinx//:_pkg (...)
+`-- @pypi_sphinxcontrib_serializinghtml//:pkg (...)
+```
+
+The `experimental_requirement_cycles` attribute allows you to work around these
+issues by specifying groups of packages which form cycles. `pip_parse` will
+transparently fix the cycles for you and provide the cyclic dependencies
+simultaneously.
+
+```starlark
+ ...
+ experimental_requirement_cycles = {
+ "sphinx": [
+ "sphinx",
+ "sphinxcontrib-serializinghtml",
+ ]
+ },
+)
+```
+
+`pip_parse` supports fixing multiple cycles simultaneously, however cycles must
+be distinct. `apache-airflow` for instance has dependency cycles with a number
+of its optional dependencies, which means those optional dependencies must all
+be a part of the `airflow` cycle. For instance --
+
+```starlark
+ ...
+ experimental_requirement_cycles = {
+ "airflow": [
+ "apache-airflow",
+ "apache-airflow-providers-common-sql",
+ "apache-airflow-providers-postgres",
+ "apache-airflow-providers-sqlite",
+ ]
+ }
+)
+```
+
+Alternatively, one could resolve the cycle by removing one leg of it.
+
+For example while `apache-airflow-providers-sqlite` is "baked into" the Airflow
+package, `apache-airflow-providers-postgres` is not and is an optional feature.
+Rather than listing `apache-airflow[postgres]` in your `requirements.txt` which
+would expose a cycle via the extra, one could either _manually_ depend on
+`apache-airflow` and `apache-airflow-providers-postgres` separately as
+requirements. Bazel rules which need only `apache-airflow` can take it as a
+dependency, and rules which explicitly want to mix in
+`apache-airflow-providers-postgres` now can.
+
+Alternatively, one could use `rules_python`'s patching features to remove one
+leg of the dependency manually. For instance by making
+`apache-airflow-providers-postgres` not explicitly depend on `apache-airflow` or
+perhaps `apache-airflow-providers-common-sql`.
diff --git a/docs/pypi/download-workspace.md b/docs/pypi/download-workspace.md
new file mode 100644
index 0000000..4871009
--- /dev/null
+++ b/docs/pypi/download-workspace.md
@@ -0,0 +1,107 @@
+:::{default-domain} bzl
+:::
+
+# Download (WORKSPACE)
+
+This documentation page covers how to download the PyPI dependencies in the legacy `WORKSPACE` setup.
+
+To add pip dependencies to your `WORKSPACE`, load the `pip_parse` function and
+call it to create the central external repo and individual wheel external repos.
+
+```starlark
+load("@rules_python//python:pip.bzl", "pip_parse")
+
+# Create a central repo that knows about the dependencies needed from
+# requirements_lock.txt.
+pip_parse(
+ name = "my_deps",
+ requirements_lock = "//path/to:requirements_lock.txt",
+)
+
+# Load the starlark macro, which will define your dependencies.
+load("@my_deps//:requirements.bzl", "install_deps")
+
+# Call it to define repos for your requirements.
+install_deps()
+```
+
+## Interpreter selection
+
+Note that pip parse runs before the Bazel before decides which Python toolchain to use, it cannot
+enforce that the interpreter used to invoke `pip` matches the interpreter used to run `py_binary`
+targets. By default, `pip_parse` uses the system command `"python3"`. To override this, pass in the
+{attr}`pip_parse.python_interpreter` attribute or {attr}`pip_parse.python_interpreter_target`.
+
+You can have multiple `pip_parse`s in the same workspace. This configuration will create multiple
+external repos that have no relation to one another and may result in downloading the same wheels
+numerous times.
+
+As with any repository rule, if you would like to ensure that `pip_parse` is
+re-executed to pick up a non-hermetic change to your environment (e.g., updating
+your system `python` interpreter), you can force it to re-execute by running
+`bazel sync --only [pip_parse name]`.
+
+(per-os-arch-requirements)=
+## Requirements for a specific OS/Architecture
+
+In some cases you may need to use different requirements files for different OS, Arch combinations.
+This is enabled via the {attr}`pip_parse.requirements_by_platform` attribute. The keys of the
+dictionary are labels to the file and the values are a list of comma separated target (os, arch)
+tuples.
+
+For example:
+```starlark
+ # ...
+ requirements_by_platform = {
+ "requirements_linux_x86_64.txt": "linux_x86_64",
+ "requirements_osx.txt": "osx_*",
+ "requirements_linux_exotic.txt": "linux_exotic",
+ "requirements_some_platforms.txt": "linux_aarch64,windows_*",
+ },
+ # For the list of standard platforms that the rules_python has toolchains for, default to
+ # the following requirements file.
+ requirements_lock = "requirements_lock.txt",
+```
+
+In case of duplicate platforms, `rules_python` will raise an error as there has
+to be unambiguous mapping of the requirement files to the (os, arch) tuples.
+
+An alternative way is to use per-OS requirement attributes.
+```starlark
+ # ...
+ requirements_windows = "requirements_windows.txt",
+ requirements_darwin = "requirements_darwin.txt",
+ # For the remaining platforms (which is basically only linux OS), use this file.
+ requirements_lock = "requirements_lock.txt",
+)
+```
+
+:::{note}
+If you are using a universal lock file but want to restrict the list of platforms that
+the lock file will be evaluated against, consider using the aforementioned
+`requirements_by_platform` attribute and listing the platforms explicitly.
+:::
+
+(vendoring-requirements)=
+## Vendoring the requirements.bzl file
+
+:::{note}
+For `bzlmod`, refer to standard `bazel vendor` usage if you want to really vendor it, otherwise
+just use the `pip` extension as you would normally.
+
+However, be aware that there are caveats when doing so.
+:::
+
+In some cases you may not want to generate the requirements.bzl file as a repository rule
+while Bazel is fetching dependencies. For example, if you produce a reusable Bazel module
+such as a ruleset, you may want to include the `requirements.bzl` file rather than make your users
+install the `WORKSPACE` setup to generate it, see {gh-issue}`608`.
+
+This is the same workflow as Gazelle, which creates `go_repository` rules with
+[`update-repos`](https://github.com/bazelbuild/bazel-gazelle#update-repos)
+
+To do this, use the "write to source file" pattern documented in
+<https://blog.aspect.dev/bazel-can-write-to-the-source-folder>
+to put a copy of the generated `requirements.bzl` into your project.
+Then load the requirements.bzl file directly rather than from the generated repository.
+See the example in {gh-path}`examples/pip_parse_vendored`.
diff --git a/docs/pypi/download.md b/docs/pypi/download.md
new file mode 100644
index 0000000..18d6699
--- /dev/null
+++ b/docs/pypi/download.md
@@ -0,0 +1,302 @@
+:::{default-domain} bzl
+:::
+
+# Download (bzlmod)
+
+:::{seealso}
+For WORKSPACE instructions see [here](./download-workspace).
+:::
+
+To add PyPI dependencies to your `MODULE.bazel` file, use the `pip.parse`
+extension, and call it to create the central external repo and individual wheel
+external repos. Include in the `MODULE.bazel` the toolchain extension as shown
+in the first bzlmod example above.
+
+```starlark
+pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
+
+pip.parse(
+ hub_name = "my_deps",
+ python_version = "3.13",
+ requirements_lock = "//:requirements_lock_3_11.txt",
+)
+
+use_repo(pip, "my_deps")
+```
+
+For more documentation, see the bzlmod examples under the {gh-path}`examples` folder or the documentation
+for the {obj}`@rules_python//python/extensions:pip.bzl` extension.
+
+:::note}
+We are using a host-platform compatible toolchain by default to setup pip dependencies.
+During the setup phase, we create some symlinks, which may be inefficient on Windows
+by default. In that case use the following `.bazelrc` options to improve performance if
+you have admin privileges:
+
+ startup --windows_enable_symlinks
+
+This will enable symlinks on Windows and help with bootstrap performance of setting up the
+hermetic host python interpreter on this platform. Linux and OSX users should see no
+difference.
+:::
+
+## Interpreter selection
+
+The {obj}`pip.parse` `bzlmod` extension by default uses the hermetic python toolchain for the host
+platform, but you can customize the interpreter using {attr}`pip.parse.python_interpreter` and
+{attr}`pip.parse.python_interpreter_target`.
+
+You can use the pip extension multiple times. This configuration will create
+multiple external repos that have no relation to one another and may result in
+downloading the same wheels numerous times.
+
+As with any repository rule or extension, if you would like to ensure that `pip_parse` is
+re-executed to pick up a non-hermetic change to your environment (e.g., updating your system
+`python` interpreter), you can force it to re-execute by running `bazel sync --only [pip_parse
+name]`.
+
+(per-os-arch-requirements)=
+## Requirements for a specific OS/Architecture
+
+In some cases you may need to use different requirements files for different OS, Arch combinations.
+This is enabled via the `requirements_by_platform` attribute in `pip.parse` extension and the
+{obj}`pip.parse` tag class. The keys of the dictionary are labels to the file and the values are a
+list of comma separated target (os, arch) tuples.
+
+For example:
+```starlark
+ # ...
+ requirements_by_platform = {
+ "requirements_linux_x86_64.txt": "linux_x86_64",
+ "requirements_osx.txt": "osx_*",
+ "requirements_linux_exotic.txt": "linux_exotic",
+ "requirements_some_platforms.txt": "linux_aarch64,windows_*",
+ },
+ # For the list of standard platforms that the rules_python has toolchains for, default to
+ # the following requirements file.
+ requirements_lock = "requirements_lock.txt",
+```
+
+In case of duplicate platforms, `rules_python` will raise an error as there has
+to be unambiguous mapping of the requirement files to the (os, arch) tuples.
+
+An alternative way is to use per-OS requirement attributes.
+```starlark
+ # ...
+ requirements_windows = "requirements_windows.txt",
+ requirements_darwin = "requirements_darwin.txt",
+ # For the remaining platforms (which is basically only linux OS), use this file.
+ requirements_lock = "requirements_lock.txt",
+)
+```
+
+:::{note}
+If you are using a universal lock file but want to restrict the list of platforms that
+the lock file will be evaluated against, consider using the aforementioned
+`requirements_by_platform` attribute and listing the platforms explicitly.
+:::
+
+## Multi-platform support
+
+Historically the {obj}`pip_parse` and {obj}`pip.parse` have been only downloading/building
+Python dependencies for the host platform that the `bazel` commands are executed on. Over
+the years people started needing support for building containers and usually that involves
+fetching dependencies for a particular target platform that may be other than the host
+platform.
+
+Multi-platform support of cross-building the wheels can be done in two ways:
+1. using {attr}`experimental_index_url` for the {bzl:obj}`pip.parse` bzlmod tag class
+2. using {attr}`pip.parse.download_only` setting.
+
+:::{warning}
+This will not for sdists with C extensions, but pure Python sdists may still work using the first
+approach.
+:::
+
+### Using `download_only` attribute
+
+Let's say you have 2 requirements files:
+```
+# requirements.linux_x86_64.txt
+--platform=manylinux_2_17_x86_64
+--python-version=39
+--implementation=cp
+--abi=cp39
+
+foo==0.0.1 --hash=sha256:deadbeef
+bar==0.0.1 --hash=sha256:deadb00f
+```
+
+```
+# requirements.osx_aarch64.txt contents
+--platform=macosx_10_9_arm64
+--python-version=39
+--implementation=cp
+--abi=cp39
+
+foo==0.0.3 --hash=sha256:deadbaaf
+```
+
+With these 2 files your {bzl:obj}`pip.parse` could look like:
+```starlark
+pip.parse(
+ hub_name = "pip",
+ python_version = "3.9",
+ # Tell `pip` to ignore sdists
+ download_only = True,
+ requirements_by_platform = {
+ "requirements.linux_x86_64.txt": "linux_x86_64",
+ "requirements.osx_aarch64.txt": "osx_aarch64",
+ },
+)
+```
+
+With this, the `pip.parse` will create a hub repository that is going to
+support only two platforms - `cp39_osx_aarch64` and `cp39_linux_x86_64` and it
+will only use `wheels` and ignore any sdists that it may find on the PyPI
+compatible indexes.
+
+:::{warning}
+Because bazel is not aware what exactly is downloaded, the same wheel may be downloaded
+multiple times.
+:::
+
+:::{note}
+This will only work for wheel-only setups, i.e. all of your dependencies need to have wheels
+available on the PyPI index that you use.
+:::
+
+### Customizing `Requires-Dist` resolution
+
+:::{note}
+Currently this is disabled by default, but you can turn it on using
+{envvar}`RULES_PYTHON_ENABLE_PIPSTAR` environment variable.
+:::
+
+In order to understand what dependencies to pull for a particular package
+`rules_python` parses the `whl` file [`METADATA`][metadata].
+Packages can express dependencies via `Requires-Dist` and they can add conditions using
+"environment markers", which represent the Python version, OS, etc.
+
+While the PyPI integration provides reasonable defaults to support most
+platforms and environment markers, the values it uses can be customized in case
+more esoteric configurations are needed.
+
+To customize the values used, you need to do two things:
+1. Define a target that returns {obj}`EnvMarkerInfo`
+2. Set the {obj}`//python/config_settings:pip_env_marker_config` flag to
+ the target defined in (1).
+
+The keys and values should be compatible with the [PyPA dependency specifiers
+specification](https://packaging.python.org/en/latest/specifications/dependency-specifiers/).
+This is not strictly enforced, however, so you can return a subset of keys or
+additional keys, which become available during dependency evaluation.
+
+[metadata]: https://packaging.python.org/en/latest/specifications/core-metadata/
+
+(bazel-downloader)=
+### Bazel downloader and multi-platform wheel hub repository.
+
+:::{warning}
+This is currently still experimental and whilst it has been proven to work in quite a few
+environments, the APIs are still being finalized and there may be changes to the APIs for this
+feature without much notice.
+
+The issues that you can subscribe to for updates are:
+* {gh-issue}`260`
+* {gh-issue}`1357`
+:::
+
+The {obj}`pip` extension supports pulling information from `PyPI` (or a compatible mirror) and it
+will ensure that the [bazel downloader][bazel_downloader] is used for downloading the wheels.
+
+This provides the following benefits:
+* Integration with the [credential_helper](#credential-helper) to authenticate with private
+ mirrors.
+* Cache the downloaded wheels speeding up the consecutive re-initialization of the repositories.
+* Reuse the same instance of the wheel for multiple target platforms.
+* Allow using transitions and targeting free-threaded and musl platforms more easily.
+* Avoids `pip` for wheel fetching and results in much faster dependency fetching.
+
+To enable the feature specify {attr}`pip.parse.experimental_index_url` as shown in
+the {gh-path}`examples/bzlmod/MODULE.bazel` example.
+
+Similar to [uv](https://docs.astral.sh/uv/configuration/indexes/), one can override the
+index that is used for a single package. By default we first search in the index specified by
+{attr}`pip.parse.experimental_index_url`, then we iterate through the
+{attr}`pip.parse.experimental_extra_index_urls` unless there are overrides specified via
+{attr}`pip.parse.experimental_index_url_overrides`.
+
+When using this feature during the `pip` extension evaluation you will see the accessed indexes similar to below:
+```console
+Loading: 0 packages loaded
+ Fetching module extension @@//python/extensions:pip.bzl%pip; Fetch package lists from PyPI index
+ Fetching https://pypi.org/simple/jinja2/
+
+```
+
+This does not mean that `rules_python` is fetching the wheels eagerly, but it
+rather means that it is calling the PyPI server to get the Simple API response
+to get the list of all available source and wheel distributions. Once it has
+got all of the available distributions, it will select the right ones depending
+on the `sha256` values in your `requirements_lock.txt` file. If `sha256` hashes
+are not present in the requirements file, we will fallback to matching by version
+specified in the lock file.
+
+Fetching the distribution information from the PyPI allows `rules_python` to
+know which `whl` should be used on which target platform and it will determine
+that by parsing the `whl` filename based on [PEP600], [PEP656] standards. This
+allows the user to configure the behaviour by using the following publicly
+available flags:
+* {obj}`--@rules_python//python/config_settings:py_linux_libc` for selecting the Linux libc variant.
+* {obj}`--@rules_python//python/config_settings:pip_whl` for selecting `whl` distribution preference.
+* {obj}`--@rules_python//python/config_settings:pip_whl_osx_arch` for selecting MacOS wheel preference.
+* {obj}`--@rules_python//python/config_settings:pip_whl_glibc_version` for selecting the GLIBC version compatibility.
+* {obj}`--@rules_python//python/config_settings:pip_whl_muslc_version` for selecting the musl version compatibility.
+* {obj}`--@rules_python//python/config_settings:pip_whl_osx_version` for selecting MacOS version compatibility.
+
+[bazel_downloader]: https://bazel.build/rules/lib/builtins/repository_ctx#download
+[pep600]: https://peps.python.org/pep-0600/
+[pep656]: https://peps.python.org/pep-0656/
+
+(credential-helper)=
+## Credential Helper
+
+The [Bazel downloader](#bazel-downloader) usage allows for the Bazel
+[Credential Helper][cred-helper-design].
+Your python artifact registry may provide a credential helper for you.
+Refer to your index's docs to see if one is provided.
+
+The simplest form of a credential helper is a bash script that accepts an arg and spits out JSON to
+stdout. For a service like Google Artifact Registry that uses ['Basic' HTTP Auth][rfc7617] and does
+not provide a credential helper that conforms to the [spec][cred-helper-spec], the script might
+look like:
+
+```bash
+#!/bin/bash
+# cred_helper.sh
+ARG=$1 # but we don't do anything with it as it's always "get"
+
+# formatting is optional
+echo '{'
+echo ' "headers": {'
+echo ' "Authorization": ["Basic dGVzdDoxMjPCow=="]'
+echo ' }'
+echo '}'
+```
+
+Configure Bazel to use this credential helper for your python index `example.com`:
+
+```
+# .bazelrc
+build --credential_helper=example.com=/full/path/to/cred_helper.sh
+```
+
+Bazel will call this file like `cred_helper.sh get` and use the returned JSON to inject headers
+into whatever HTTP(S) request it performs against `example.com`.
+
+See the [Credential Helper Spec][cred-helper-spec] for more details.
+
+[rfc7617]: https://datatracker.ietf.org/doc/html/rfc7617
+[cred-helper-design]: https://github.com/bazelbuild/proposals/blob/main/designs/2022-06-07-bazel-credential-helpers.md
+[cred-helper-spec]: https://github.com/EngFlow/credential-helper-spec/blob/main/spec.md
diff --git a/docs/pypi/index.md b/docs/pypi/index.md
new file mode 100644
index 0000000..c300124
--- /dev/null
+++ b/docs/pypi/index.md
@@ -0,0 +1,27 @@
+:::{default-domain} bzl
+:::
+
+# Using PyPI
+
+Using PyPI packages (aka "pip install") involves the following main steps.
+
+1. [Generating requirements file](./lock)
+2. Installing third party packages in [bzlmod](./download) or [WORKSPACE](./download-workspace).
+3. [Using third party packages as dependencies](./use)
+
+With the advanced topics covered separately:
+* Dealing with [circular dependencies](./circular-dependencies).
+
+```{toctree}
+lock
+download
+download-workspace
+use
+```
+
+## Advanced topics
+
+```{toctree}
+circular-dependencies
+patch
+```
diff --git a/docs/pypi/lock.md b/docs/pypi/lock.md
new file mode 100644
index 0000000..c937603
--- /dev/null
+++ b/docs/pypi/lock.md
@@ -0,0 +1,46 @@
+:::{default-domain} bzl
+:::
+
+# Lock
+
+:::{note}
+Currently `rules_python` only supports `requirements.txt` format.
+:::
+
+## requirements.txt
+
+### pip compile
+
+Generally, when working on a Python project, you'll have some dependencies that themselves have other dependencies. You might also specify dependency bounds instead of specific versions. So you'll need to generate a full list of all transitive dependencies and pinned versions for every dependency.
+
+Typically, you'd have your project dependencies specified in `pyproject.toml` or `requirements.in` and generate the full pinned list of dependencies in `requirements_lock.txt`, which you can manage with the {obj}`compile_pip_requirements`:
+
+```starlark
+load("@rules_python//python:pip.bzl", "compile_pip_requirements")
+
+compile_pip_requirements(
+ name = "requirements",
+ src = "requirements.in",
+ requirements_txt = "requirements_lock.txt",
+)
+```
+
+This rule generates two targets:
+- `bazel run [name].update` will regenerate the `requirements_txt` file
+- `bazel test [name]_test` will test that the `requirements_txt` file is up to date
+
+Once you generate this fully specified list of requirements, you can install the requirements ([bzlmod](./download)/[WORKSPACE](./download-workspace)).
+
+:::{warning}
+If you're specifying dependencies in `pyproject.toml`, make sure to include the `[build-system]` configuration, with pinned dependencies. `compile_pip_requirements` will use the build system specified to read your project's metadata, and you might see non-hermetic behavior if you don't pin the build system.
+
+Not specifying `[build-system]` at all will result in using a default `[build-system]` configuration, which uses unpinned versions ([ref](https://peps.python.org/pep-0518/#build-system-table)).
+:::
+
+### uv pip compile (bzlmod only)
+
+We also have experimental setup for the `uv pip compile` way of generating lock files.
+This is well tested with the public PyPI index, but you may hit some rough edges with private
+mirrors.
+
+For more documentation see {obj}`lock` documentation.
diff --git a/docs/pypi/patch.md b/docs/pypi/patch.md
new file mode 100644
index 0000000..f341bd1
--- /dev/null
+++ b/docs/pypi/patch.md
@@ -0,0 +1,10 @@
+:::{default-domain} bzl
+:::
+
+# Patching wheels
+
+Sometimes the wheels have to be patched to:
+* Workaround the lack of a standard `site-packages` layout ({gh-issue}`2156`)
+* Include certain PRs of your choice on top of wheels and avoid building from sdist,
+
+You can patch the wheels by using the {attr}`pip.override.patches` attribute.
diff --git a/docs/pypi/use.md b/docs/pypi/use.md
new file mode 100644
index 0000000..7a16b7d
--- /dev/null
+++ b/docs/pypi/use.md
@@ -0,0 +1,133 @@
+:::{default-domain} bzl
+:::
+
+# Use in BUILD.bazel files
+
+Once you have setup the dependencies, you are ready to start using them in your `BUILD.bazel`
+files. If you haven't done so yet, set it up by following the following docs:
+1. [WORKSPACE](./download-workspace)
+1. [bzlmod](./download)
+
+To refer to targets in a hub repo `pypi`, you can do one of two things:
+```starlark
+py_library(
+ name = "my_lib",
+ deps = [
+ "@pypi//numpy",
+ ],
+)
+```
+
+Or use the `requirement` helper that needs to be loaded from the `hub` repo itself:
+```starlark
+load("@pypi//:requirements.bzl", "requirement")
+
+py_library(
+ deps = [
+ requirement("numpy")
+ ],
+)
+```
+
+Note, that the usage of the `requirement` helper is not advised and can be problematic. See the
+[notes below](#requirement-helper).
+
+Note, that the hub repo contains the following targets for each package:
+* `@pypi//numpy` which is a shorthand for `@pypi//numpy:numpy`. This is an {obj}`alias` to
+ `@pypi//numpy:pkg`.
+* `@pypi//numpy:pkg` - the {obj}`py_library` target automatically generated by the repository
+ rules.
+* `@pypi//numpy:data` - the {obj}`filegroup` that is for all of the extra files that are included
+ as data in the `pkg` target.
+* `@pypi//numpy:dist_info` - the {obj}`filegroup` that is for all of the files in the `<pkg prefix with version>.distinfo` directory.
+* `@pypi//numpy:whl` - the {obj}`filegroup` that is the `.whl` file itself which includes all of
+ the transitive dependencies via the {attr}`filegroup.data` attribute.
+
+## Entry points
+
+If you would like to access [entry points][whl_ep], see the `py_console_script_binary` rule documentation,
+which can help you create a `py_binary` target for a particular console script exposed by a package.
+
+[whl_ep]: https://packaging.python.org/specifications/entry-points/
+
+## 'Extras' dependencies
+
+Any 'extras' specified in the requirements lock file will be automatically added
+as transitive dependencies of the package. In the example above, you'd just put
+`requirement("useful_dep")` or `@pypi//useful_dep`.
+
+## Consuming Wheel Dists Directly
+
+If you need to depend on the wheel dists themselves, for instance, to pass them
+to some other packaging tool, you can get a handle to them with the
+`whl_requirement` macro. For example:
+
+```starlark
+load("@pypi//:requirements.bzl", "whl_requirement")
+
+filegroup(
+ name = "whl_files",
+ data = [
+ # This is equivalent to "@pypi//boto3:whl"
+ whl_requirement("boto3"),
+ ]
+)
+```
+
+## Creating a filegroup of files within a whl
+
+The rule {obj}`whl_filegroup` exists as an easy way to extract the necessary files
+from a whl file without the need to modify the `BUILD.bazel` contents of the
+whl repositories generated via `pip_repository`. Use it similarly to the `filegroup`
+above. See the API docs for more information.
+
+(requirement-helper)=
+## A note about using the requirement helper
+
+Each extracted wheel repo contains a `py_library` target representing
+the wheel's contents. There are two ways to access this library. The
+first uses the `requirement()` function defined in the central
+repo's `//:requirements.bzl` file. This function maps a pip package
+name to a label:
+
+```starlark
+load("@my_deps//:requirements.bzl", "requirement")
+
+py_library(
+ name = "mylib",
+ srcs = ["mylib.py"],
+ deps = [
+ ":myotherlib",
+ requirement("some_pip_dep"),
+ requirement("another_pip_dep"),
+ ]
+)
+```
+
+The reason `requirement()` exists is to insulate from
+changes to the underlying repository and label strings. However, those
+labels have become directly used, so aren't able to easily change regardless.
+
+On the other hand, using `requirement()` helper has several drawbacks:
+
+- It doesn't work with `buildifier`
+- It doesn't work with `buildozer`
+- It adds extra layer on top of normal mechanisms to refer to targets.
+- It does not scale well as each type of target needs a new macro to be loaded and imported.
+
+If you don't want to use `requirement()`, you can use the library labels directly instead. For
+`pip_parse`, the labels are of the following form:
+
+```starlark
+@{name}//{package}
+```
+
+Here `name` is the `name` attribute that was passed to `pip_parse` and
+`package` is the pip package name with characters that are illegal in
+Bazel label names (e.g. `-`, `.`) replaced with `_`. If you need to
+update `name` from "old" to "new", then you can run the following
+`buildozer` command:
+
+```shell
+buildozer 'substitute deps @old//([^/]+) @new//${1}' //...:*
+```
diff --git a/docs/requirements.txt b/docs/requirements.txt
index e4ec16f..87c13aa 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1,6 +1,5 @@
# This file was autogenerated by uv via the following command:
# bazel run //docs:requirements.update
---index-url https://pypi.org/simple
absl-py==2.2.2 \
--hash=sha256:bf25b2c2eed013ca456918c453d687eab4e8309fba81ee2f4c1a6aa2494175eb \
diff --git a/python/private/pypi/BUILD.bazel b/python/private/pypi/BUILD.bazel
index e9036c3..d89dc6c 100644
--- a/python/private/pypi/BUILD.bazel
+++ b/python/private/pypi/BUILD.bazel
@@ -398,6 +398,7 @@
":pep508_requirement_bzl",
":pypi_repo_utils_bzl",
":whl_metadata_bzl",
+ ":whl_target_platforms_bzl",
"//python/private:auth_bzl",
"//python/private:bzlmod_enabled_bzl",
"//python/private:envsubst_bzl",
diff --git a/python/private/pypi/pkg_aliases.bzl b/python/private/pypi/pkg_aliases.bzl
index 28d70ff..d71c37c 100644
--- a/python/private/pypi/pkg_aliases.bzl
+++ b/python/private/pypi/pkg_aliases.bzl
@@ -237,9 +237,10 @@
Exposed only for unit tests.
Args:
- aliases: {type}`str | dict[whl_config_setting | str, str]`: The aliases
+ aliases: {type}`str | dict[struct | str, str]`: The aliases
to process. Any aliases that have the filename set will be
- converted to a dict of config settings to repo names.
+ converted to a dict of config settings to repo names. The
+ struct is created by {func}`whl_config_setting`.
glibc_versions: {type}`list[tuple[int, int]]` list of versions that can be
used in this hub repo.
muslc_versions: {type}`list[tuple[int, int]]` list of versions that can be
diff --git a/python/private/pypi/simpleapi_download.bzl b/python/private/pypi/simpleapi_download.bzl
index e8d7d09..164d4e8 100644
--- a/python/private/pypi/simpleapi_download.bzl
+++ b/python/private/pypi/simpleapi_download.bzl
@@ -83,6 +83,7 @@
found_on_index = {}
warn_overrides = False
+ ctx.report_progress("Fetch package lists from PyPI index")
for i, index_url in enumerate(index_urls):
if i != 0:
# Warn the user about a potential fix for the overrides
diff --git a/python/private/pypi/whl_config_setting.bzl b/python/private/pypi/whl_config_setting.bzl
index 6e10eb4..3b81e46 100644
--- a/python/private/pypi/whl_config_setting.bzl
+++ b/python/private/pypi/whl_config_setting.bzl
@@ -21,14 +21,14 @@
aliases in a hub repository.
Args:
- version: optional(str), the version of the python toolchain that this
+ version: {type}`str | None`the version of the python toolchain that this
whl alias is for. If not set, then non-version aware aliases will be
constructed. This is mainly used for better error messages when there
is no match found during a select.
- config_setting: optional(Label or str), the config setting that we should use. Defaults
+ config_setting: {type}`str | Label | None` the config setting that we should use. Defaults
to "//_config:is_python_{version}".
- filename: optional(str), the distribution filename to derive the config_setting.
- target_platforms: optional(list[str]), the list of target_platforms for this
+ filename: {type}`str | None` the distribution filename to derive the config_setting.
+ target_platforms: {type}`list[str] | None` the list of target_platforms for this
distribution.
Returns:
diff --git a/sphinxdocs/inventories/bazel_inventory.txt b/sphinxdocs/inventories/bazel_inventory.txt
index bbd200d..e14ea76 100644
--- a/sphinxdocs/inventories/bazel_inventory.txt
+++ b/sphinxdocs/inventories/bazel_inventory.txt
@@ -15,6 +15,7 @@
RunEnvironmentInfo bzl:type 1 rules/lib/providers/RunEnvironmentInfo -
Target bzl:type 1 rules/lib/builtins/Target -
ToolchainInfo bzl:type 1 rules/lib/providers/ToolchainInfo.html -
+alias bzl:rule 1 reference/be/general#alias -
attr.bool bzl:type 1 rules/lib/toplevel/attr#bool -
attr.int bzl:type 1 rules/lib/toplevel/attr#int -
attr.int_list bzl:type 1 rules/lib/toplevel/attr#int_list -
@@ -40,6 +41,7 @@
config.target bzl:function 1 rules/lib/toplevel/config#target -
config_common.FeatureFlagInfo bzl:type 1 rules/lib/toplevel/config_common#FeatureFlagInfo -
config_common.toolchain_type bzl:function 1 rules/lib/toplevel/config_common#toolchain_type -
+config_setting bzl:rule 1 reference/be/general#config_setting -
ctx bzl:type 1 rules/lib/builtins/repository_ctx -
ctx.actions bzl:obj 1 rules/lib/builtins/ctx#actions -
ctx.aspect_ids bzl:obj 1 rules/lib/builtins/ctx#aspect_ids -
@@ -79,6 +81,8 @@
dict bzl:type 1 rules/lib/dict -
exec_compatible_with bzl:attr 1 reference/be/common-definitions#common.exec_compatible_with -
exec_group bzl:function 1 rules/lib/globals/bzl#exec_group -
+filegroup bzl:rule 1 reference/be/general#filegroup -
+filegroup.data bzl:attr 1 reference/be/general#filegroup.data -
int bzl:type 1 rules/lib/int -
label bzl:type 1 concepts/labels -
list bzl:type 1 rules/lib/list -
diff --git a/tests/pypi/simpleapi_download/simpleapi_download_tests.bzl b/tests/pypi/simpleapi_download/simpleapi_download_tests.bzl
index ce214d6..a96815c 100644
--- a/tests/pypi/simpleapi_download/simpleapi_download_tests.bzl
+++ b/tests/pypi/simpleapi_download/simpleapi_download_tests.bzl
@@ -43,6 +43,7 @@
contents = simpleapi_download(
ctx = struct(
os = struct(environ = {}),
+ report_progress = lambda _: None,
),
attr = struct(
index_url_overrides = {},
@@ -95,6 +96,7 @@
simpleapi_download(
ctx = struct(
os = struct(environ = {}),
+ report_progress = lambda _: None,
),
attr = struct(
index_url_overrides = {},
@@ -136,6 +138,7 @@
ctx = struct(
os = struct(environ = {}),
download = download,
+ report_progress = lambda _: None,
read = lambda i: "contents of " + i,
path = lambda i: "path/for/" + i,
),
@@ -171,6 +174,7 @@
ctx = struct(
os = struct(environ = {}),
download = download,
+ report_progress = lambda _: None,
read = lambda i: "contents of " + i,
path = lambda i: "path/for/" + i,
),
@@ -206,6 +210,7 @@
ctx = struct(
os = struct(environ = {"INDEX_URL": "https://example.com/main/simple/"}),
download = download,
+ report_progress = lambda _: None,
read = lambda i: "contents of " + i,
path = lambda i: "path/for/" + i,
),
