feat(pypi): generate requirements.bzl in the unified @pypi hub (#4223)
The unified `@pypi` hub (#3837) does not generate a `requirements.bzl`.
That makes renaming a hub away from the now-reserved `pypi` name an
all-at-once change: the moment the concrete hub stops being `@pypi`,
every `load("@pypi//:requirements.bzl", "requirement")` in the repo
fails to load. In our monorepo that is ~940 BUILD files, and any BUILD
file that lands on main mid-migration re-breaks it. The same thing would
happen to every such repo if `RULES_PYTHON_PYPI_HUB_RESERVED` is flipped
on by default, as discussed in #3837.
This adds an opt-in, `pip.default(unified_hub_requirements_bzl = True)`
(root module only, off by default), that generates a `requirements.bzl`
in the unified hub with the per-package macros of a concrete hub's. It's
opt-in because the docs already steer users to `@pypi//<pkg>` labels
over `requirement()`; this is a migration aid for repos whose hub used
to be named `pypi`, not a new default surface. With it on:
* `requirement()`, `whl_requirement()`, `data_requirement()` and
`dist_info_requirement()` return labels in the unified hub
(`@@<unified>//<pkg>:pkg` etc., the same canonical-name form the
concrete hubs use), so they route through
`--@rules_python//python/config_settings:venv` exactly like a plain
`@pypi//<pkg>` label. All of those targets already exist in every
unified package (`_STANDARD_ALIASES`).
* The `all_*` lists (`all_requirements`,
`all_whl_requirements(_by_package)`, `all_data_requirements`) are
deliberately not generated. They are fixed at loading time, before the
venv flag is known, so they could only list one hub's packages and would
silently not follow the flag like the rest of the unified hub. Code that
wants a whole lock loads them from that concrete hub, and a stale
`load("@pypi//:requirements.bzl", "all_requirements")` fails loudly
instead.
Before: after renaming `hub_name = "pypi"` to e.g. `"pypi_main"` with
`pip.default(default_hub = "pypi_main")`,
`load("@pypi//:requirements.bzl", ...)` fails with "no such file".
After, with the opt-in: it keeps working and resolves to the same
wheels, and targets can migrate to `"@pypi//<pkg>"` labels incrementally
(we're moving our own repo to labels and gating new `requirement()`
calls in CI). Without the opt-in nothing changes.
Tests:
* `tests/pypi/extension`: the setting is off by default, on when the
root module sets it, and ignored from a non-root module.
* `tests/integration/unified_pypi` (which now opts in): `requirement()`
with a non-normalized name on the default hub, `requirement()` under a
venv transition, and loading `@pypi//:requirements.bzl` failing once the
opt-in is removed. Passes on `bazel_self` and `bazel_7.7.0`.
Context: we're carrying this as a local patch in a large monorepo, where
renaming our main hub and loading `requirement` from the unified hub
leaves the resolved wheel set unchanged. #4172 also touches
`tests/integration/unified_pypi`; the two should merge independently,
but one of them may need a trivial rebase.
(Done with help from an agent.)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This repository is the home of the core Python rules -- py_library, py_binary, py_test, and related symbols that provide the basis for Python support in Bazel. It also contains package installation rules for integrating with PyPI and other indices.
Documentation for rules_python is at https://rules-python.readthedocs.io and in the Bazel Build Encyclopedia.
Examples live in the examples directory.
The core rules are stable. Their implementation is subject to Bazel's backward compatibility policy. This repository aims to follow semantic versioning.
The Bazel community maintains this repository. Neither Google nor the Bazel team provides support for the code. However, this repository is part of the test suite used to vet new Bazel releases. See How to contribute page for information on our development workflow.
requirements.txt is how users have been defining dependencies for a long time. We support this to support legacy usecases or package managers that we don't support directly. Any additional information that we need will be retrieved from the SimpleAPI during the bzlmod extension evaluation phase. Then it will be written to the MODULE.bazel.lock file for future reuse. We have plans to support uv.lock file directly. uv is recommended for generating a fully locked requirements.txt file and we do provide a rule for it.py_binary, py_test rules should scale to large monorepos and we work hard to minimize the work done during analysis and build phase. What is more, the space requirements for should be minimal, so we strive to use symlinks rather than extracting wheels at build time. This means that for different configurations of the same build, we are not extracting the wheel multiple times thus scaling better over the time. From 2.0 onwards we are creating a virtual env for each target by creating an actual minimal virtual environment using symlinks. We plan on creating the traditional site-packages layout in the future by default.rules_python and this has resulted in a few PEPs supported within pure starlark - PEP440, PEP509.Common misconceptions:
rules_python has to keep backwards compatibility with google3. Whilst this might have been true in the past, rules_python is an open source project and any compatibility needs should come from the community - we have no requirement to keep this compatibility and are allowed to make our decisions. However, we do want to keep backwards compatibility as long as possible to not upset users with never ending migrations.rules_python is not caching pip downloads. With 2.0, we use Bazel's downloader by default and rely on bazel to provide the repository caching mechanisms. This means that for simpler setups this should result in transparent and scalable caching with the most recent bazel versions unless there are issues in the bazel itself.For detailed documentation, see https://rules-python.readthedocs.io
See Bzlmod support for more details.