docs: rules_python bzlmod GA and 1.0 prep (#2296)
This is documenting the current state and closing the last remaining
TODO items for 1.0 release.
Work towards #1361.
---------
Co-authored-by: Greg Roodt <groodt@gmail.com>
Co-authored-by: Richard Levasseur <richardlev@gmail.com>
diff --git a/BZLMOD_SUPPORT.md b/BZLMOD_SUPPORT.md
index d3d0607..85e28ac 100644
--- a/BZLMOD_SUPPORT.md
+++ b/BZLMOD_SUPPORT.md
@@ -2,10 +2,12 @@
## `rules_python` `bzlmod` support
-- Status: Beta
+- Status: GA
- Full Feature Parity: No
+ - `rules_python`: Yes
+ - `rules_python_gazelle_plugin`: No (see below).
-Some features are missing or broken, and the public APIs are not yet stable.
+In general `bzlmod` has more features than `WORKSPACE` and users are encouraged to migrate.
## Configuration
@@ -27,15 +29,6 @@
A second example, in [examples/bzlmod_build_file_generation](examples/bzlmod_build_file_generation) demonstrates the use of `bzlmod` to configure `gazelle` support for `rules_python`.
-## Feature parity
-
-This rule set does not have full feature partity with the older `WORKSPACE` type configuration:
-
-1. Gazelle does not support finding deps in sub-modules. For instance we can have a dep like ` "@our_other_module//other_module/pkg:lib",` in a `py_test` definition.
-2. We have some features that are still not fully flushed out, and the user interface may change.
-
-Check ["issues"](/bazelbuild/rules_python/issues) for an up to date list.
-
## Differences in behavior from WORKSPACE
### Default toolchain is not the local system Python
@@ -52,10 +45,35 @@
If you want to use the same toolchain as what WORKSPACE used, then manually
register the builtin Bazel Python toolchain by doing
`register_toolchains("@bazel_tools//tools/python:autodetecting_toolchain")`.
-**IMPORTANT: this should only be done in a root module, and may intefere with
+
+Note that using this builtin Bazel toolchain is deprecated and unsupported.
+See the {obj}`runtime_env_toolchains` docs for a replacement that is marginally
+better supported.
+**IMPORTANT: this should only be done in a root module, and may interfere with
the toolchains rules_python registers**.
NOTE: Regardless of your toolchain, due to
[#691](https://github.com/bazelbuild/rules_python/issues/691), `rules_python`
still relies on a local Python being available to bootstrap the program before
handing over execution to the toolchain Python.
+
+To override this behaviour see {obj}`--bootstrap_impl=script`, which switches
+to `bash`-based bootstrap on UNIX systems.
+
+### Better PyPI package downloading on bzlmod
+
+On `bzlmod` users have the option to use the `bazel_downloader` to download packages
+and work correctly when `host` platform is not the same as the `target` platform. This
+provides faster package download times and integration with the credentials helper.
+
+### Extra targets in `whl_library` repos
+
+Due to how `bzlmod` is designed and the visibility rules that it enforces, it is best to use
+the targets in the `whl` repos as they do not rely on using the `annotations` API to
+add extra targets to so-called `spoke` repos. For alternatives that should cover most of the
+existing usecases please see:
+* {bzl:obj}`py_console_script_binary` to create `entry_point` targets.
+* {bzl:obj}`whl_filegroup` to extract filegroups from the `whl` targets (e.g. `@pip//numpy:whl`)
+* {bzl:obj}`pip.override` to patch the downloaded `whl` files. Using that you
+ can change the `METADATA` of the `whl` file that will influence how
+ `rules_python` code generation behaves.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 33b296f..a31781a 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -194,8 +194,13 @@
of. The API for the control mechanism can be removed in this release.
Note that the `+1` and `+2` releases are just examples; the steps are not
-required to happen in immedially subsequent releases.
+required to happen in immediately subsequent releases.
+Once The first major version is released, the process will be:
+1. In `N.M.0` we introduce the new behaviour, but it is disabled by a feature flag.
+2. In `N.M+1.0` we may choose the behaviour to become the default if it is not too
+ disruptive.
+3. In `N+1.0.0` we get rid of the old behaviour.
### How to control breaking changes
diff --git a/docs/getting-started.md b/docs/getting-started.md
index 45d1962..9f52243 100644
--- a/docs/getting-started.md
+++ b/docs/getting-started.md
@@ -1,7 +1,7 @@
# Getting started
-This doc is a simplified guide to help get started started quickly. It provides
-a simplified introduction to having a working Python program for both bzlmod
+This doc is a simplified guide to help get started quickly. It provides
+a simplified introduction to having a working Python program for both `bzlmod`
and the older way of using `WORKSPACE`.
It assumes you have a `requirements.txt` file with your PyPI dependencies.
@@ -23,11 +23,11 @@
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
- hub_name = "my_deps",
+ hub_name = "pypi",
python_version = "3.11",
requirements_lock = "//:requirements.txt",
)
-use_repo(pip, "my_deps")
+use_repo(pip, "pypi")
```
## Using a WORKSPACE file
@@ -38,19 +38,14 @@
```starlark
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
-
-# Update the SHA and VERSION to the lastest version available here:
-# https://github.com/bazelbuild/rules_python/releases.
-
-SHA="84aec9e21cc56fbc7f1335035a71c850d1b9b5cc6ff497306f84cced9a769841"
-
-VERSION="0.23.1"
+# Update the snippet based on the latest release below
+# https://github.com/bazelbuild/rules_python/releases
http_archive(
name = "rules_python",
- sha256 = SHA,
- strip_prefix = "rules_python-{}".format(VERSION),
- url = "https://github.com/bazelbuild/rules_python/releases/download/{}/rules_python-{}.tar.gz".format(VERSION,VERSION),
+ sha256 = "ca77768989a7f311186a29747e3e95c936a41dffac779aff6b443db22290d913",
+ strip_prefix = "rules_python-0.36.0",
+ url = "https://github.com/bazelbuild/rules_python/releases/download/0.36.0/rules_python-0.36.0.tar.gz",
)
load("@rules_python//python:repositories.bzl", "py_repositories")
@@ -66,14 +61,12 @@
python_version = "3.11",
)
-load("@python_3_11//:defs.bzl", "interpreter")
-
load("@rules_python//python:pip.bzl", "pip_parse")
pip_parse(
- ...
- python_interpreter_target = interpreter,
- ...
+ name = "pypi",
+ python_interpreter_target = "@python_3_11_host//:python",
+ requirements_lock = "//:requirements.txt",
)
```
@@ -89,8 +82,8 @@
name = "main",
srcs = ["main.py"],
deps = [
- "@my_deps//foo",
- "@my_deps//bar",
+ "@pypi//foo",
+ "@pypi//bar",
]
)
```
diff --git a/docs/index.md b/docs/index.md
index c06c31e..378aac2 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,39 +1,73 @@
# Python Rules for Bazel
-rules_python is the home of the core Python rules -- `py_library`,
-`py_binary`, `py_test`, `py_proto_library`, and related symbols that provide the basis for Python
-support in Bazel. It also contains package installation rules for integrating with PyPI and other indices.
+`rules_python` is the home for 4 major components with varying maturity levels.
-Documentation for rules_python lives here and in the
-[Bazel Build Encyclopedia](https://docs.bazel.build/versions/master/be/python.html).
+:::{topic} Core rules
-Examples are in the {gh-path}`examples` directory.
+The core Python rules -- `py_library`, `py_binary`, `py_test`,
+`py_proto_library`, and related symbols that provide the basis for Python
+support in Bazel.
-Currently, the core rules build into the Bazel binary, and the symbols in this
-repository are simple aliases. However, we are migrating the rules to Starlark and removing them from the Bazel binary. Therefore, the future-proof way to depend on Python rules is via this repository. See
-{ref}`Migrating from the Bundled Rules` below.
+When using Bazel 6 (or earlier), the core rules are bundled into the Bazel binary, and the symbols
+in this repository are simple aliases. On Bazel 7 and above `rules_python` uses
+a separate Starlark implementation,
+see {ref}`Migrating from the Bundled Rules` below.
-The core rules are stable. Their implementation in Bazel is subject to Bazel's
-[backward compatibility policy](https://docs.bazel.build/versions/master/backward-compatibility.html).
-Once migrated to rules_python, they may evolve at a different
-rate, but this repository will still follow [semantic versioning](https://semver.org).
+Once rules_python 1.0 is released, they will follow
+[semantic versioning](https://semver.org) and the breaking change policy
+outlined in the [support](support) page.
-The Bazel community maintains this repository. Neither Google nor the Bazel team provides support for the code. However, this repository is part of the test suite used to vet new Bazel releases. See
-{gh-path}`How to contribute <CONTRIBUTING.md>` for information on our development workflow.
+:::
-## Bzlmod support
+:::{topic} PyPI integration
-- Status: Beta
-- Full Feature Parity: No
+Package installation rules for integrating with PyPI and other SimpleAPI
+compatible indexes.
-See {gh-path}`Bzlmod support <BZLMOD_SUPPORT.md>` for more details
+These rules work and can be used in production, but the cross-platform building
+that supports pulling PyPI dependencies for a target platform that is different
+from the host platform is still in beta and the APIs that are subject to potential
+change are marked as `experimental`.
+
+:::
+
+:::{topic} Sphinxdocs
+
+`sphinxdocs` rules allow users to generate documentation using Sphinx powered by Bazel, with additional functionality for documenting
+Starlark and Bazel code.
+
+The functionality is exposed because other projects find it useful, but
+it is available as is and **the semantic versioning and
+compatibility policy used by `rules_python` does not apply**.
+
+:::
+
+:::{topic} Gazelle plugin
+
+`gazelle` plugin for generating `BUILD.bazel` files based on Python source
+code.
+
+This is available as is and the semantic versioning used by `rules_python` does
+not apply.
+
+:::
+
+The Bazel community maintains this repository. Neither Google nor the Bazel
+team provides support for the code. However, this repository is part of the
+test suite used to vet new Bazel releases. See {gh-path}`How to contribute
+<CONTRIBUTING.md>` for information on our development workflow.
+
+## Examples
+
+This documentation is an example of `sphinxdocs` rules and the rest of the
+components have examples in the {gh-path}`examples` directory.
## Migrating from the bundled rules
The core rules are currently available in Bazel as built-in symbols, but this
form is deprecated. Instead, you should depend on rules_python in your
-`WORKSPACE` file and load the Python rules from
-`@rules_python//python:defs.bzl`.
+`WORKSPACE` or `MODULE.bazel` file and load the Python rules from
+`@rules_python//python:defs.bzl` or load paths described in the API documentation.
A [buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md)
fix is available to automatically migrate `BUILD` and `.bzl` files to add the
@@ -44,13 +78,18 @@
buildifier --lint=fix --warnings=native-py <files>
```
-Currently, the `WORKSPACE` file needs to be updated manually as per [Getting
-started](getting-started).
+Currently, the `WORKSPACE` file needs to be updated manually as per
+[Getting started](getting-started).
Note that Starlark-defined bundled symbols underneath
`@bazel_tools//tools/python` are also deprecated. These are not yet rewritten
by buildifier.
+## Migrating to bzlmod
+
+See {gh-path}`Bzlmod support <BZLMOD_SUPPORT.md>` for any behaviour differences between
+`bzlmod` and `WORKSPACE`.
+
```{toctree}
:hidden:
diff --git a/examples/bzlmod/MODULE.bazel.lock b/examples/bzlmod/MODULE.bazel.lock
index 8c66a13..8cda22c 100644
--- a/examples/bzlmod/MODULE.bazel.lock
+++ b/examples/bzlmod/MODULE.bazel.lock
@@ -1392,7 +1392,7 @@
},
"@@rules_python~//python/extensions:pip.bzl%pip": {
"general": {
- "bzlTransitiveDigest": "iikkSIkMsBiM/vadkEf9xEoVbaxZqrkUg08hiHr/LKk=",
+ "bzlTransitiveDigest": "Okeg+CvZ5figeuuknTrCNwQR1dUzHr1+YvojOpIGXEI=",
"usagesDigest": "MChlcSw99EuW3K7OOoMcXQIdcJnEh6YmfyjJm+9mxIg=",
"recordedFileInputs": {
"@@other_module~//requirements_lock_3_11.txt": "a7d0061366569043d5efcf80e34a32c732679367cb3c831c4cdc606adc36d314",
@@ -6299,7 +6299,7 @@
},
"@@rules_python~//python/private/pypi:pip.bzl%pip_internal": {
"general": {
- "bzlTransitiveDigest": "WPfU9gogl29lCI8A/N2aYn7RAhsCpZikVU1Hw7nMtAc=",
+ "bzlTransitiveDigest": "dF4Tc81oqhFbL+/QjHfkMmWICMXhiduPI6eIB5d9VJY=",
"usagesDigest": "Y8ihY+R57BAFhalrVLVdJFrpwlbsiKz9JPJ99ljF7HA=",
"recordedFileInputs": {
"@@rules_python~//tools/publish/requirements.txt": "031e35d03dde03ae6305fe4b3d1f58ad7bdad857379752deede0f93649991b8a",
diff --git a/gazelle/manifest/defs.bzl b/gazelle/manifest/defs.bzl
index eacf1c1..3a65bff 100644
--- a/gazelle/manifest/defs.bzl
+++ b/gazelle/manifest/defs.bzl
@@ -39,7 +39,7 @@
manifest, meaning testing it is just as expensive as generating it,
but modifying it is much less likely to result in a merge conflict.
pip_repository_name: the name of the pip_install or pip_repository target.
- pip_deps_repository_name: deprecated - the old pip_install target name.
+ pip_deps_repository_name: deprecated - the old {bzl:obj}`pip_parse` target name.
manifest: the Gazelle manifest file.
defaults to the same value as manifest.
**kwargs: other bazel attributes passed to the generate and test targets
diff --git a/gazelle/manifest/generate/generate.go b/gazelle/manifest/generate/generate.go
index 19ca08a..27cf2a2 100644
--- a/gazelle/manifest/generate/generate.go
+++ b/gazelle/manifest/generate/generate.go
@@ -55,7 +55,7 @@
&pipRepositoryName,
"pip-repository-name",
"",
- "The name of the pip_install or pip_repository target.")
+ "The name of the pip_parse or pip.parse target.")
flag.StringVar(
&modulesMappingPath,
"modules-mapping",
diff --git a/internal_setup.bzl b/internal_setup.bzl
index b3dc326..b28c0e2 100644
--- a/internal_setup.bzl
+++ b/internal_setup.bzl
@@ -43,7 +43,6 @@
python_versions = sorted(TOOL_VERSIONS.keys()),
)
- # Because we don't use the pip_install rule, we have to call this to fetch its deps
pypi_deps()
bazel_skylib_workspace()
diff --git a/python/extensions/pip.bzl b/python/extensions/pip.bzl
index e9d4726..62a51c6 100644
--- a/python/extensions/pip.bzl
+++ b/python/extensions/pip.bzl
@@ -12,7 +12,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-"pip module extension for use with bzlmod"
+"""
+This is the successor to {bzl:obj}`pip_parse` for including third party PyPI dependencies into your bazel module using `bzlmod`.
+
+:::{seealso}
+For user documentation see the [PyPI dependencies section](pypi-dependencies).
+:::
+"""
load("//python/private/pypi:pip.bzl", _pip = "pip")
diff --git a/python/pip.bzl b/python/pip.bzl
index a1a6720..44ee69d 100644
--- a/python/pip.bzl
+++ b/python/pip.bzl
@@ -17,6 +17,10 @@
dependencies via fully locked `requirements.txt` files. Some of the exported
symbols should not be used and they are either undocumented here or marked as
for internal use only.
+
+If you are using a bazel version 7 or above with `bzlmod`, you should only care
+about the {bzl:obj}`compile_pip_requirements` macro exposed in this file. The
+rest of the symbols are for legacy `WORKSPACE` setups.
"""
load("//python/private:normalize_name.bzl", "normalize_name")
diff --git a/python/private/pypi/extension.bzl b/python/private/pypi/extension.bzl
index 36fb20e..6f8ca58 100644
--- a/python/private/pypi/extension.bzl
+++ b/python/private/pypi/extension.bzl
@@ -764,8 +764,7 @@
attrs = _pip_parse_ext_attrs(),
doc = """\
This tag class is used to create a pip hub and all of the spokes that are part of that hub.
-This tag class reuses most of the pip attributes that are found in
-@rules_python//python/pip_install:pip_repository.bzl.
+This tag class reuses most of the attributes found in {bzl:obj}`pip_parse`.
The exception is it does not use the arg 'repo_prefix'. We set the repository
prefix for the user and the alias arg is always True in bzlmod.
""",
@@ -814,8 +813,7 @@
),
doc = """\
This tag class is used to create a pypi hub and all of the spokes that are part of that hub.
-This tag class reuses most of the pypi attributes that are found in
-@rules_python//python/pip_install:pip_repository.bzl.
+This tag class reuses most of the attributes found in {bzl:obj}`pip_parse`.
The exception is it does not use the arg 'repo_prefix'. We set the repository
prefix for the user and the alias arg is always True in bzlmod.
""",
