docs: give some general guidance on how to define custom toolchains (#2220)
From the discussion in #2216, it's clear that some better docs are
needed to help people
figure out how to define a toolchain and what the different pieces me.
This gives some more explanation of the toolchain types, what they do,
and how to
define them.
Along the way:
* Add some more bazel objects to the inventory
* Fix attribute lookups in the bazel inventory
* Allow using parens in crossrefs, e.g. `foo()`
diff --git a/docs/api/rules_python/python/cc/index.md b/docs/api/rules_python/python/cc/index.md
index acaaf4f..233b130 100644
--- a/docs/api/rules_python/python/cc/index.md
+++ b/docs/api/rules_python/python/cc/index.md
@@ -25,3 +25,10 @@
* `CcInfo`: The C++ information about the Python libraries.
:::
+
+:::{bzl:target} toolchain_type
+
+Toolchain type identifier for the Python C toolchain.
+
+This toolchain type is typically implemented by {obj}`py_cc_toolchain`.
+:::
diff --git a/docs/api/rules_python/python/index.md b/docs/api/rules_python/python/index.md
index d1e633f..6ce5e7c 100644
--- a/docs/api/rules_python/python/index.md
+++ b/docs/api/rules_python/python/index.md
@@ -8,6 +8,14 @@
:::{bzl:target} toolchain_type
Identifier for the toolchain type for the target platform.
+
+This toolchain type gives information about the runtime for the target platform.
+It is typically implemented by the {obj}`py_runtime` rule
+
+::::{seealso}
+{any}`Custom Toolchains` for how to define custom toolchains
+::::
+
:::
:::{bzl:target} exec_tools_toolchain_type
diff --git a/docs/index.md b/docs/index.md
index 445cf20..c06c31e 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -57,7 +57,7 @@
self
getting-started
pypi-dependencies
-toolchains
+Toolchains <toolchains>
pip
coverage
precompiling
diff --git a/docs/toolchains.md b/docs/toolchains.md
index fac1bfc..b5f664f 100644
--- a/docs/toolchains.md
+++ b/docs/toolchains.md
@@ -161,9 +161,13 @@
#### Toolchain usage in other rules
-Python toolchains can be utilized in other bazel rules, such as `genrule()`, by adding the `toolchains=["@rules_python//python:current_py_toolchain"]` attribute. You can obtain the path to the Python interpreter using the `$(PYTHON2)` and `$(PYTHON3)` ["Make" Variables](https://bazel.build/reference/be/make-variables). See the
-{gh-path}`test_current_py_toolchain <tests/load_from_macro/BUILD.bazel>` target for an example.
-
+Python toolchains can be utilized in other bazel rules, such as `genrule()`, by
+adding the `toolchains=["@rules_python//python:current_py_toolchain"]`
+attribute. You can obtain the path to the Python interpreter using the
+`$(PYTHON2)` and `$(PYTHON3)` ["Make"
+Variables](https://bazel.build/reference/be/make-variables). See the
+{gh-path}`test_current_py_toolchain <tests/load_from_macro/BUILD.bazel>` target
+for an example.
## Workspace configuration
@@ -242,3 +246,193 @@
To aid migration off the Bazel-builtin toolchain, rules_python provides
{obj}`@rules_python//python/runtime_env_toolchains:all`. This is an equivalent
toolchain, but is implemented using rules_python's objects.
+
+
+## Custom toolchains
+
+While rules_python provides toolchains by default, it is not required to use
+them, and you can define your own toolchains to use instead. This section
+gives an introduction for how to define them yourself.
+
+:::{note}
+* Defining your own toolchains is an advanced feature.
+* APIs used for defining them are less stable and may change more often.
+:::
+
+Under the hood, there are multiple toolchains that comprise the different
+information necessary to build Python targets. Each one has an
+associated _toolchain type_ that identifies it. We call the collection of these
+toolchains a "toolchain suite".
+
+One of the underlying design goals of the toolchains is to support complex and
+bespoke environments. Such environments may use an arbitrary combination of
+{obj}`RBE`, cross-platform building, multiple Python versions,
+building Python from source, embeding Python (as opposed to building separate
+interpreters), using prebuilt binaries, or using binaries built from source. To
+that end, many of the attributes they accept, and fields they provide, are
+optional.
+
+### Target toolchain type
+
+The target toolchain type is {obj}`//python:toolchain_type`, and it
+is for _target configuration_ runtime information, e.g., the Python version
+and interpreter binary that a program will use.
+
+The is typically implemented using {obj}`py_runtime()`, which
+provides the {obj}`PyRuntimeInfo` provider. For historical reasons from the
+Python 2 transition, `py_runtime` is wrapped in {obj}`py_runtime_pair`,
+which provides {obj}`ToolchainInfo` with the field `py3_runtime`, which is an
+instance of `PyRuntimeInfo`.
+
+This toolchain type is intended to hold only _target configuration_ values. As
+such, when defining its associated {external:bzl:obj}`toolchain` target, only
+set {external:bzl:obj}`toolchain.target_compatible_with` and/or
+{external:bzl:obj}`toolchain.target_settings` constraints; there is no need to
+set {external:bzl:obj}`toolchain.exec_compatible_with`.
+
+### Python C toolchain type
+
+The Python C toolchain type ("py cc") is {obj}`//python/cc:toolchain_type`, and
+it has C/C++ information for the _target configuration_, e.g. the C headers that
+provide `Python.h`.
+
+This is typically implemented using {obj}`py_cc_toolchain()`, which provides
+{obj}`ToolchainInfo` with the field `py_cc_toolchain` set, which is a
+{obj}`PyCcToolchainInfo` provider instance.
+
+This toolchain type is intended to hold only _target configuration_ values
+relating to the C/C++ information for the Python runtime. As such, when defining
+its associated {external:obj}`toolchain` target, only set
+{external:bzl:obj}`toolchain.target_compatible_with` and/or
+{external:bzl:obj}`toolchain.target_settings` constraints; there is no need to
+set {external:bzl:obj}`toolchain.exec_compatible_with`.
+
+### Exec tools toolchain type
+
+The exec tools toolchain type is {obj}`//python:exec_tools_toolchain_type`,
+and it is for supporting tools for _building_ programs, e.g. the binary to
+precompile code at build time.
+
+This toolchain type is intended to hold only _exec configuration_ values --
+usually tools (prebuilt or from-source) used to build Python targets.
+
+This is typically implemented using {obj}`py_exec_tools_toolchain`, which
+provides {obj}`ToolchainInfo` with the field `exec_tools` set, which is an
+instance of {obj}`PyExecToolsInfo`.
+
+The toolchain constraints of this toolchain type can be a bit more nuanced than
+the other toolchain types. Typically, you set
+{external:bzl:obj}`toolchain.target_settings` to the Python version the tools
+are for, and {external:bzl:obj}`toolchain.exec_compatible_with` to the platform
+they can run on. This allows the toolchain to first be considered based on the
+target configuration (e.g. Python version), then for one to be chosen based on
+finding one compatible with the available host platforms to run the tool on.
+
+However, what `target_compatible_with`/`target_settings` and
+`exec_compatible_with` values to use depend on details of the tools being used.
+For example:
+* If you had a precompiler that supported any version of Python, then
+ putting the Python version in `target_settings` is unnecessary.
+* If you had a prebuilt polyglot precompiler binary that could run on any
+ platform, then setting `exec_compatible_with` is unnecessary.
+
+This can work because, when the rules invoke these build tools, they pass along
+all necessary information so that the tool can be entirely independent of the
+target configuration being built for.
+
+Alternatively, if you had a precompiler that only ran on linux, and only
+produced valid output for programs intended to run on linux, then _both_
+`exec_compatible_with` and `target_compatible_with` must be set to linux.
+
+### Custom toolchain example
+
+Here, we show an example for a semi-complicated toolchain suite, one that is:
+
+* A CPython-based interpreter
+* For Python version 3.12.0
+* Using an in-build interpreter built from source
+* That only runs on Linux
+* Using a prebuilt precompiler that only runs on Linux, and only produces byte
+ code valid for 3.12
+* With the exec tools interpreter disabled (unnecessary with a prebuild
+ precompiler)
+* Providing C headers and libraries
+
+Defining toolchains for this might look something like this:
+
+```
+# File: toolchain_impls/BUILD
+load("@rules_python//python:py_cc_toolchain.bzl", "py_cc_toolchain")
+load("@rules_python//python:py_exec_tools_toolchain.bzl", "py_exec_tools_toolchain")
+load("@rules_python//python:py_runtime.bzl", "py_runtime")
+load("@rules_python//python:py_runtime_pair.bzl", "py_runtime_pair")
+
+MAJOR = 3
+MINOR = 12
+MICRO = 0
+
+py_runtime(
+ name = "runtime",
+ interpreter = ":python",
+ interpreter_version_info = {
+ "major": str(MAJOR),
+ "minor": str(MINOR),
+ "micro": str(MICRO),
+ }
+ implementation = "cpython"
+)
+py_runtime_pair(
+ name = "runtime_pair",
+ py3_runtime = ":runtime"
+)
+
+py_cc_toolchain(
+ name = "py_cc_toolchain_impl",
+ headers = ":headers",
+ libs = ":libs",
+ python_version = "{}.{}".format(MAJOR, MINOR)
+)
+
+py_exec_tools_toolchain(
+ name = "exec_tools_toolchain_impl",
+ exec_interpreter = "@rules_python/python:null_target",
+ precompiler = "precompiler-cpython-3.12"
+)
+
+cc_binary(name = "python3.12", ...)
+cc_library(name = "headers", ...)
+cc_library(name = "libs", ...)
+
+# File: toolchains/BUILD
+# Putting toolchain() calls in a separate package from the toolchain
+# implementations minimizes Bazel loading overhead
+
+toolchain(
+ name = "runtime_toolchain",
+ toolchain = "//toolchain_impl:runtime_pair",
+ toolchain_type = "@rules_python//python:toolchain_type",
+ target_compatible_with = ["@platforms/os:linux"]
+)
+toolchain(
+ name = "py_cc_toolchain",
+ toolchain = "//toolchain_impl:py_cc_toolchain_impl",
+ toolchain_type = "@rules_python//python/cc:toolchain_type",
+ target_compatible_with = ["@platforms/os:linux"]
+)
+
+toolchain(
+ name = "exec_tools_toolchain",
+ toolchain = "//toolchain_impl:exec_tools_toolchain_impl",
+ toolchain_type = "@rules_python//python:exec_tools_toolchain_type",
+ target_settings = [
+ "@rules_python//python/config_settings:is_python_3.12",
+ ],
+ exec_comaptible_with = ["@platforms/os:linux"]
+)
+```
+
+:::{note}
+The toolchain() calls should be in a separate BUILD file from everything else.
+This avoids Bazel having to perform unnecessary work when it discovers the list
+of available toolchains.
+:::
diff --git a/sphinxdocs/docs/sphinx-bzl.md b/sphinxdocs/docs/sphinx-bzl.md
index bb51b6c..73ae138 100644
--- a/sphinxdocs/docs/sphinx-bzl.md
+++ b/sphinxdocs/docs/sphinx-bzl.md
@@ -90,7 +90,7 @@
{obj}`the binary rule <py_binary>`
```
-Finally, specific types of objects (rules, functions, providers, etc) can be
+Specific types of objects (rules, functions, providers, etc) can be
specified to help disambiguate short names:
```
@@ -98,12 +98,21 @@
{rule}`py_binary` # Refers to the underlying rule
```
+Finally, objects built into Bazel can be explicitly referenced by forcing
+a lookup outside the local project using `{external}`. For example, the symbol
+`toolchain` is a builtin Bazel function, but it could also be the name of a tag
+class in the local project. To force looking up the builtin Bazel `toolchain` rule,
+`{external:bzl:rule}` can be used, e.g.:
+
+```
+{external:bzl:obj}`toolchain`
+```
+
Those are the basics of cross referencing. Sphinx has several additional
syntaxes for finding and referencing objects; see
[the MyST docs for supported
syntaxes](https://myst-parser.readthedocs.io/en/latest/syntax/cross-referencing.html#reference-roles)
-
### Cross reference roles
A cross reference role is the `obj` portion of `{bzl:obj}`. It affects what is
diff --git a/sphinxdocs/inventories/bazel_inventory.txt b/sphinxdocs/inventories/bazel_inventory.txt
index ee507d1..fd2bac7 100644
--- a/sphinxdocs/inventories/bazel_inventory.txt
+++ b/sphinxdocs/inventories/bazel_inventory.txt
@@ -9,6 +9,7 @@
File bzl:type 1 rules/lib/File -
Label bzl:type 1 rules/lib/Label -
Name bzl:type 1 concepts/labels#target-names -
+RBE bzl:obj 1 remote/rbe -
RunEnvironmentInfo bzl:type 1 rules/lib/providers/RunEnvironmentInfo -
Target bzl:type 1 rules/lib/builtins/Target -
ToolchainInfo bzl:type 1 rules/lib/providers/ToolchainInfo.html -
@@ -58,6 +59,7 @@
ctx.workspace_name bzl:obj 1 rules/lib/builtins/ctx#workspace_name -
depset bzl:type 1 rules/lib/depset -
dict bzl:type 1 rules/lib/dict -
+exec_compatible_with bzl:attribute 1 reference/be/common-definitions#common.exec_compatible_with -
int bzl:type 1 rules/lib/int -
label bzl:type 1 concepts/labels -
list bzl:type 1 rules/lib/list -
@@ -131,8 +133,13 @@
runfiles.symlinks bzl:type 1 rules/lib/builtins/runfiles#symlinks -
str bzl:type 1 rules/lib/string -
struct bzl:type 1 rules/lib/builtins/struct -
+target_compatible_with bzl:attribute 1 reference/be/common-definitions#common.target_compatible_with -
testing bzl:obj 1 rules/lib/toplevel/testing -
testing.ExecutionInfo bzl:function 1 rules/lib/toplevel/testing#ExecutionInfo -
testing.TestEnvironment bzl:function 1 rules/lib/toplevel/testing#TestEnvironment -
testing.analysis_test bzl:rule 1 rules/lib/toplevel/testing#analysis_test -
-toolchain_type bzl:type 1 ules/lib/builtins/toolchain_type.html -
+toolchain bzl:rule 1 reference/be/platforms-and-toolchains#toolchain -
+toolchain.exec_compatible_with bzl:rule 1 reference/be/platforms-and-toolchains#toolchain.exec_compatible_with -
+toolchain.target_settings bzl:attribute 1 reference/be/platforms-and-toolchains#toolchain.target_settings -
+toolchain.target_compatible_with bzl:attribute 1 reference/be/platforms-and-toolchains#toolchain.target_compatible_with -
+toolchain_type bzl:type 1 rules/lib/builtins/toolchain_type.html -
diff --git a/sphinxdocs/src/sphinx_bzl/bzl.py b/sphinxdocs/src/sphinx_bzl/bzl.py
index 2980ecb..cbd35a9 100644
--- a/sphinxdocs/src/sphinx_bzl/bzl.py
+++ b/sphinxdocs/src/sphinx_bzl/bzl.py
@@ -34,7 +34,7 @@
from sphinx.util import inspect, logging
from sphinx.util import nodes as sphinx_nodes
from sphinx.util import typing as sphinx_typing
-from typing_extensions import override, TypeAlias
+from typing_extensions import TypeAlias, override
_logger = logging.getLogger(__name__)
_LOG_PREFIX = f"[{_logger.name}] "
@@ -552,7 +552,9 @@
@override
def transform_content(self, content_node: addnodes.desc_content) -> None:
- def first_child_with_class_name(root, class_name) -> typing.Union[None, docutils_nodes.Element]:
+ def first_child_with_class_name(
+ root, class_name
+ ) -> typing.Union[None, docutils_nodes.Element]:
matches = root.findall(
lambda node: isinstance(node, docutils_nodes.Element)
and class_name in node["classes"]
@@ -1437,7 +1439,9 @@
object_types = {
"arg": domains.ObjType("arg", "arg", "obj"), # macro/function arg
"aspect": domains.ObjType("aspect", "aspect", "obj"),
- "attribute": domains.ObjType("attribute", "attribute", "obj"), # rule attribute
+ "attribute": domains.ObjType(
+ "attribute", "attribute", "attr", "obj"
+ ), # rule attribute
"function": domains.ObjType("function", "func", "obj"),
"method": domains.ObjType("method", "method", "obj"),
"module-extension": domains.ObjType(
@@ -1509,7 +1513,9 @@
}
@override
- def get_full_qualified_name(self, node: docutils_nodes.Element) -> typing.Union[str, None]:
+ def get_full_qualified_name(
+ self, node: docutils_nodes.Element
+ ) -> typing.Union[str, None]:
bzl_file = node.get("bzl:file")
symbol_name = node.get("bzl:symbol")
ref_target = node.get("reftarget")
@@ -1574,6 +1580,10 @@
if target.startswith("--"):
target = target.strip("-")
object_type = "flag"
+
+ # Allow using parentheses, e.g. `foo()` or `foo(x=...)`
+ target, _, _ = target.partition("(")
+
# Elide the value part of --foo=bar flags
# Note that the flag value could contain `=`
if "=" in target:
