feat(rules): allow deriving custom rules from core rules (#2666)
This exposes public functions for creating builders for py_binary,
py_test, and py_library.
It also adds some docs and examples for how to use them.
I'm calling this a "volatile" API -- it's public, but the pieces that
comprise
it (e.g. all the rule args, attributes, the attribute args, etc) are
likely to change
in various ways, and not all modifications to them can be supported in a
backward
compatible way. Hence the "volatile" term:
* hold it gently and its fine
* shake it a bit and its probably fine
* shake it moderately and something may or may not blow up
* shake it a lot and something will certainly blow up.
Work towards https://github.com/bazelbuild/rules_python/issues/1647
---------
Co-authored-by: Ignas Anikevicius <240938+aignas@users.noreply.github.com>
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9029794..c5bf986 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -91,6 +91,9 @@
* (pypi) Direct HTTP urls for wheels and sdists are now supported when using
{obj}`experimental_index_url` (bazel downloader).
Partially fixes [#2363](https://github.com/bazelbuild/rules_python/issues/2363).
+* (rules) APIs for creating custom rules based on the core py_binary, py_test,
+ and py_library rules
+ ([#1647](https://github.com/bazelbuild/rules_python/issues/1647))
{#v0-0-0-removed}
### Removed
diff --git a/docs/BUILD.bazel b/docs/BUILD.bazel
index e19c221..09de21b 100644
--- a/docs/BUILD.bazel
+++ b/docs/BUILD.bazel
@@ -100,6 +100,8 @@
"//python:py_test_bzl",
"//python:repositories_bzl",
"//python/api:api_bzl",
+ "//python/api:executables_bzl",
+ "//python/api:libraries_bzl",
"//python/cc:py_cc_toolchain_bzl",
"//python/cc:py_cc_toolchain_info_bzl",
"//python/entry_points:py_console_script_binary_bzl",
diff --git a/docs/_includes/volatile_api.md b/docs/_includes/volatile_api.md
new file mode 100644
index 0000000..b79f5f7
--- /dev/null
+++ b/docs/_includes/volatile_api.md
@@ -0,0 +1,5 @@
+:::{important}
+
+**Public, but volatile, API.** Some parts are stable, while others are
+implementation details and may change more frequently.
+:::
diff --git a/docs/extending.md b/docs/extending.md
new file mode 100644
index 0000000..dbd63e5
--- /dev/null
+++ b/docs/extending.md
@@ -0,0 +1,143 @@
+# Extending the rules
+
+:::{important}
+**This is public, but volatile, functionality.**
+
+Extending and customizing the rules is supported functionality, but with weaker
+backwards compatibility guarantees, and is not fully subject to the normal
+backwards compatibility procedures and policies. It's simply not feasible to
+support every possible customization with strong backwards compatibility
+guarantees.
+:::
+
+Because of the rich ecosystem of tools and variety of use cases, APIs are
+provided to make it easy to create custom rules using the existing rules as a
+basis. This allows implementing behaviors that aren't possible using
+wrapper macros around the core rules, and can make certain types of changes
+much easier and transparent to implement.
+
+:::{note}
+It is not required to extend a core rule. The minimum requirement for a custom
+rule is to return the appropriate provider (e.g. {bzl:obj}`PyInfo` etc).
+Extending the core rules is most useful when you want all or most of the
+behavior of a core rule.
+:::
+
+Follow or comment on https://github.com/bazelbuild/rules_python/issues/1647
+for the development of APIs to support custom derived rules.
+
+## Creating custom rules
+
+Custom rules can be created using the core rules as a basis by using their rule
+builder APIs.
+
+* [`//python/apis:executables.bzl`](#python-apis-executables-bzl): builders for
+ executables.
+* [`//python/apis:libraries.bzl`](#python-apis-libraries-bzl): builders for
+ libraries.
+
+These builders create {bzl:obj}`ruleb.Rule` objects, which are thin
+wrappers around the keyword arguments eventually passed to the `rule()`
+function. These builder APIs give access to the _entire_ rule definition and
+allow arbitrary modifications.
+
+This is level of control is powerful, but also volatile. A rule definition
+contains many details that _must_ change as the implementation changes. What
+is more or less likely to change isn't known in advance, but some general
+rules are:
+
+* Additive behavior to public attributes will be less prone to breaking.
+* Internal attributes that directly support a public attribute are likely
+ reliable.
+* Internal attributes that support an action are more likely to change.
+* Rule toolchains are moderately stable (toolchains are mostly internal to
+ how a rule works, but custom toolchains are supported).
+
+## Example: validating a source file
+
+In this example, we derive from `py_library` a custom rule that verifies source
+code contains the word "snakes". It does this by:
+
+* Adding an implicit dependency on a checker program
+* Calling the base implementation function
+* Running the checker on the srcs files
+* Adding the result to the `_validation` output group (a special output
+ group for validation behaviors).
+
+To users, they can use `has_snakes_library` the same as `py_library`. The same
+is true for other targets that might consume the rule.
+
+```
+load("@rules_python//python/api:libraries.bzl", "libraries")
+load("@rules_python//python/api:attr_builders.bzl", "attrb")
+
+def _has_snakes_impl(ctx, base):
+ providers = base(ctx)
+
+ out = ctx.actions.declare_file(ctx.label.name + "_snakes.check")
+ ctx.actions.run(
+ inputs = ctx.files.srcs,
+ outputs = [out],
+ executable = ctx.attr._checker[DefaultInfo].files_to_run,
+ args = [out.path] + [f.path for f in ctx.files.srcs],
+ )
+ prior_ogi = None
+ for i, p in enumerate(providers):
+ if type(p) == "OutputGroupInfo":
+ prior_ogi = (i, p)
+ break
+ if prior_ogi:
+ groups = {k: getattr(prior_ogi[1], k) for k in dir(prior_ogi)}
+ if "_validation" in groups:
+ groups["_validation"] = depset([out], transitive=groups["_validation"])
+ else:
+ groups["_validation"] = depset([out])
+ providers[prior_ogi[0]] = OutputGroupInfo(**groups)
+ else:
+ providers.append(OutputGroupInfo(_validation=depset([out])))
+ return providers
+
+def create_has_snakes_rule():
+ r = libraries.py_library_builder()
+ base_impl = r.implementation()
+ r.set_implementation(lambda ctx: _has_snakes_impl(ctx, base_impl))
+ r.attrs["_checker"] = attrb.Label(
+ default="//:checker",
+ executable = True,
+ )
+ return r.build()
+has_snakes_library = create_has_snakes_rule()
+```
+
+## Example: adding transitions
+
+In this example, we derive from `py_binary` to force building for a particular
+platform. We do this by:
+
+* Adding an additional output to the rule's cfg
+* Calling the base transition function
+* Returning the new transition outputs
+
+```starlark
+
+load("@rules_python//python/api:executables.bzl", "executables")
+
+def _force_linux_impl(settings, attr, base_impl):
+ settings = base_impl(settings, attr)
+ settings["//command_line_option:platforms"] = ["//my/platforms:linux"]
+ return settings
+
+def create_rule():
+ r = executables.py_binary_rule_builder()
+ base_impl = r.cfg.implementation()
+ r.cfg.set_implementation(
+ lambda settings, attr: _force_linux_impl(settings, attr, base_impl)
+ )
+ r.cfg.add_output("//command_line_option:platforms")
+ return r.build()
+
+py_linux_binary = create_linux_binary_rule()
+```
+
+Users can then use `py_linux_binary` the same as a regular py_binary. It will
+act as if `--platforms=//my/platforms:linux` was specified when building it.
diff --git a/docs/index.md b/docs/index.md
index dd2e147..04a7688 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -101,6 +101,7 @@
coverage
precompiling
gazelle
+Extending <extending>
Contributing <contributing>
support
Changelog <changelog>
diff --git a/python/api/BUILD.bazel b/python/api/BUILD.bazel
index 1df6877..f0e0494 100644
--- a/python/api/BUILD.bazel
+++ b/python/api/BUILD.bazel
@@ -25,6 +25,26 @@
deps = ["//python/private/api:api_bzl"],
)
+bzl_library(
+ name = "executables_bzl",
+ srcs = ["executables.bzl"],
+ visibility = ["//visibility:public"],
+ deps = [
+ "//python/private:py_binary_rule_bzl",
+ "//python/private:py_executable_bzl",
+ "//python/private:py_test_rule_bzl",
+ ],
+)
+
+bzl_library(
+ name = "libraries_bzl",
+ srcs = ["libraries.bzl"],
+ visibility = ["//visibility:public"],
+ deps = [
+ "//python/private:py_library_bzl",
+ ],
+)
+
filegroup(
name = "distribution",
srcs = glob(["**"]),
diff --git a/python/api/executables.bzl b/python/api/executables.bzl
new file mode 100644
index 0000000..4715c0f
--- /dev/null
+++ b/python/api/executables.bzl
@@ -0,0 +1,31 @@
+# Copyright 2025 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+{#python-apis-executables-bzl}
+Loading-phase APIs specific to executables (binaries/tests).
+
+:::{versionadded} VERSION_NEXT_FEATURE
+:::
+"""
+
+load("//python/private:py_binary_rule.bzl", "create_py_binary_rule_builder")
+load("//python/private:py_executable.bzl", "create_executable_rule_builder")
+load("//python/private:py_test_rule.bzl", "create_py_test_rule_builder")
+
+executables = struct(
+ py_binary_rule_builder = create_py_binary_rule_builder,
+ py_test_rule_builder = create_py_test_rule_builder,
+ executable_rule_builder = create_executable_rule_builder,
+)
diff --git a/python/api/libraries.bzl b/python/api/libraries.bzl
new file mode 100644
index 0000000..c4ad598
--- /dev/null
+++ b/python/api/libraries.bzl
@@ -0,0 +1,27 @@
+# Copyright 2025 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+{#python-apis-libraries-bzl}
+Loading-phase APIs specific to libraries.
+
+:::{versionadded} VERSION_NEXT_FEATURE
+:::
+"""
+
+load("//python/private:py_library.bzl", "create_py_library_rule_builder")
+
+libraries = struct(
+ py_library_rule_builder = create_py_library_rule_builder,
+)
diff --git a/python/private/BUILD.bazel b/python/private/BUILD.bazel
index b7e52a3..8b07fbd 100644
--- a/python/private/BUILD.bazel
+++ b/python/private/BUILD.bazel
@@ -427,6 +427,7 @@
":attributes_bzl",
":common_bzl",
":flags_bzl",
+ ":precompile_bzl",
":py_cc_link_params_info_bzl",
":py_internal_bzl",
":rule_builders_bzl",
@@ -446,8 +447,6 @@
name = "py_library_rule_bzl",
srcs = ["py_library_rule.bzl"],
deps = [
- ":common_bzl",
- ":precompile_bzl",
":py_library_bzl",
],
)
diff --git a/python/private/attr_builders.bzl b/python/private/attr_builders.bzl
index acd1d40..efcbfa6 100644
--- a/python/private/attr_builders.bzl
+++ b/python/private/attr_builders.bzl
@@ -12,7 +12,11 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-"""Builders for creating attributes et al."""
+"""Builders for creating attributes et al.
+
+:::{versionadded} VERSION_NEXT_FEATURE
+:::
+"""
load("@bazel_skylib//lib:types.bzl", "types")
load(
diff --git a/python/private/py_binary_rule.bzl b/python/private/py_binary_rule.bzl
index 0e1912c..38e3a69 100644
--- a/python/private/py_binary_rule.bzl
+++ b/python/private/py_binary_rule.bzl
@@ -27,7 +27,20 @@
inherited_environment = [],
)
-def create_binary_rule_builder():
+# NOTE: Exported publicly
+def create_py_binary_rule_builder():
+ """Create a rule builder for a py_binary.
+
+ :::{include} /_includes/volatile_api.md
+ :::
+
+ :::{versionadded} VERSION_NEXT_FEATURE
+ :::
+
+ Returns:
+ {type}`ruleb.Rule` with the necessary settings
+ for creating a `py_binary` rule.
+ """
builder = create_executable_rule_builder(
implementation = _py_binary_impl,
executable = True,
@@ -35,4 +48,4 @@
builder.attrs.update(AGNOSTIC_BINARY_ATTRS)
return builder
-py_binary = create_binary_rule_builder().build()
+py_binary = create_py_binary_rule_builder().build()
diff --git a/python/private/py_executable.bzl b/python/private/py_executable.bzl
index f85f242..bcbff70 100644
--- a/python/private/py_executable.bzl
+++ b/python/private/py_executable.bzl
@@ -1737,7 +1737,24 @@
"""
return create_executable_rule_builder().build()
+# NOTE: Exported publicly
def create_executable_rule_builder(implementation, **kwargs):
+ """Create a rule builder for an executable Python program.
+
+ :::{include} /_includes/volatile_api.md
+ :::
+
+ An executable rule is one that sets either `executable=True` or `test=True`,
+ and the output is something that can be run directly (e.g. `bazel run`,
+ `exec(...)` etc)
+
+ :::{versionadded} VERSION_NEXT_FEATURE
+ :::
+
+ Returns:
+ {type}`ruleb.Rule` with the necessary settings
+ for creating an executable Python rule.
+ """
builder = ruleb.Rule(
implementation = implementation,
attrs = EXECUTABLE_ATTRS,
diff --git a/python/private/py_library.bzl b/python/private/py_library.bzl
index a774104..7b024a0 100644
--- a/python/private/py_library.bzl
+++ b/python/private/py_library.bzl
@@ -25,16 +25,9 @@
"REQUIRED_EXEC_GROUP_BUILDERS",
)
load(":builders.bzl", "builders")
-load(
- ":common.bzl",
- "collect_imports",
- "collect_runfiles",
- "create_instrumented_files_info",
- "create_output_group_info",
- "create_py_info",
- "filter_to_py_srcs",
-)
+load(":common.bzl", "collect_cc_info", "collect_imports", "collect_runfiles", "create_instrumented_files_info", "create_library_semantics_struct", "create_output_group_info", "create_py_info", "filter_to_py_srcs", "get_imports")
load(":flags.bzl", "AddSrcsToRunfilesFlag", "PrecompileFlag")
+load(":precompile.bzl", "maybe_precompile")
load(":py_cc_link_params_info.bzl", "PyCcLinkParamsInfo")
load(":py_internal.bzl", "py_internal")
load(":rule_builders.bzl", "ruleb")
@@ -57,6 +50,16 @@
},
)
+def _py_library_impl_with_semantics(ctx):
+ return py_library_impl(
+ ctx,
+ semantics = create_library_semantics_struct(
+ get_imports = get_imports,
+ maybe_precompile = maybe_precompile,
+ get_cc_info_for_library = collect_cc_info,
+ ),
+ )
+
def py_library_impl(ctx, *, semantics):
"""Abstract implementation of py_library rule.
@@ -141,32 +144,29 @@
:::
"""
-def create_py_library_rule_builder(*, attrs = {}, **kwargs):
- """Creates a py_library rule.
+# NOTE: Exported publicaly
+def create_py_library_rule_builder():
+ """Create a rule builder for a py_library.
- Args:
- attrs: dict of rule attributes.
- **kwargs: Additional kwargs to pass onto {obj}`ruleb.Rule()`.
+ :::{include} /_includes/volatile_api.md
+ :::
+
+ :::{versionadded} VERSION_NEXT_FEATURE
+ :::
Returns:
- {type}`ruleb.Rule` builder object.
+ {type}`ruleb.Rule` with the necessary settings
+ for creating a `py_library` rule.
"""
-
- # Within Google, the doc attribute is overridden
- kwargs.setdefault("doc", _DEFAULT_PY_LIBRARY_DOC)
-
- # TODO: b/253818097 - fragments=py is only necessary so that
- # RequiredConfigFragmentsTest passes
- fragments = kwargs.pop("fragments", None) or []
- kwargs["exec_groups"] = REQUIRED_EXEC_GROUP_BUILDERS | (kwargs.get("exec_groups") or {})
-
builder = ruleb.Rule(
- attrs = dicts.add(LIBRARY_ATTRS, attrs),
- fragments = fragments + ["py"],
+ implementation = _py_library_impl_with_semantics,
+ doc = _DEFAULT_PY_LIBRARY_DOC,
+ exec_groups = dict(REQUIRED_EXEC_GROUP_BUILDERS),
+ attrs = LIBRARY_ATTRS,
+ fragments = ["py"],
toolchains = [
ruleb.ToolchainType(TOOLCHAIN_TYPE, mandatory = False),
ruleb.ToolchainType(EXEC_TOOLS_TOOLCHAIN_TYPE, mandatory = False),
],
- **kwargs
)
return builder
diff --git a/python/private/py_library_rule.bzl b/python/private/py_library_rule.bzl
index 44382a7..ac256bc 100644
--- a/python/private/py_library_rule.bzl
+++ b/python/private/py_library_rule.bzl
@@ -13,20 +13,6 @@
# limitations under the License.
"""Implementation of py_library rule."""
-load(":common.bzl", "collect_cc_info", "create_library_semantics_struct", "get_imports")
-load(":precompile.bzl", "maybe_precompile")
-load(":py_library.bzl", "create_py_library_rule_builder", "py_library_impl")
+load(":py_library.bzl", "create_py_library_rule_builder")
-def _py_library_impl_with_semantics(ctx):
- return py_library_impl(
- ctx,
- semantics = create_library_semantics_struct(
- get_imports = get_imports,
- maybe_precompile = maybe_precompile,
- get_cc_info_for_library = collect_cc_info,
- ),
- )
-
-py_library = create_py_library_rule_builder(
- implementation = _py_library_impl_with_semantics,
-).build()
+py_library = create_py_library_rule_builder().build()
diff --git a/python/private/py_test_rule.bzl b/python/private/py_test_rule.bzl
index 72e8bab..f21fdc7 100644
--- a/python/private/py_test_rule.bzl
+++ b/python/private/py_test_rule.bzl
@@ -30,7 +30,20 @@
maybe_add_test_execution_info(providers, ctx)
return providers
-def create_test_rule_builder():
+# NOTE: Exported publicaly
+def create_py_test_rule_builder():
+ """Create a rule builder for a py_test.
+
+ :::{include} /_includes/volatile_api.md
+ :::
+
+ :::{versionadded} VERSION_NEXT_FEATURE
+ :::
+
+ Returns:
+ {type}`ruleb.Rule` with the necessary settings
+ for creating a `py_test` rule.
+ """
builder = create_executable_rule_builder(
implementation = _py_test_impl,
test = True,
@@ -38,4 +51,4 @@
builder.attrs.update(AGNOSTIC_TEST_ATTRS)
return builder
-py_test = create_test_rule_builder().build()
+py_test = create_py_test_rule_builder().build()
diff --git a/python/private/rule_builders.bzl b/python/private/rule_builders.bzl
index 6d9fb3f..4607285 100644
--- a/python/private/rule_builders.bzl
+++ b/python/private/rule_builders.bzl
@@ -91,6 +91,9 @@
custom_foo_binary = create_custom_foo_binary()
```
+
+:::{versionadded} VERSION_NEXT_FEATURE
+:::
"""
load("@bazel_skylib//lib:types.bzl", "types")
diff --git a/tests/support/sh_py_run_test.bzl b/tests/support/sh_py_run_test.bzl
index d1e3b8e..7b3b617 100644
--- a/tests/support/sh_py_run_test.bzl
+++ b/tests/support/sh_py_run_test.bzl
@@ -20,9 +20,9 @@
load("@rules_shell//shell:sh_test.bzl", "sh_test")
load("//python/private:attr_builders.bzl", "attrb") # buildifier: disable=bzl-visibility
load("//python/private:py_binary_macro.bzl", "py_binary_macro") # buildifier: disable=bzl-visibility
-load("//python/private:py_binary_rule.bzl", "create_binary_rule_builder") # buildifier: disable=bzl-visibility
+load("//python/private:py_binary_rule.bzl", "create_py_binary_rule_builder") # buildifier: disable=bzl-visibility
load("//python/private:py_test_macro.bzl", "py_test_macro") # buildifier: disable=bzl-visibility
-load("//python/private:py_test_rule.bzl", "create_test_rule_builder") # buildifier: disable=bzl-visibility
+load("//python/private:py_test_rule.bzl", "create_py_test_rule_builder") # buildifier: disable=bzl-visibility
load("//python/private:toolchain_types.bzl", "TARGET_TOOLCHAIN_TYPE") # buildifier: disable=bzl-visibility
load("//tests/support:support.bzl", "VISIBLE_FOR_TESTING")
@@ -79,9 +79,9 @@
builder.cfg.update_outputs(_RECONFIG_OUTPUTS)
return builder.build()
-_py_reconfig_binary = _create_reconfig_rule(create_binary_rule_builder())
+_py_reconfig_binary = _create_reconfig_rule(create_py_binary_rule_builder())
-_py_reconfig_test = _create_reconfig_rule(create_test_rule_builder())
+_py_reconfig_test = _create_reconfig_rule(create_py_test_rule_builder())
def py_reconfig_test(**kwargs):
"""Create a py_test with customized build settings for testing.
