fix(precompiling)!: make binary-level precompile opt-in/opt-opt work (#2243)
This makes binary-level opt-in/opt-out of precompiling work as intended.
Previously,
when `pyc_collection=include_pyc` was set on a binary, only transitive
libraries that
had explicitly enabled precompiling were being included (which was moot
anyways --
libraries put their files in runfiles, so no matter what, their files
were included).
The intent was that, when a binary set `pyc_collection=include_pyc`,
then precompiled
files would be used for all its transitive dependencies (unless they
had, at the
target-level, disabled precompiling). Conversely, if
`pyc_collection=disabled` was set,
the precompiled files would not be used (unless a target had, at the
target level,
enabled precompiling).
To make it work as desired, the basic fix is to make it so that
libraries have a place to
put the implicit pyc files (the ones automatically generated), and have
the binaries
include those when requested. The net effect is a library has 4 sets of
files it produces:
* required py files: py source files that should always go into the
binary's runfiles
* required pyc files: precompiled pyc files that should always go into
the binary's
runfiles (e.g., when a library sets `precompile=enabled` directly).
* implicit pyc files: precompiled pyc files for a library that are
always generated, but
it's up to the binary if they go into the runfiles
* implicit pyc source files: the source py file for an implicit pyc
file. When a binary
*doesn't* include the implicit pyc file, it must include the source py
file (otherwise
none of the library's code ends up included).
Similarly, in order to allow a binary to decide what files are used,
libraries must
stop putting the py/pyc files into runfiles themselves. While this is
potentially
a breaking change, I found that, within Google, there was no reliance on
this behavior,
so should be safe enough. That said, I added `--add_srcs_to_runfiles` to
restore
the previous behavior to aid in transitioning.
**BREAKING CHANGES**
1. `py_library` no longer puts its srcs into runfiles directly.
2. Removed `--precompile_add_to_runfiles`
3. Removed `--pyc_collection`
4. `precompile=if_generated_source` removed
5. `precompile_source_retention=omit_if_generated_source` removed
Though 2 through 5 are technically breaking changes, I don't think
precompiling
was very usable anyways, so usages of those flags/values is rare.
Fixes https://github.com/bazelbuild/rules_python/issues/2212
diff --git a/docs/precompiling.md b/docs/precompiling.md
index 52678e6..6eadc40 100644
--- a/docs/precompiling.md
+++ b/docs/precompiling.md
@@ -20,24 +20,24 @@
## Binary-level opt-in
-Because of the costs of precompiling, it may not be feasible to globally enable it
-for your repo for everything. For example, some binaries may be
-particularly large, and doubling the number of runfiles isn't doable.
+Binary-level opt-in allows enabling precompiling on a per-target basic. This is
+useful for situations such as:
-If this is the case, there's an alternative way to more selectively and
-incrementally control precompiling on a per-binry basis.
+* Globally enabling precompiling in your `.bazelrc` isn't feasible. This may
+ be because some targets don't work with precompiling, e.g. because they're too
+ big.
+* Enabling precompiling for build tools (exec config targets) separately from
+ target-config programs.
-To use this approach, the two basic steps are:
-1. Disable pyc files from being automatically added to runfiles:
- {bzl:obj}`--@rules_python//python/config_settings:precompile_add_to_runfiles=decided_elsewhere`,
-2. Set the `pyc_collection` attribute on the binaries/tests that should or should
- not use precompiling.
+To use this approach, set the {bzl:attr}`pyc_collection` attribute on the
+binaries/tests that should or should not use precompiling. Then change the
+{bzl:flag}`--precompile` default.
-The default for the `pyc_collection` attribute is controlled by the flag
-{bzl:obj}`--@rules_python//python/config_settings:pyc_collection`, so you
+The default for the {bzl:attr}`pyc_collection` attribute is controlled by the flag
+{bzl:obj}`--@rules_python//python/config_settings:precompile`, so you
can use an opt-in or opt-out approach by setting its value:
-* targets must opt-out: `--@rules_python//python/config_settings:pyc_collection=include_pyc`
-* targets must opt-in: `--@rules_python//python/config_settings:pyc_collection=disabled`
+* targets must opt-out: `--@rules_python//python/config_settings:precompile=enabled`
+* targets must opt-in: `--@rules_python//python/config_settings:precompile=disabled`
## Advanced precompiler customization
@@ -48,7 +48,7 @@
mechanisms are available:
* The exec tools toolchain allows customizing the precompiler binary used with
- the `precompiler` attribute. Arbitrary binaries are supported.
+ the {bzl:attr}`precompiler` attribute. Arbitrary binaries are supported.
* The execution requirements can be customized using
`--@rules_python//tools/precompiler:execution_requirements`. This is a list
flag that can be repeated. Each entry is a key=value that is added to the
@@ -92,3 +92,9 @@
`foo.cpython-39.opt-2.pyc`). This works fine (it's all byte code), but also
means the interpreter `-O` argument can't be used -- doing so will cause the
interpreter to look for the non-existent `opt-N` named files.
+* Targets with the same source files and different exec properites will result
+ in action conflicts. This most commonly occurs when a `py_binary` and
+ `py_library` have the same source files. To fix, modify both targets so
+ they have the same exec properties. If this is difficult because unsupported
+ exec groups end up being passed to the Python rules, please file an issue
+ to have those exec groups added to the Python rules.
