| commit | 4fb634ebb9d07d50fd708d007cbf75d59e8094cc | [log] [tgz] |
|---|---|---|
| author | Toby Harradine <tobyh@canva.com> | Mon Nov 10 11:43:36 2025 +1100 |
| committer | GitHub <noreply@github.com> | Mon Nov 10 00:43:36 2025 +0000 |
| tree | b19bf980008120db115f826b77af1d80ee9dd998 | |
| parent | c037d83aaad33808abc5f1cce5a797ca1550cf0c [diff] |
refactor: defer zip manifest building to execution phase to improve analysis phase performance (#3381) When py_binary/py_test were being built, they were flattening the runfiles depsets at analysis time in order to create the zip file mapping manifest for their implicit zipapp outputs. This flattening was necessary because they had to filter out the original main executable from the runfiles that didn't belong in the zipapp. This flattening is expensive for large builds, in some cases adding over 400 seconds of time and significant memory overhead. To fix, have the zip file manifest use the `runfiles_with_exe` object, which is the runfiles, but pre-filtered for the files zip building doesn't want. This then allows passing the depsets directly to `Args.add_all` and using map_each to transform them. Additionally, pass `runfiles.empty_filenames` using a lambda. Accessing that attribute implicitly flattens the runfiles. Finally, because the original profiles indicated `str.format()` was a non-trivial amount of time (46 seconds / 15% of build time), switch to using `+` instead. This is a more incremental alternative to #3380 which achieves _most_ of the same optimization with only Starlark changes, as opposed to introducing an external script written in C++. [Profile of a large build](https://github.com/user-attachments/assets/e90ae699-a04d-44df-b53c-1156aa890af5), which shows a Starlark CPU profile. It shows an overall build time of 305 seconds. 46 seconds (15%) are spent in `map_zip_runfiles`, half of which is in `str.startswith()` and the other half in `str.format()`. --------- Co-authored-by: Richard Levasseur <rlevasseur@google.com>
This repository is the home of the core Python rules -- py_library, py_binary, py_test, py_proto_library, and related symbols that provide the basis for Python support in Bazel. It also contains package installation rules for integrating with PyPI and other indices.
Documentation for rules_python is at https://rules-python.readthedocs.io and in the Bazel Build Encyclopedia.
Examples live in the examples directory.
The core rules are stable. Their implementation is subject to Bazel's backward compatibility policy. This repository aims to follow semantic versioning.
The Bazel community maintains this repository. Neither Google nor the Bazel team provides support for the code. However, this repository is part of the test suite used to vet new Bazel releases. See How to contribute page for information on our development workflow.
For detailed documentation, see https://rules-python.readthedocs.io
See Bzlmod support for more details.