refactor(zipapp): port zipper to Rust and add it to the shrubbery toolchain (#4407) Replaces the Python zipper used by py_zipapp's PyZipAppCreateZip action with a Rust implementation, so creating zipapps no longer requires a Python interpreter in the exec configuration. The crates/zipper library mirrors tools/zipapp/zipper.py: same manifest format, symlink-precedence sorting, sandboxed source symlink detection, fixed 1980-01-01 timestamps, and permission preservation. It is exposed as the `zipper` shrubbery subcommand and selected via get_shrubbery_tool, falling back to the Python tool when shrubbery is unavailable. The `zip` crate is pinned to 8.x: 9.x adds target-conditional OS deps that fail on powerpc64le until rules_rust supports that platform (see the NOTE in MODULE.bazel). Error handling uses `anyhow` so failures name the manifest line or the path they concern. Rust code is linted in CI by a new `rust_lint` BazelCI task (`bazel build --config=rust-lint //...`), which runs clippy with warnings as errors and `unwrap()` forbidden, checks rustfmt, and denies `expect()` in the non-test crates. ## Performance Zip creation time only (x86_64 Linux, `-c opt`; median of 5 runs for the small cases, single run for the large ones): ``` Workload Comp Python Rust Ratio 3k .py + 5x8 MB blobs (46 MB) 0 0.43s 0.26s 1.6x same 6 1.48s 1.59s 0.9x 10k small .py files 0 1.19s 0.70s 1.7x same 6 1.42s 1.07s 1.3x CPython runtime + 10k .py (405 MB) 0 4.08s 1.19s 3.4x + torch/CUDA wheels (7.3 GB, ZIP64) 0 9.7s 7.9s 1.2x + torch/CUDA wheels 6 335s 370s 0.9x ``` Rust is faster on per-entry overhead, which dominates the default uncompressed case. Deflate throughput is ~10% slower because `flate2` uses the pure-Rust miniz_oxide backend (chosen so the tool cross-compiles for all `dev/platforms`); CPython's `zipfile` uses C zlib.
This repository is the home of the core Python rules -- py_library, py_binary, py_test, and related symbols that provide the basis for Python support in Bazel. It also contains package installation rules for integrating with PyPI and other indices.
Documentation for rules_python is at https://rules-python.readthedocs.io and in the Bazel Build Encyclopedia.
Examples live in the examples directory.
The core rules are stable. Their implementation is subject to Bazel's backward compatibility policy. This repository aims to follow semantic versioning.
The Bazel community maintains this repository. Neither Google nor the Bazel team provides support for the code. However, this repository is part of the test suite used to vet new Bazel releases. See How to contribute page for information on our development workflow.
requirements.txt is how users have been defining dependencies for a long time. We support this to support legacy usecases or package managers that we don't support directly. Any additional information that we need will be retrieved from the SimpleAPI during the bzlmod extension evaluation phase. Then it will be written to the MODULE.bazel.lock file for future reuse. We also support uv.lock files directly under bzlmod, which already contain cross-platform distribution URLs and hashes and do not require SimpleAPI queries or storing distribution hashes in MODULE.bazel.lock. uv is recommended for generating uv.lock or a fully locked requirements.txt file, and we provide a rule for it.py_binary, py_test rules should scale to large monorepos and we work hard to minimize the work done during analysis and build phase. What is more, the space requirements for should be minimal, so we strive to use symlinks rather than extracting wheels at build time. This means that for different configurations of the same build, we are not extracting the wheel multiple times thus scaling better over the time. From 2.0 onwards we are creating a virtual env for each target by creating an actual minimal virtual environment using symlinks. We plan on creating the traditional site-packages layout in the future by default.rules_python and this has resulted in a few PEPs supported within pure starlark - PEP440, PEP509.Common misconceptions:
rules_python has to keep backwards compatibility with google3. Whilst this might have been true in the past, rules_python is an open source project and any compatibility needs should come from the community - we have no requirement to keep this compatibility and are allowed to make our decisions. However, we do want to keep backwards compatibility as long as possible to not upset users with never ending migrations.rules_python is not caching pip downloads. With 2.0, we use Bazel's downloader by default and rely on bazel to provide the repository caching mechanisms. This means that for simpler setups this should result in transparent and scalable caching with the most recent bazel versions unless there are issues in the bazel itself.For detailed documentation, see https://rules-python.readthedocs.io
See Bzlmod support for more details.