feat: allow populating binary's venv site-packages with symlinks (#2617) This implements functionality to allow libraries to populate the site-packages directory of downstream binaries. The basic implementation is: * Libraries provide tuples of `(runfile path, site packages path)` in the `PyInfo.site_packages_symlinks` field. * Binaries create symlinks (using declare_symlink) in their site-packages directory pointing to the runfiles paths libraries provide. The design was chosen because of the following properties: * The site-packages directory is relocatable * Populating site packages is cheap ( `O(number 3p dependencies)` ) * Dependencies are only created once in the runfiles, no matter how many how many binaries there that use them. This minimizes disk usage, file counts, inodes, etc. The `site_packages_symlinks` field is a depset with topological ordering. Using topological ordering allows dependencies closer to the binary to have precedence, which gives some basic control over what entries are used. Additionally, the runfiles path to link to can be None/empty, in which case, the directory in site-packages won't be created. This allows binaries to prevent creation of directories that might e.g. conflict. For now, this functionality is disabled by default. The flag `--venvs_site_packages=yes` can be set to allow using it, which is automatically enable it for pypi generated targets. When enabled, it does basic detection of implicit namespace directories, which allows multiple distributions to "install" into the the same site-packages directory. Though this functionality is primarily useful for dependencies from pypi (e.g. via pip.parse), it is not yet activated for those targets, for two main reasons: 1. The wheel extraction code creates pkgutil-style `__init__.py` shims during the repo-phase. The build phase can't distinguish these artifical rules_python generated shims from actual `__init__.py` files, which breaks the implicit namespace detection logic. 2. A flag guard is needed before changing the behavior. Even though how 3p libraries are added to sys.path is an implementation detail, the behavior has been there for many years, so an escape hatch should be added. Work towards https://github.com/bazelbuild/rules_python/issues/2156
This repository is the home of the core Python rules -- py_library, py_binary, py_test, py_proto_library, and related symbols that provide the basis for Python support in Bazel. It also contains package installation rules for integrating with PyPI and other indices.
Documentation for rules_python is at https://rules-python.readthedocs.io and in the Bazel Build Encyclopedia.
Examples live in the examples directory.
The core rules are stable. Their implementation is subject to Bazel's backward compatibility policy. This repository aims to follow semantic versioning.
The Bazel community maintains this repository. Neither Google nor the Bazel team provides support for the code. However, this repository is part of the test suite used to vet new Bazel releases. See How to contribute page for information on our development workflow.
For detailed documentation, see https://rules-python.readthedocs.io
See Bzlmod support for more details.