feat: allow populating binary's venv site-packages with symlinks (#2617)

This implements functionality to allow libraries to populate the
site-packages directory
of downstream binaries. The basic implementation is:
* Libraries provide tuples of `(runfile path, site packages path)` in
the
  `PyInfo.site_packages_symlinks` field.
* Binaries create symlinks (using declare_symlink) in their
site-packages directory
  pointing to the runfiles paths libraries provide.

The design was chosen because of the following properties:
* The site-packages directory is relocatable
* Populating site packages is cheap ( `O(number 3p dependencies)` )
* Dependencies are only created once in the runfiles, no matter
how many how many binaries there that use them. This minimizes disk
usage,
  file counts, inodes, etc.

The `site_packages_symlinks` field is a depset with topological
ordering. Using topological
ordering allows dependencies closer to the binary to have precedence,
which gives some
basic control over what entries are used.

Additionally, the runfiles path to link to can be None/empty, in which
case, the
directory in site-packages won't be created. This allows binaries to
prevent creation
of directories that might e.g. conflict.

For now, this functionality is disabled by default. The flag
`--venvs_site_packages=yes`
can be set to allow using it, which is automatically enable it for pypi
generated targets.

When enabled, it does basic detection of implicit namespace directories,
which
allows multiple distributions to "install" into the the same
site-packages directory.

Though this functionality is primarily useful for dependencies from pypi
(e.g. via
pip.parse), it is not yet activated for those targets, for two main
reasons:
1. The wheel extraction code creates pkgutil-style `__init__.py` shims
during the repo-phase.
The build phase can't distinguish these artifical rules_python generated
shims from
actual `__init__.py` files, which breaks the implicit namespace
detection logic.
2. A flag guard is needed before changing the behavior. Even though how
3p libraries are
added to sys.path is an implementation detail, the behavior has been
there for many
   years, so an escape hatch should be added.

Work towards https://github.com/bazelbuild/rules_python/issues/2156
34 files changed
tree: 83ada1811986287eb6884532ef990e63b6f67a59
  1. .bazelci/
  2. .bcr/
  3. .ci/
  4. .github/
  5. docs/
  6. examples/
  7. gazelle/
  8. private/
  9. python/
  10. sphinxdocs/
  11. tests/
  12. third_party/
  13. tools/
  14. .bazelignore
  15. .bazelrc
  16. .bazelversion
  17. .git-blame-ignore-revs
  18. .gitattributes
  19. .gitignore
  20. .pre-commit-config.yaml
  21. .readthedocs.yml
  22. addlicense.sh
  23. AUTHORS
  24. BUILD.bazel
  25. BZLMOD_SUPPORT.md
  26. CHANGELOG.md
  27. CONTRIBUTING.md
  28. CONTRIBUTORS
  29. DEVELOPING.md
  30. internal_dev_deps.bzl
  31. internal_dev_setup.bzl
  32. LICENSE
  33. MODULE.bazel
  34. README.md
  35. RELEASING.md
  36. version.bzl
  37. WORKSPACE
  38. WORKSPACE.bzlmod
README.md

Python Rules for Bazel

Build status

Overview

This repository is the home of the core Python rules -- py_library, py_binary, py_test, py_proto_library, and related symbols that provide the basis for Python support in Bazel. It also contains package installation rules for integrating with PyPI and other indices.

Documentation for rules_python is at https://rules-python.readthedocs.io and in the Bazel Build Encyclopedia.

Examples live in the examples directory.

The core rules are stable. Their implementation is subject to Bazel's backward compatibility policy. This repository aims to follow semantic versioning.

The Bazel community maintains this repository. Neither Google nor the Bazel team provides support for the code. However, this repository is part of the test suite used to vet new Bazel releases. See How to contribute page for information on our development workflow.

Documentation

For detailed documentation, see https://rules-python.readthedocs.io

Bzlmod support

  • Status: Beta
  • Full Feature Parity: No

See Bzlmod support for more details.