fix: fixes to prepare for making bootstrap=script the default for Linux (#2760)
Various cleanup and prep work to switch bootstrap=script to be the
default.
* Change `bootstrap_impl` to always be disabled for windows. This allows
setting it to
true in a bazelrc without worrying about the target platform. This is
done by using
FeatureFlagInfo to force the value to disabled for windows. This allows
any downstream
usages of the flag to Just Work and not have to add selects() for
windows themselves.
* Switch pip_repository_annotations test to `import python.runfiles`.
The script bootstrap
doesn't add the runfiles root to sys.path, so `import rules_python`
stops working.
* Switch gazelle workspace to using the runtime-env toolchain. It was
previously
implicitly using the deprecated one built into bazel, which doesn't
provide various
necessary provider fields.
* Make the local toolchain use `sys._base_executable` instead of
`sys.executable`
when finding the interpreter. Otherwise, it might find a venv
interpreter or not
properly handle wrapper scripts like pyenv.
* Adds a toolchain attribute/field to indicate if the toolchain supports
a build-time
created venv. This is due to the runtime_env toolchain. See PR comments
for details,
but in short: if we don't know the python interpreter path and version
at
build time, the venv may not properly activate or find site-packages.
If it isn't supported, then the stage1 bootstrap creates a temporary
venv, similar
to how the zip case is handled. Unfortunately, this requires invoking
Python itself
as part of program startup, but I don't see a way around that -- note
this is
only triggered by the runtime-env toolchain.
* Make the runtime-env toolchain better support virtualenvs. Because
it's a wrapper
that re-invokes Python, Python can't automatically detect its in a venv.
Two
tricks are used (`exec -a` and PYTHONEXECUTABLE) to help address this
(but they
aren't guaranteed to work, hence the "recreate at runtime" logic).
* Fix a subtle issue where `sys._base_executable` isn't set correctly
due to `home`
missing in the pyvenv.cfg file. This mostly only affected the creation
of venvs
from within the bazel-created venv.
* Change the bazel site init to always add the build-time created
site-packages
(if it exists) as a site directory. This matches the system_python
bootstrap
behavior a bit better, which just shoved everything onto sys.path using
PYTHONPATH.
* Skip running runtime_env_toolchains tests on RBE. RBE's system python
is 3.6,
but the script bootstrap uses 3.9 features. (Running it on RBE is
questionable
anyways).
Along the way...
* Ignore gazelle convenience symlinks
* Switch pip_repository_annotations test to use
non-legacy_external_runfiles based
paths. The legacy behavior is disabled in Bazel 8+ by default.
* Also document why the script bootstrap doesn't add the runfiles root
to sys.path.
Work towards https://github.com/bazel-contrib/rules_python/issues/2521
---------
Co-authored-by: Ignas Anikevicius <240938+aignas@users.noreply.github.com>
diff --git a/.bazelignore b/.bazelignore
index e10af20..fb99909 100644
--- a/.bazelignore
+++ b/.bazelignore
@@ -25,6 +25,7 @@
examples/pip_parse_vendored/bazel-pip_parse_vendored
examples/pip_repository_annotations/bazel-pip_repository_annotations
examples/py_proto_library/bazel-py_proto_library
+gazelle/bazel-gazelle
tests/integration/compile_pip_requirements/bazel-compile_pip_requirements
tests/integration/ignore_root_user_error/bazel-ignore_root_user_error
tests/integration/local_toolchains/bazel-local_toolchains
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 154b661..f696cef 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -54,13 +54,21 @@
{#v0-0-0-changed}
### Changed
-* Nothing changed.
+* (rules) On Windows, {obj}`--bootstrap_impl=system_python` is forced. This
+ allows setting `--bootstrap_impl=script` in bazelrc for mixed-platform
+ environments.
{#v0-0-0-fixed}
### Fixed
+
* (rules) PyInfo provider is now advertised by py_test, py_binary, and py_library;
this allows aspects using required_providers to function correctly.
([#2506](https://github.com/bazel-contrib/rules_python/issues/2506)).
+* Fixes when using {obj}`--bootstrap_impl=script`:
+ * `compile_pip_requirements` now works with it
+ * The `sys._base_executable` value will reflect the underlying interpreter,
+ not venv interpreter.
+ * The {obj}`//python/runtime_env_toolchains:all` toolchain now works with it.
{#v0-0-0-added}
### Added
diff --git a/examples/pip_repository_annotations/.bazelrc b/examples/pip_repository_annotations/.bazelrc
index c16c5a2..9397bd3 100644
--- a/examples/pip_repository_annotations/.bazelrc
+++ b/examples/pip_repository_annotations/.bazelrc
@@ -5,4 +5,5 @@
# is in examples/bzlmod as the `whl_mods` feature.
common --noenable_bzlmod
common --enable_workspace
+common --legacy_external_runfiles=false
common --incompatible_python_disallow_native_rules
diff --git a/examples/pip_repository_annotations/pip_repository_annotations_test.py b/examples/pip_repository_annotations/pip_repository_annotations_test.py
index e41dd4f..219be1b 100644
--- a/examples/pip_repository_annotations/pip_repository_annotations_test.py
+++ b/examples/pip_repository_annotations/pip_repository_annotations_test.py
@@ -21,7 +21,7 @@
import unittest
from pathlib import Path
-from rules_python.python.runfiles import runfiles
+from python.runfiles import runfiles
class PipRepositoryAnnotationsTest(unittest.TestCase):
@@ -34,11 +34,7 @@
def test_build_content_and_data(self):
r = runfiles.Create()
- rpath = r.Rlocation(
- "pip_repository_annotations_example/external/{}/generated_file.txt".format(
- self.wheel_pkg_dir()
- )
- )
+ rpath = r.Rlocation("{}/generated_file.txt".format(self.wheel_pkg_dir()))
generated_file = Path(rpath)
self.assertTrue(generated_file.exists())
@@ -47,11 +43,7 @@
def test_copy_files(self):
r = runfiles.Create()
- rpath = r.Rlocation(
- "pip_repository_annotations_example/external/{}/copied_content/file.txt".format(
- self.wheel_pkg_dir()
- )
- )
+ rpath = r.Rlocation("{}/copied_content/file.txt".format(self.wheel_pkg_dir()))
copied_file = Path(rpath)
self.assertTrue(copied_file.exists())
@@ -61,7 +53,7 @@
def test_copy_executables(self):
r = runfiles.Create()
rpath = r.Rlocation(
- "pip_repository_annotations_example/external/{}/copied_content/executable{}".format(
+ "{}/copied_content/executable{}".format(
self.wheel_pkg_dir(),
".exe" if platform.system() == "windows" else ".py",
)
@@ -82,7 +74,7 @@
current_wheel_version = "0.38.4"
r = runfiles.Create()
- dist_info_dir = "pip_repository_annotations_example/external/{}/site-packages/wheel-{}.dist-info".format(
+ dist_info_dir = "{}/site-packages/wheel-{}.dist-info".format(
self.wheel_pkg_dir(),
current_wheel_version,
)
@@ -113,11 +105,8 @@
# This test verifies that annotations work correctly for pip packages with extras
# specified, in this case requests[security].
r = runfiles.Create()
- rpath = r.Rlocation(
- "pip_repository_annotations_example/external/{}/generated_file.txt".format(
- self.requests_pkg_dir()
- )
- )
+ path = "{}/generated_file.txt".format(self.requests_pkg_dir())
+ rpath = r.Rlocation(path)
generated_file = Path(rpath)
self.assertTrue(generated_file.exists())
diff --git a/gazelle/WORKSPACE b/gazelle/WORKSPACE
index 14a124d..ad428b1 100644
--- a/gazelle/WORKSPACE
+++ b/gazelle/WORKSPACE
@@ -42,6 +42,8 @@
internal_dev_deps()
+register_toolchains("@rules_python//python/runtime_env_toolchains:all")
+
load("//:deps.bzl", _py_gazelle_deps = "gazelle_deps")
# gazelle:repository_macro deps.bzl%go_deps
diff --git a/python/config_settings/BUILD.bazel b/python/config_settings/BUILD.bazel
index 45354e2..872d7d1 100644
--- a/python/config_settings/BUILD.bazel
+++ b/python/config_settings/BUILD.bazel
@@ -11,6 +11,7 @@
"PrecompileSourceRetentionFlag",
"VenvsSitePackages",
"VenvsUseDeclareSymlinkFlag",
+ rp_string_flag = "string_flag",
)
load(
"//python/private/pypi:flags.bzl",
@@ -87,14 +88,27 @@
visibility = ["//visibility:public"],
)
-string_flag(
+rp_string_flag(
name = "bootstrap_impl",
build_setting_default = BootstrapImplFlag.SYSTEM_PYTHON,
+ override = select({
+ # Windows doesn't yet support bootstrap=script, so force disable it
+ ":_is_windows": BootstrapImplFlag.SYSTEM_PYTHON,
+ "//conditions:default": "",
+ }),
values = sorted(BootstrapImplFlag.__members__.values()),
# NOTE: Only public because it's an implicit dependency
visibility = ["//visibility:public"],
)
+# For some reason, @platforms//os:windows can't be directly used
+# in the select() for the flag. But it can be used when put behind
+# a config_setting().
+config_setting(
+ name = "_is_windows",
+ constraint_values = ["@platforms//os:windows"],
+)
+
# This is used for pip and hermetic toolchain resolution.
string_flag(
name = "py_linux_libc",
diff --git a/python/private/BUILD.bazel b/python/private/BUILD.bazel
index b63f446..9cc8ffc 100644
--- a/python/private/BUILD.bazel
+++ b/python/private/BUILD.bazel
@@ -86,6 +86,7 @@
name = "runtime_env_toolchain_bzl",
srcs = ["runtime_env_toolchain.bzl"],
deps = [
+ ":config_settings_bzl",
":py_exec_tools_toolchain_bzl",
":toolchain_types_bzl",
"//python:py_runtime_bzl",
diff --git a/python/private/config_settings.bzl b/python/private/config_settings.bzl
index e5f9d86..2cf7968 100644
--- a/python/private/config_settings.bzl
+++ b/python/private/config_settings.bzl
@@ -209,3 +209,33 @@
"_template": attr.string(default = _DEBUG_ENV_MESSAGE_TEMPLATE),
},
)
+
+def is_python_version_at_least(name, **kwargs):
+ flag_name = "_{}_flag".format(name)
+ native.config_setting(
+ name = name,
+ flag_values = {
+ flag_name: "yes",
+ },
+ )
+ _python_version_at_least(
+ name = flag_name,
+ visibility = ["//visibility:private"],
+ **kwargs
+ )
+
+def _python_version_at_least_impl(ctx):
+ at_least = tuple(ctx.attr.at_least.split("."))
+ current = tuple(
+ ctx.attr._major_minor[config_common.FeatureFlagInfo].value.split("."),
+ )
+ value = "yes" if current >= at_least else "no"
+ return [config_common.FeatureFlagInfo(value = value)]
+
+_python_version_at_least = rule(
+ implementation = _python_version_at_least_impl,
+ attrs = {
+ "at_least": attr.string(mandatory = True),
+ "_major_minor": attr.label(default = _PYTHON_VERSION_MAJOR_MINOR_FLAG),
+ },
+)
diff --git a/python/private/flags.bzl b/python/private/flags.bzl
index c53e461..40ce63b 100644
--- a/python/private/flags.bzl
+++ b/python/private/flags.bzl
@@ -35,8 +35,38 @@
is_enabled = _AddSrcsToRunfilesFlag_is_enabled,
)
+def _string_flag_impl(ctx):
+ if ctx.attr.override:
+ value = ctx.attr.override
+ else:
+ value = ctx.build_setting_value
+
+ if value not in ctx.attr.values:
+ fail((
+ "Invalid value for {name}: got {value}, must " +
+ "be one of {allowed}"
+ ).format(
+ name = ctx.label,
+ value = value,
+ allowed = ctx.attr.values,
+ ))
+
+ return [
+ BuildSettingInfo(value = value),
+ config_common.FeatureFlagInfo(value = value),
+ ]
+
+string_flag = rule(
+ implementation = _string_flag_impl,
+ build_setting = config.string(flag = True),
+ attrs = {
+ "override": attr.string(),
+ "values": attr.string_list(),
+ },
+)
+
def _bootstrap_impl_flag_get_value(ctx):
- return ctx.attr._bootstrap_impl_flag[BuildSettingInfo].value
+ return ctx.attr._bootstrap_impl_flag[config_common.FeatureFlagInfo].value
# buildifier: disable=name-conventions
BootstrapImplFlag = enum(
diff --git a/python/private/get_local_runtime_info.py b/python/private/get_local_runtime_info.py
index 0207f56..19db3a2 100644
--- a/python/private/get_local_runtime_info.py
+++ b/python/private/get_local_runtime_info.py
@@ -22,6 +22,7 @@
"micro": sys.version_info.micro,
"include": sysconfig.get_path("include"),
"implementation_name": sys.implementation.name,
+ "base_executable": sys._base_executable,
}
config_vars = [
diff --git a/python/private/local_runtime_repo.bzl b/python/private/local_runtime_repo.bzl
index fb1a8e2..ec0643e 100644
--- a/python/private/local_runtime_repo.bzl
+++ b/python/private/local_runtime_repo.bzl
@@ -84,6 +84,20 @@
info = json.decode(exec_result.stdout)
logger.info(lambda: _format_get_info_result(info))
+ # We use base_executable because we want the path within a Python
+ # installation directory ("PYTHONHOME"). The problems with sys.executable
+ # are:
+ # * If we're in an activated venv, then we don't want the venv's
+ # `bin/python3` path to be used -- it isn't an actual Python installation.
+ # * If sys.executable is a wrapper (e.g. pyenv), then (1) it may not be
+ # located within an actual Python installation directory, and (2) it
+ # can interfer with Python recognizing when it's within a venv.
+ #
+ # In some cases, it may be a symlink (usually e.g. `python3->python3.12`),
+ # but we don't realpath() it to respect what it has decided is the
+ # appropriate path.
+ interpreter_path = info["base_executable"]
+
# NOTE: Keep in sync with recursive glob in define_local_runtime_toolchain_impl
repo_utils.watch_tree(rctx, rctx.path(info["include"]))
diff --git a/python/private/py_executable.bzl b/python/private/py_executable.bzl
index b4cda21..a8c669a 100644
--- a/python/private/py_executable.bzl
+++ b/python/private/py_executable.bzl
@@ -350,6 +350,7 @@
main_py = main_py,
imports = imports,
runtime_details = runtime_details,
+ venv = venv,
)
extra_runfiles = ctx.runfiles([stage2_bootstrap] + venv.files_without_interpreter)
zip_main = _create_zip_main(
@@ -538,11 +539,14 @@
ctx.actions.write(pyvenv_cfg, "")
runtime = runtime_details.effective_runtime
+
venvs_use_declare_symlink_enabled = (
VenvsUseDeclareSymlinkFlag.get_value(ctx) == VenvsUseDeclareSymlinkFlag.YES
)
+ recreate_venv_at_runtime = False
- if not venvs_use_declare_symlink_enabled:
+ if not venvs_use_declare_symlink_enabled or not runtime.supports_build_time_venv:
+ recreate_venv_at_runtime = True
if runtime.interpreter:
interpreter_actual_path = runfiles_root_path(ctx, runtime.interpreter.short_path)
else:
@@ -557,6 +561,8 @@
ctx.actions.write(interpreter, "actual:{}".format(interpreter_actual_path))
elif runtime.interpreter:
+ # Some wrappers around the interpreter (e.g. pyenv) use the program
+ # name to decide what to do, so preserve the name.
py_exe_basename = paths.basename(runtime.interpreter.short_path)
# Even though ctx.actions.symlink() is used, using
@@ -594,7 +600,8 @@
if "t" in runtime.abi_flags:
version += "t"
- site_packages = "{}/lib/python{}/site-packages".format(venv, version)
+ venv_site_packages = "lib/python{}/site-packages".format(version)
+ site_packages = "{}/{}".format(venv, venv_site_packages)
pth = ctx.actions.declare_file("{}/bazel.pth".format(site_packages))
ctx.actions.write(pth, "import _bazel_site_init\n")
@@ -616,10 +623,12 @@
return struct(
interpreter = interpreter,
- recreate_venv_at_runtime = not venvs_use_declare_symlink_enabled,
+ recreate_venv_at_runtime = recreate_venv_at_runtime,
# Runfiles root relative path or absolute path
interpreter_actual_path = interpreter_actual_path,
files_without_interpreter = [pyvenv_cfg, pth, site_init] + site_packages_symlinks,
+ # string; venv-relative path to the site-packages directory.
+ venv_site_packages = venv_site_packages,
)
def _create_site_packages_symlinks(ctx, site_packages):
@@ -716,7 +725,8 @@
output_sibling,
main_py,
imports,
- runtime_details):
+ runtime_details,
+ venv = None):
output = ctx.actions.declare_file(
# Prepend with underscore to prevent pytest from trying to
# process the bootstrap for files starting with `test_`
@@ -731,6 +741,14 @@
main_py_path = "{}/{}".format(ctx.workspace_name, main_py.short_path)
else:
main_py_path = ""
+
+ # The stage2 bootstrap uses the venv site-packages location to fix up issues
+ # that occur when the toolchain doesn't support the build-time venv.
+ if venv and not runtime.supports_build_time_venv:
+ venv_rel_site_packages = venv.venv_site_packages
+ else:
+ venv_rel_site_packages = ""
+
ctx.actions.expand_template(
template = template,
output = output,
@@ -741,6 +759,7 @@
"%main%": main_py_path,
"%main_module%": ctx.attr.main_module,
"%target%": str(ctx.label),
+ "%venv_rel_site_packages%": venv_rel_site_packages,
"%workspace_name%": ctx.workspace_name,
},
is_executable = True,
@@ -766,6 +785,12 @@
python_binary_actual = venv.interpreter_actual_path if venv else ""
+ # Runtime may be None on Windows due to the --python_path flag.
+ if runtime and runtime.supports_build_time_venv:
+ resolve_python_binary_at_runtime = "0"
+ else:
+ resolve_python_binary_at_runtime = "1"
+
subs = {
"%interpreter_args%": "\n".join([
'"{}"'.format(v)
@@ -775,7 +800,9 @@
"%python_binary%": python_binary_path,
"%python_binary_actual%": python_binary_actual,
"%recreate_venv_at_runtime%": str(int(venv.recreate_venv_at_runtime)) if venv else "0",
+ "%resolve_python_binary_at_runtime%": resolve_python_binary_at_runtime,
"%target%": str(ctx.label),
+ "%venv_rel_site_packages%": venv.venv_site_packages if venv else "",
"%workspace_name%": ctx.workspace_name,
}
diff --git a/python/private/py_runtime_info.bzl b/python/private/py_runtime_info.bzl
index 4297391..d2ae17e 100644
--- a/python/private/py_runtime_info.bzl
+++ b/python/private/py_runtime_info.bzl
@@ -67,7 +67,8 @@
stage2_bootstrap_template = None,
zip_main_template = None,
abi_flags = "",
- site_init_template = None):
+ site_init_template = None,
+ supports_build_time_venv = True):
if (interpreter_path and interpreter) or (not interpreter_path and not interpreter):
fail("exactly one of interpreter or interpreter_path must be specified")
@@ -119,6 +120,7 @@
"site_init_template": site_init_template,
"stage2_bootstrap_template": stage2_bootstrap_template,
"stub_shebang": stub_shebang,
+ "supports_build_time_venv": supports_build_time_venv,
"zip_main_template": zip_main_template,
}
@@ -313,6 +315,28 @@
script used when executing {obj}`py_binary` targets. Does not
apply to Windows.
""",
+ "supports_build_time_venv": """
+:type: bool
+
+True if this toolchain supports the build-time created virtual environment.
+False if not or unknown. If build-time venv creation isn't supported, then binaries may
+fallback to non-venv solutions or creating a venv at runtime.
+
+In order to use the build-time created virtual environment, a toolchain needs
+to meet two criteria:
+1. Specifying the underlying executable (e.g. `/usr/bin/python3`, as reported by
+ `sys._base_executable`) for the venv executable (`$venv/bin/python3`, as reported
+ by `sys.executable`). This typically requires relative symlinking the venv
+ path to the underlying path at build time, or using the `PYTHONEXECUTABLE`
+ environment variable (Python 3.11+) at runtime.
+2. Having the build-time created site-packages directory
+ (`<venv>/lib/python{version}/site-packages`) recognized by the runtime
+ interpreter. This typically requires the Python version to be known at
+ build-time and match at runtime.
+
+:::{versionadded} VERSION_NEXT_FEATURE
+:::
+""",
"zip_main_template": """
:type: File
diff --git a/python/private/py_runtime_rule.bzl b/python/private/py_runtime_rule.bzl
index a85f5b2..6dadcfe 100644
--- a/python/private/py_runtime_rule.bzl
+++ b/python/private/py_runtime_rule.bzl
@@ -130,6 +130,7 @@
zip_main_template = ctx.file.zip_main_template,
abi_flags = abi_flags,
site_init_template = ctx.file.site_init_template,
+ supports_build_time_venv = ctx.attr.supports_build_time_venv,
))
if not IS_BAZEL_7_OR_HIGHER:
@@ -353,6 +354,17 @@
Does not apply to Windows.
""",
),
+ "supports_build_time_venv": attr.bool(
+ doc = """
+Whether this runtime supports virtualenvs created at build time.
+
+See {obj}`PyRuntimeInfo.supports_build_time_venv` for docs.
+
+:::{versionadded} VERSION_NEXT_FEATURE
+:::
+""",
+ default = True,
+ ),
"zip_main_template": attr.label(
default = "//python/private:zip_main_template",
allow_single_file = True,
diff --git a/python/private/runtime_env_toolchain.bzl b/python/private/runtime_env_toolchain.bzl
index 2116012..1956ad5 100644
--- a/python/private/runtime_env_toolchain.bzl
+++ b/python/private/runtime_env_toolchain.bzl
@@ -17,6 +17,7 @@
load("//python:py_runtime.bzl", "py_runtime")
load("//python:py_runtime_pair.bzl", "py_runtime_pair")
load("//python/cc:py_cc_toolchain.bzl", "py_cc_toolchain")
+load("//python/private:config_settings.bzl", "is_python_version_at_least")
load(":py_exec_tools_toolchain.bzl", "py_exec_tools_toolchain")
load(":toolchain_types.bzl", "EXEC_TOOLS_TOOLCHAIN_TYPE", "PY_CC_TOOLCHAIN_TYPE", "TARGET_TOOLCHAIN_TYPE")
@@ -38,6 +39,11 @@
"""
base_name = name.replace("_toolchain", "")
+ supports_build_time_venv = select({
+ ":_is_at_least_py3.11": True,
+ "//conditions:default": False,
+ })
+
py_runtime(
name = "_runtime_env_py3_runtime",
interpreter = "//python/private:runtime_env_toolchain_interpreter.sh",
@@ -45,6 +51,7 @@
stub_shebang = "#!/usr/bin/env python3",
visibility = ["//visibility:private"],
tags = ["manual"],
+ supports_build_time_venv = supports_build_time_venv,
)
# This is a dummy runtime whose interpreter_path triggers the native rule
@@ -56,6 +63,7 @@
python_version = "PY3",
visibility = ["//visibility:private"],
tags = ["manual"],
+ supports_build_time_venv = supports_build_time_venv,
)
py_runtime_pair(
@@ -110,3 +118,7 @@
toolchain_type = PY_CC_TOOLCHAIN_TYPE,
visibility = ["//visibility:public"],
)
+ is_python_version_at_least(
+ name = "_is_at_least_py3.11",
+ at_least = "3.11",
+ )
diff --git a/python/private/runtime_env_toolchain_interpreter.sh b/python/private/runtime_env_toolchain_interpreter.sh
index b09bc53..6159d4f 100755
--- a/python/private/runtime_env_toolchain_interpreter.sh
+++ b/python/private/runtime_env_toolchain_interpreter.sh
@@ -53,5 +53,29 @@
(https://github.com/bazel-contrib/rules_python/blob/master/docs/python.md#py_runtime_pair)."
fi
-exec "$PYTHON_BIN" "$@"
+# Because this is a wrapper script that invokes Python, it prevents Python from
+# detecting virtualenvs like normal (i.e. using the venv symlink to find the
+# real interpreter). To work around this, we have to manually detect the venv,
+# then trick the interpreter into understanding we're in a virtual env.
+self_dir=$(dirname "$0")
+if [ -e "$self_dir/pyvenv.cfg" ] || [ -e "$self_dir/../pyvenv.cfg" ]; then
+ case "$0" in
+ /*)
+ venv_bin="$0"
+ ;;
+ *)
+ venv_bin="$PWD/$0"
+ ;;
+ esac
+ # PYTHONEXECUTABLE is also used because `exec -a` doesn't fully trick the
+ # pyenv wrappers.
+ # NOTE: The PYTHONEXECUTABLE envvar only works for non-Mac starting in Python 3.11
+ export PYTHONEXECUTABLE="$venv_bin"
+ # Python looks at argv[0] to determine sys.executable, so use exec -a
+ # to make it think it's the venv's binary, not the actual one invoked.
+ # NOTE: exec -a isn't strictly posix-compatible, but very widespread
+ exec -a "$venv_bin" "$PYTHON_BIN" "$@"
+else
+ exec "$PYTHON_BIN" "$@"
+fi
diff --git a/python/private/site_init_template.py b/python/private/site_init_template.py
index 40fb4e4..a87a0d2 100644
--- a/python/private/site_init_template.py
+++ b/python/private/site_init_template.py
@@ -125,6 +125,14 @@
def _setup_sys_path():
+ """Perform Bazel/binary specific sys.path setup.
+
+ NOTE: We do not add _RUNFILES_ROOT to sys.path for two reasons:
+ 1. Under workspace, it makes every external repository importable. If a Bazel
+ repository matches a Python import name, they conflict.
+ 2. Under bzlmod, the repo names in the runfiles directory aren't importable
+ Python names, so there's no point in adding the runfiles root to sys.path.
+ """
seen = set(sys.path)
python_path_entries = []
@@ -195,5 +203,27 @@
return coverage_setup
+def _fixup_sys_base_executable():
+ """Fixup sys._base_executable to account for Bazel-specific pyvenv.cfg
+
+ The pyvenv.cfg created for py_binary leaves the `home` key unset. A
+ side-effect of this is `sys._base_executable` points to the venv executable,
+ not the actual executable. This mostly doesn't matter, but does affect
+ using the venv module to create venvs (they point to the venv executable, not
+ the actual executable).
+ """
+ # Must have been set correctly?
+ if sys.executable != sys._base_executable:
+ return
+ # Not in a venv, so don't touch anything.
+ if sys.prefix == sys.base_prefix:
+ return
+ exe = os.path.realpath(sys.executable)
+ _print_verbose("setting sys._base_executable:", exe)
+ sys._base_executable = exe
+
+
+_fixup_sys_base_executable()
+
COVERAGE_SETUP = _setup_sys_path()
_print_verbose("DONE")
diff --git a/python/private/stage1_bootstrap_template.sh b/python/private/stage1_bootstrap_template.sh
index c487624..d992b55 100644
--- a/python/private/stage1_bootstrap_template.sh
+++ b/python/private/stage1_bootstrap_template.sh
@@ -9,7 +9,8 @@
# runfiles-relative path
STAGE2_BOOTSTRAP="%stage2_bootstrap%"
-# runfiles-relative path to python interpreter to use
+# runfiles-relative path to python interpreter to use.
+# This is the `bin/python3` path in the binary's venv.
PYTHON_BINARY='%python_binary%'
# The path that PYTHON_BINARY should symlink to.
# runfiles-relative path, absolute path, or single word.
@@ -18,8 +19,17 @@
# 0 or 1
IS_ZIPFILE="%is_zipfile%"
-# 0 or 1
+# 0 or 1.
+# If 1, then a venv will be created at runtime that replicates what would have
+# been the build-time structure.
RECREATE_VENV_AT_RUNTIME="%recreate_venv_at_runtime%"
+# 0 or 1
+# If 1, then the path to python will be resolved by running
+# PYTHON_BINARY_ACTUAL to determine the actual underlying interpreter.
+RESOLVE_PYTHON_BINARY_AT_RUNTIME="%resolve_python_binary_at_runtime%"
+# venv-relative path to the site-packages
+# e.g. lib/python3.12t/site-packages
+VENV_REL_SITE_PACKAGES="%venv_rel_site_packages%"
# array of strings
declare -a INTERPRETER_ARGS_FROM_TARGET=(
@@ -152,34 +162,72 @@
fi
fi
- if [[ "$PYTHON_BINARY_ACTUAL" == /* ]]; then
- # An absolute path, i.e. platform runtime, e.g. /usr/bin/python3
- symlink_to=$PYTHON_BINARY_ACTUAL
- elif [[ "$PYTHON_BINARY_ACTUAL" == */* ]]; then
- # A runfiles-relative path
- symlink_to="$RUNFILES_DIR/$PYTHON_BINARY_ACTUAL"
- else
- # A plain word, e.g. "python3". Symlink to where PATH leads
- symlink_to=$(which $PYTHON_BINARY_ACTUAL)
- # Guard against trying to symlink to an empty value
- if [[ $? -ne 0 ]]; then
- echo >&2 "ERROR: Python to use not found on PATH: $PYTHON_BINARY_ACTUAL"
- exit 1
- fi
- fi
- mkdir -p "$venv/bin"
# Match the basename; some tools, e.g. pyvenv key off the executable name
python_exe="$venv/bin/$(basename $PYTHON_BINARY_ACTUAL)"
+
if [[ ! -e "$python_exe" ]]; then
- ln -s "$symlink_to" "$python_exe"
+ if [[ "$PYTHON_BINARY_ACTUAL" == /* ]]; then
+ # An absolute path, i.e. platform runtime, e.g. /usr/bin/python3
+ python_exe_actual=$PYTHON_BINARY_ACTUAL
+ elif [[ "$PYTHON_BINARY_ACTUAL" == */* ]]; then
+ # A runfiles-relative path
+ python_exe_actual="$RUNFILES_DIR/$PYTHON_BINARY_ACTUAL"
+ else
+ # A plain word, e.g. "python3". Symlink to where PATH leads
+ python_exe_actual=$(which $PYTHON_BINARY_ACTUAL)
+ # Guard against trying to symlink to an empty value
+ if [[ $? -ne 0 ]]; then
+ echo >&2 "ERROR: Python to use not found on PATH: $PYTHON_BINARY_ACTUAL"
+ exit 1
+ fi
+ fi
+
+ runfiles_venv="$RUNFILES_DIR/$(dirname $(dirname $PYTHON_BINARY))"
+ # When RESOLVE_PYTHON_BINARY_AT_RUNTIME is true, it means the toolchain
+ # has thrown two complications at us:
+ # 1. The build-time assumption of the Python version may not match the
+ # runtime Python version. The site-packages directory path includes the
+ # Python version, so when the versions don't match, the runtime won't
+ # find it.
+ # 2. The interpreter might be a wrapper script, which interferes with Python's
+ # ability to detect when it's within a venv. Starting in Python 3.11,
+ # the PYTHONEXECUTABLE environment variable can fix this, but due to (1),
+ # we don't know if that is supported without running Python.
+ # To fix (1), we symlink the desired site-packages path to the build-time
+ # directory. Hopefully the version mismatch is OK :D.
+ # To fix (2), we determine the actual underlying interpreter and symlink
+ # to that.
+ if [[ "$RESOLVE_PYTHON_BINARY_AT_RUNTIME" == "1" ]]; then
+ {
+ read -r resolved_py_exe
+ read -r resolved_site_packages
+ } < <("$python_exe_actual" -I <<EOF
+import sys, site, os
+print(sys.executable)
+print(site.getsitepackages(["$venv"])[-1])
+EOF
+)
+ python_exe_actual="$resolved_py_exe"
+ runfiles_venv_site_packages=$runfiles_venv/$VENV_REL_SITE_PACKAGES
+ venv_site_packages="$resolved_site_packages"
+ else
+ # For simplicity, just symlink to the whole lib directory.
+ runfiles_venv_site_packages=$runfiles_venv/lib
+ venv_site_packages=$venv/lib
+ fi
+
+ mkdir -p "$venv/bin"
+ ln -s "$python_exe_actual" "$python_exe"
+
+ if [[ ! -e "$venv_site_packages" ]]; then
+ mkdir -p $(dirname $venv_site_packages)
+ ln -s "$runfiles_venv_site_packages" "$venv_site_packages"
+ fi
fi
- runfiles_venv="$RUNFILES_DIR/$(dirname $(dirname $PYTHON_BINARY))"
+
if [[ ! -e "$venv/pyvenv.cfg" ]]; then
ln -s "$runfiles_venv/pyvenv.cfg" "$venv/pyvenv.cfg"
fi
- if [[ ! -e "$venv/lib" ]]; then
- ln -s "$runfiles_venv/lib" "$venv/lib"
- fi
else
use_exec=1
fi
diff --git a/python/private/stage2_bootstrap_template.py b/python/private/stage2_bootstrap_template.py
index e8228ed..fcc323e 100644
--- a/python/private/stage2_bootstrap_template.py
+++ b/python/private/stage2_bootstrap_template.py
@@ -32,6 +32,12 @@
# Module name to execute. Empty if MAIN is used.
MAIN_MODULE = "%main_module%"
+# venv-relative path to the expected location of the binary's site-packages
+# directory.
+# Only set when the toolchain doesn't support the build-time venv. Empty
+# string otherwise.
+VENV_SITE_PACKAGES = "%venv_rel_site_packages%"
+
# ===== Template substitutions end =====
@@ -365,6 +371,22 @@
print_verbose("initial environ:", mapping=os.environ)
print_verbose("initial sys.path:", values=sys.path)
+ if VENV_SITE_PACKAGES:
+ site_packages = os.path.join(sys.prefix, VENV_SITE_PACKAGES)
+ if site_packages not in sys.path and os.path.exists(site_packages):
+ # NOTE: if this happens, it likely means we're running with a different
+ # Python version than was built with. Things may or may not work.
+ # Such a situation is likely due to the runtime_env toolchain, or some
+ # toolchain configuration. In any case, this better matches how the
+ # previous bootstrap=system_python bootstrap worked (using PYTHONPATH,
+ # which isn't version-specific).
+ print_verbose(
+ f"sys.path missing expected site-packages: adding {site_packages}"
+ )
+ import site
+
+ site.addsitedir(site_packages)
+
main_rel_path = None
# todo: things happen to work because find_runfiles_root
# ends up using stage2_bootstrap, and ends up computing the proper
diff --git a/tests/integration/local_toolchains/BUILD.bazel b/tests/integration/local_toolchains/BUILD.bazel
index 6fbf548..02b126b 100644
--- a/tests/integration/local_toolchains/BUILD.bazel
+++ b/tests/integration/local_toolchains/BUILD.bazel
@@ -17,4 +17,6 @@
py_test(
name = "test",
srcs = ["test.py"],
+ # Make this test better respect pyenv
+ env_inherit = ["PYENV_VERSION"],
)
diff --git a/tests/integration/local_toolchains/test.py b/tests/integration/local_toolchains/test.py
index d85a4c3..8e37fff 100644
--- a/tests/integration/local_toolchains/test.py
+++ b/tests/integration/local_toolchains/test.py
@@ -1,6 +1,8 @@
+import os.path
import shutil
import subprocess
import sys
+import tempfile
import unittest
@@ -8,19 +10,58 @@
maxDiff = None
def test_python_from_path_used(self):
+ # NOTE: This is a bit brittle. It assumes the environment during the
+ # repo-phase and when the test is run are roughly the same. It's
+ # easy to violate this condition if there are shell-local changes
+ # that wouldn't be reflected when sub-shells are run later.
shell_path = shutil.which("python3")
# We call the interpreter and print its executable because of
# things like pyenv: they install a shim that re-execs python.
# The shim is e.g. /home/user/.pyenv/shims/python3, which then
# runs e.g. /usr/bin/python3
- expected = subprocess.check_output(
- [shell_path, "-c", "import sys; print(sys.executable)"],
- text=True,
- )
- expected = expected.strip().lower()
+ with tempfile.NamedTemporaryFile(suffix="_info.py", mode="w+") as f:
+ f.write(
+ """
+import sys
+print(sys.executable)
+print(sys._base_executable)
+"""
+ )
+ f.flush()
+ output_lines = (
+ subprocess.check_output(
+ [shell_path, f.name],
+ text=True,
+ )
+ .strip()
+ .splitlines()
+ )
+ shell_exe, shell_base_exe = output_lines
+
+ # Call realpath() to help normalize away differences from symlinks.
+ # Use base executable to ignore a venv the test may be running within.
+ expected = os.path.realpath(shell_base_exe.strip().lower())
+ actual = os.path.realpath(sys._base_executable.lower())
+
+ msg = f"""
+details of executables:
+test's runtime:
+{sys.executable=}
+{sys._base_executable=}
+realpath exe : {os.path.realpath(sys.executable)}
+realpath base_exe: {os.path.realpath(sys._base_executable)}
+
+from shell resolution:
+which python3: {shell_path=}:
+{shell_exe=}
+{shell_base_exe=}
+realpath exe : {os.path.realpath(shell_exe)}
+realpath base_exe: {os.path.realpath(shell_base_exe)}
+""".strip()
+
# Normalize case: Windows may have case differences
- self.assertEqual(expected.lower(), sys.executable.lower())
+ self.assertEqual(expected.lower(), actual.lower(), msg=msg)
if __name__ == "__main__":
diff --git a/tests/runtime_env_toolchain/BUILD.bazel b/tests/runtime_env_toolchain/BUILD.bazel
index afc6b58..59ca93b 100644
--- a/tests/runtime_env_toolchain/BUILD.bazel
+++ b/tests/runtime_env_toolchain/BUILD.bazel
@@ -30,5 +30,9 @@
CC_TOOLCHAIN,
],
main = "toolchain_runs_test.py",
+ # Our RBE has Python 3.6, which is too old for the language features
+ # we use now. Using the runtime-env toolchain on RBE is pretty
+ # questionable anyways.
+ tags = ["no-remote-exec"],
deps = ["//python/runfiles"],
)
