Add exports for core Python logic that's bundled with Bazel (#202) * Introduce defs.bzl as the official home of the core Python rules The "core" Python rules are the rules that traditionally have been bundled with Bazel. This includes native rules like `py_binary`, and Starlark-defined rules under `@bazel_tools` like `py_runtime_pair`. These should all live in or around `@rules_python//python:defs.bzl`. Currently we re-export the native rules here, with a magic tag to allow them to survive the flag flip for `--incompatible_load_python_rules_from_bzl`. When native rules are ported to Starlark their definitions will live here. * Add re-exports for Starlark-defined symbols This adds export definitions for built-in symbols like `PyInfo` and `@bazel_tools`-defined symbols like py_runtime_pair. * Vendor in runfiles library This vendors in the @bazel_tools//tools/python/runfiles target as //python/runfiles. See comment in the BUILD file for why we couldn't re-export the bundled implementation. * Fix README to prefer defs.bzl over python.bzl
diff --git a/README.md b/README.md index f222f39..f777f79 100644 --- a/README.md +++ b/README.md
@@ -46,7 +46,7 @@ ``` python load( - "@rules_python//python:python.bzl", + "@rules_python//python:defs.bzl", "py_binary", "py_library", "py_test", )
diff --git a/python/BUILD b/python/BUILD index 33b2c8d..320db04 100644 --- a/python/BUILD +++ b/python/BUILD
@@ -11,12 +11,104 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. + +"""This package contains two sets of rules: + + 1) the "core" Python rules, which were historically bundled with Bazel and + are now either re-exported or copied into this repository; and + + 2) the packaging rules, which were historically simply known as + rules_python. + +In an ideal renaming, we'd move the packaging rules to a different package so +that @rules_python//python is only concerned with the core rules. +""" + package(default_visibility = ["//visibility:public"]) licenses(["notice"]) # Apache 2.0 +# ========= Core rules ========= + +exports_files([ + "defs.bzl", + "python.bzl", # Deprecated, please use defs.bzl +]) + +# This target can be used to inspect the current Python major version. To use, +# put it in the `flag_values` attribute of a `config_setting` and test it +# against the values "PY2" or "PY3". It will always match one or the other. +# +# If you do not need to test any other flags in combination with the Python +# version, then as a convenience you may use the predefined `config_setting`s +# `@rules_python//python:PY2` and `@rules_python//python:PY3`. +# +# Example usage: +# +# config_setting( +# name = "py3_on_arm", +# values = {"cpu": "arm"}, +# flag_values = {"@rules_python//python:python_version": "PY3"}, +# ) +# +# my_target( +# ... +# some_attr = select({ +# ":py3_on_arm": ..., +# ... +# }), +# ... +# ) +# +# Caution: Do not `select()` on the built-in command-line flags `--force_python` +# or `--python_version`, as they do not always reflect the true Python version +# of the current target. `select()`-ing on them can lead to action conflicts and +# will be disallowed. +alias( + name = "python_version", + actual = "@bazel_tools//tools/python:python_version", +) + +alias( + name = "PY2", + actual = "@bazel_tools//tools/python:PY2", +) + +alias( + name = "PY3", + actual = "@bazel_tools//tools/python:PY3", +) + +# The toolchain type for Python rules. Provides a Python 2 and/or Python 3 +# runtime. +alias( + name = "toolchain_type", + actual = "@bazel_tools//tools/python:toolchain_type", +) + +# Definitions for a Python toolchain that, at execution time, attempts to detect +# a platform runtime having the appropriate major Python version. Consider this +# a toolchain of last resort. +# +# The non-strict version allows using a Python 2 interpreter for PY3 targets, +# and vice versa. The only reason to use this is if you're working around +# spurious failures due to PY2 vs PY3 validation. Even then, using this is only +# safe if you know for a fact that your build is completely compatible with the +# version of the `python` command installed on the target platform. + +alias( + name = "autodetecting_toolchain", + actual = "@bazel_tools//tools/python:autodetecting_toolchain", +) + +alias( + name = "autodetecting_toolchain_nonstrict", + actual = "@bazel_tools//tools/python:autodetecting_toolchain_nonstrict", +) + +# ========= Packaging rules ========= + exports_files([ "pip.bzl", - "python.bzl", "whl.bzl", ])
diff --git a/python/constraints/BUILD b/python/constraints/BUILD new file mode 100644 index 0000000..b93c215 --- /dev/null +++ b/python/constraints/BUILD
@@ -0,0 +1,31 @@ +# Copyright 2019 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +package(default_visibility = ["//visibility:public"]) + +licenses(["notice"]) + +# A constraint_setting to use for constraints related to the location of the +# system Python 2 interpreter on a platform. +alias( + name = "py2_interpreter_path", + actual = "@bazel_tools//tools/python:py2_interpreter_path", +) + +# A constraint_setting to use for constraints related to the location of the +# system Python 3 interpreter on a platform. +alias( + name = "py3_interpreter_path", + actual = "@bazel_tools//tools/python:py3_interpreter_path", +)
diff --git a/python/defs.bzl b/python/defs.bzl new file mode 100644 index 0000000..d5092d7 --- /dev/null +++ b/python/defs.bzl
@@ -0,0 +1,88 @@ +# Copyright 2019 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Core rules for building Python projects. + +Currently the definitions here are re-exports of the native rules, "blessed" to +work under `--incompatible_load_python_rules_from_bzl`. As the native rules get +migrated to Starlark, their implementations will be moved here. +""" + +load("@bazel_tools//tools/python:toolchain.bzl", _py_runtime_pair = "py_runtime_pair") +load("@bazel_tools//tools/python:srcs_version.bzl", _find_requirements = "find_requirements") +load(":private/reexports.bzl", "internal_PyInfo", "internal_PyRuntimeInfo") + +# Exports of native-defined providers. + +PyInfo = internal_PyInfo + +PyRuntimeInfo = internal_PyRuntimeInfo + +# The implementation of the macros and tagging mechanism follows the example +# set by rules_cc and rules_java. + +_MIGRATION_TAG = "__PYTHON_RULES_MIGRATION_DO_NOT_USE_WILL_BREAK__" + +def _add_tags(attrs): + if "tags" in attrs and attrs["tags"] != None: + attrs["tags"] += [_MIGRATION_TAG] + else: + attrs["tags"] = [_MIGRATION_TAG] + return attrs + +def py_library(**attrs): + """See the Bazel core [py_library]( + https://docs.bazel.build/versions/master/be/python.html#py_library) + documentation. + + Args: + **attrs: Rule attributes + """ + native.py_library(**_add_tags(attrs)) + +def py_binary(**attrs): + """See the Bazel core [py_binary]( + https://docs.bazel.build/versions/master/be/python.html#py_binary) + documentation. + + Args: + **attrs: Rule attributes + """ + native.py_binary(**_add_tags(attrs)) + +def py_test(**attrs): + """See the Bazel core [py_test]( + https://docs.bazel.build/versions/master/be/python.html#py_test) + documentation. + + Args: + **attrs: Rule attributes + """ + native.py_test(**_add_tags(attrs)) + +def py_runtime(**attrs): + """See the Bazel core [py_runtime]( + https://docs.bazel.build/versions/master/be/python.html#py_runtime) + documentation. + + Args: + **attrs: Rule attributes + """ + native.py_runtime(**_add_tags(attrs)) + +# Re-exports of Starlark-defined symbols in @bazel_tools//tools/python. + +py_runtime_pair = _py_runtime_pair + +find_requirements = _find_requirements
diff --git a/python/private/reexports.bzl b/python/private/reexports.bzl new file mode 100644 index 0000000..cb01b85 --- /dev/null +++ b/python/private/reexports.bzl
@@ -0,0 +1,40 @@ +# Copyright 2019 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Internal re-exports of built-in symbols. + +We want to re-export a built-in symbol as if it were defined in a Starlark +file, so that users can for instance do: + +``` +load("@rules_python//python:defs.bzl", "PyInfo") +``` + +Unfortunately, we can't just write in defs.bzl + +``` +PyInfo = PyInfo +``` + +because the declaration of module-level symbol `PyInfo` makes the builtin +inaccessible. So instead we access the builtin here and export it under a +different name. Then we can load it from defs.bzl and export it there under +the original name. +""" + +# Don't use underscore prefix, since that would make the symbol local to this +# file only. + +internal_PyInfo = PyInfo +internal_PyRuntimeInfo = PyRuntimeInfo
diff --git a/python/python.bzl b/python/python.bzl index 3f96e7c..f94d7ab 100644 --- a/python/python.bzl +++ b/python/python.bzl
@@ -12,26 +12,44 @@ # See the License for the specific language governing permissions and # limitations under the License. -def py_library(*args, **kwargs): - """See the Bazel core py_library documentation. +"""Re-exports for some of the core Bazel Python rules. - [available here]( - https://docs.bazel.build/versions/master/be/python.html#py_library). +This file is deprecated; please use the exports in defs.bzl instead. This is to +follow the new naming convention of putting core rules for a language +underneath @rules_<LANG>//<LANG>:defs.bzl. The exports in this file will be +disallowed in a future Bazel release by +`--incompatible_load_python_rules_from_bzl`. +""" + +def py_library(*args, **kwargs): + """See the Bazel core [py_library]( + https://docs.bazel.build/versions/master/be/python.html#py_library) + documentation. + + Deprecated: This symbol will become unusuable when + `--incompatible_load_python_rules_from_bzl` is enabled. Please use the + symbols in `@rules_python//python:defs.bzl` instead. """ native.py_library(*args, **kwargs) def py_binary(*args, **kwargs): - """See the Bazel core py_binary documentation. + """See the Bazel core [py_binary]( + https://docs.bazel.build/versions/master/be/python.html#py_binary) + documentation. - [available here]( - https://docs.bazel.build/versions/master/be/python.html#py_binary). + Deprecated: This symbol will become unusuable when + `--incompatible_load_python_rules_from_bzl` is enabled. Please use the + symbols in `@rules_python//python:defs.bzl` instead. """ native.py_binary(*args, **kwargs) def py_test(*args, **kwargs): - """See the Bazel core py_test documentation. + """See the Bazel core [py_test]( + https://docs.bazel.build/versions/master/be/python.html#py_test) + documentation. - [available here]( - https://docs.bazel.build/versions/master/be/python.html#py_test). + Deprecated: This symbol will become unusuable when + `--incompatible_load_python_rules_from_bzl` is enabled. Please use the + symbols in `@rules_python//python:defs.bzl` instead. """ native.py_test(*args, **kwargs)
diff --git a/python/runfiles/BUILD b/python/runfiles/BUILD new file mode 100644 index 0000000..7db3493 --- /dev/null +++ b/python/runfiles/BUILD
@@ -0,0 +1,39 @@ +# Copyright 2019 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# We'd like to alias the runfiles target @bazel_tools//tools/python/runfiles. +# However, we need its source file to exist in the runfiles tree under this +# repo's name, so that it can be imported as +# +# from rules_python.python.runfiles import runfiles +# +# in user code. This requires either adding a symlink to runfiles or copying +# the file with an action. +# +# Both solutions are made more difficult by the fact that runfiles.py is not +# directly exported by its package. We could try to get a handle on its File +# object by unpacking the runfiles target's providers, but this seems hacky +# and is probably more effort than it's worth. Also, it's not trivial to copy +# files in a cross-platform (i.e. Windows-friendly) way. +# +# So instead, we just vendor in runfiles.py here. + +load("//python:defs.bzl", "py_library") + +py_library( + name = "runfiles", + srcs = ["runfiles.py"], + visibility = ["//visibility:public"], +)
diff --git a/python/runfiles/runfiles.py b/python/runfiles/runfiles.py new file mode 100644 index 0000000..e8e867d --- /dev/null +++ b/python/runfiles/runfiles.py
@@ -0,0 +1,293 @@ +# Copyright 2018 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +############################################################################### +# Vendored in from bazelbuild/bazel (tools/python/runfiles/runfiles.py) at # +# commit 6c60a8ec049b6b8540c473969dd7bd1dad46acb9 (2019-07-19). See # +# //python/runfiles:BUILD for details. # +############################################################################### + +"""Runfiles lookup library for Bazel-built Python binaries and tests. + +USAGE: + +1. Depend on this runfiles library from your build rule: + + py_binary( + name = "my_binary", + ... + deps = ["@bazel_tools//tools/python/runfiles"], + ) + +2. Import the runfiles library. + + from bazel_tools.tools.python.runfiles import runfiles + +3. Create a Runfiles object and use rlocation to look up runfile paths: + + r = runfiles.Create() + ... + with open(r.Rlocation("my_workspace/path/to/my/data.txt"), "r") as f: + contents = f.readlines() + ... + + The code above creates a manifest- or directory-based implementations based + on the environment variables in os.environ. See `Create()` for more info. + + If you want to explicitly create a manifest- or directory-based + implementations, you can do so as follows: + + r1 = runfiles.CreateManifestBased("path/to/foo.runfiles_manifest") + + r2 = runfiles.CreateDirectoryBased("path/to/foo.runfiles/") + + If you want to start subprocesses that also need runfiles, you need to set + the right environment variables for them: + + import subprocess + from bazel_tools.tools.python.runfiles import runfiles + + r = runfiles.Create() + env = {} + ... + env.update(r.EnvVars()) + p = subprocess.Popen([r.Rlocation("path/to/binary")], env, ...) +""" + +import os +import posixpath + + +def CreateManifestBased(manifest_path): + return _Runfiles(_ManifestBased(manifest_path)) + + +def CreateDirectoryBased(runfiles_dir_path): + return _Runfiles(_DirectoryBased(runfiles_dir_path)) + + +def Create(env=None): + """Returns a new `Runfiles` instance. + + The returned object is either: + - manifest-based, meaning it looks up runfile paths from a manifest file, or + - directory-based, meaning it looks up runfile paths under a given directory + path + + If `env` contains "RUNFILES_MANIFEST_FILE" with non-empty value, this method + returns a manifest-based implementation. The object eagerly reads and caches + the whole manifest file upon instantiation; this may be relevant for + performance consideration. + + Otherwise, if `env` contains "RUNFILES_DIR" with non-empty value (checked in + this priority order), this method returns a directory-based implementation. + + If neither cases apply, this method returns null. + + Args: + env: {string: string}; optional; the map of environment variables. If None, + this function uses the environment variable map of this process. + Raises: + IOError: if some IO error occurs. + """ + env_map = os.environ if env is None else env + manifest = env_map.get("RUNFILES_MANIFEST_FILE") + if manifest: + return CreateManifestBased(manifest) + + directory = env_map.get("RUNFILES_DIR") + if directory: + return CreateDirectoryBased(directory) + + return None + + +class _Runfiles(object): + """Returns the runtime location of runfiles. + + Runfiles are data-dependencies of Bazel-built binaries and tests. + """ + + def __init__(self, strategy): + self._strategy = strategy + + def Rlocation(self, path): + """Returns the runtime path of a runfile. + + Runfiles are data-dependencies of Bazel-built binaries and tests. + + The returned path may not be valid. The caller should check the path's + validity and that the path exists. + + The function may return None. In that case the caller can be sure that the + rule does not know about this data-dependency. + + Args: + path: string; runfiles-root-relative path of the runfile + Returns: + the path to the runfile, which the caller should check for existence, or + None if the method doesn't know about this runfile + Raises: + TypeError: if `path` is not a string + ValueError: if `path` is None or empty, or it's absolute or not normalized + """ + if not path: + raise ValueError() + if not isinstance(path, str): + raise TypeError() + if (path.startswith("../") or "/.." in path or path.startswith("./") or + "/./" in path or path.endswith("/.") or "//" in path): + raise ValueError("path is not normalized: \"%s\"" % path) + if path[0] == "\\": + raise ValueError("path is absolute without a drive letter: \"%s\"" % path) + if os.path.isabs(path): + return path + return self._strategy.RlocationChecked(path) + + def EnvVars(self): + """Returns environment variables for subprocesses. + + The caller should set the returned key-value pairs in the environment of + subprocesses in case those subprocesses are also Bazel-built binaries that + need to use runfiles. + + Returns: + {string: string}; a dict; keys are environment variable names, values are + the values for these environment variables + """ + return self._strategy.EnvVars() + + +class _ManifestBased(object): + """`Runfiles` strategy that parses a runfiles-manifest to look up runfiles.""" + + def __init__(self, path): + if not path: + raise ValueError() + if not isinstance(path, str): + raise TypeError() + self._path = path + self._runfiles = _ManifestBased._LoadRunfiles(path) + + def RlocationChecked(self, path): + return self._runfiles.get(path) + + @staticmethod + def _LoadRunfiles(path): + """Loads the runfiles manifest.""" + result = {} + with open(path, "r") as f: + for line in f: + line = line.strip() + if line: + tokens = line.split(" ", 1) + if len(tokens) == 1: + result[line] = line + else: + result[tokens[0]] = tokens[1] + return result + + def _GetRunfilesDir(self): + if self._path.endswith("/MANIFEST") or self._path.endswith("\\MANIFEST"): + return self._path[:-len("/MANIFEST")] + elif self._path.endswith(".runfiles_manifest"): + return self._path[:-len("_manifest")] + else: + return "" + + def EnvVars(self): + directory = self._GetRunfilesDir() + return { + "RUNFILES_MANIFEST_FILE": self._path, + "RUNFILES_DIR": directory, + # TODO(laszlocsomor): remove JAVA_RUNFILES once the Java launcher can + # pick up RUNFILES_DIR. + "JAVA_RUNFILES": directory, + } + + +class _DirectoryBased(object): + """`Runfiles` strategy that appends runfiles paths to the runfiles root.""" + + def __init__(self, path): + if not path: + raise ValueError() + if not isinstance(path, str): + raise TypeError() + self._runfiles_root = path + + def RlocationChecked(self, path): + # Use posixpath instead of os.path, because Bazel only creates a runfiles + # tree on Unix platforms, so `Create()` will only create a directory-based + # runfiles strategy on those platforms. + return posixpath.join(self._runfiles_root, path) + + def EnvVars(self): + return { + "RUNFILES_DIR": self._runfiles_root, + # TODO(laszlocsomor): remove JAVA_RUNFILES once the Java launcher can + # pick up RUNFILES_DIR. + "JAVA_RUNFILES": self._runfiles_root, + } + + +def _PathsFrom(argv0, runfiles_mf, runfiles_dir, is_runfiles_manifest, + is_runfiles_directory): + """Discover runfiles manifest and runfiles directory paths. + + Args: + argv0: string; the value of sys.argv[0] + runfiles_mf: string; the value of the RUNFILES_MANIFEST_FILE environment + variable + runfiles_dir: string; the value of the RUNFILES_DIR environment variable + is_runfiles_manifest: lambda(string):bool; returns true if the argument is + the path of a runfiles manifest file + is_runfiles_directory: lambda(string):bool; returns true if the argument is + the path of a runfiles directory + + Returns: + (string, string) pair, first element is the path to the runfiles manifest, + second element is the path to the runfiles directory. If the first element + is non-empty, then is_runfiles_manifest returns true for it. Same goes for + the second element and is_runfiles_directory respectively. If both elements + are empty, then this function could not find a manifest or directory for + which is_runfiles_manifest or is_runfiles_directory returns true. + """ + mf_alid = is_runfiles_manifest(runfiles_mf) + dir_valid = is_runfiles_directory(runfiles_dir) + + if not mf_alid and not dir_valid: + runfiles_mf = argv0 + ".runfiles/MANIFEST" + runfiles_dir = argv0 + ".runfiles" + mf_alid = is_runfiles_manifest(runfiles_mf) + dir_valid = is_runfiles_directory(runfiles_dir) + if not mf_alid: + runfiles_mf = argv0 + ".runfiles_manifest" + mf_alid = is_runfiles_manifest(runfiles_mf) + + if not mf_alid and not dir_valid: + return ("", "") + + if not mf_alid: + runfiles_mf = runfiles_dir + "/MANIFEST" + mf_alid = is_runfiles_manifest(runfiles_mf) + if not mf_alid: + runfiles_mf = runfiles_dir + "_manifest" + mf_alid = is_runfiles_manifest(runfiles_mf) + + if not dir_valid: + runfiles_dir = runfiles_mf[:-9] # "_manifest" or "/MANIFEST" + dir_valid = is_runfiles_directory(runfiles_dir) + + return (runfiles_mf if mf_alid else "", runfiles_dir if dir_valid else "")