Add pip3_import (#256)
This adds `pip3_import` as a wrapper around `pip_import` that sets `python_interpreter` to `"python3"`. This is important for requesting the Python 3 version of a package on systems where the `"python"` command is Python 2 (which is most of them).
We decline to add an analogous `pip2_import` wrapper at this time, because the command `"python2"` does not exist on all platforms by default (e.g. macOS).
`piptool.py` is updated to prefix the names of the wheel repos (an implementation
detail of rules_python) with the name given to `pip_import`. This is needed to
avoid shadowing wheel repos when the same wheel name is used by separate
`pip_import` invocations -- in particular when the same wheel is used for both
PY2 and PY3. (Thanks to @joshclimacell for pointing this detail out in
his prototype 90a70d5a550a01646966ce61156e6f83f02b4b73.)
Regenerated the .par files and docs.
Also updated the README to better explain the structure of the packaging rules. This includes mentioning `pip3_import`, concerns around versioning / hermeticity, and not depending on the wheel repo names (use `requirement()` instead).
diff --git a/README.md b/README.md
index ccb80d3..ae63f0f 100644
--- a/README.md
+++ b/README.md
@@ -5,7 +5,17 @@
## Recent updates
-* 2019-07-26: The canonical name of this repo has been changed from `@io_bazel_rules_python` to just `@rules_python`, in accordance with [convention](https://docs.bazel.build/versions/master/skylark/deploying.html#workspace). Please update your WORKSPACE file and labels that reference this repo accordingly.
+* 2019-11-15: Added support for `pip3_import` (and more generally, a
+`python_interpreter` attribute to `pip_import`). The canonical naming for wheel
+repositories has changed to accomodate loading wheels for both `pip_import` and
+`pip3_import` in the same build. To avoid breakage, please use `requirement()`
+instead of depending directly on wheel repo labels.
+
+* 2019-07-26: The canonical name of this repo has been changed from
+`@io_bazel_rules_python` to just `@rules_python`, in accordance with
+[convention](https://docs.bazel.build/versions/master/skylark/deploying.html#workspace).
+Please update your `WORKSPACE` file and labels that reference this repo
+accordingly.
## Overview
@@ -95,7 +105,7 @@
# above.
```
-Once you've imported the rule set into your WORKSPACE using any of these
+Once you've imported the rule set into your `WORKSPACE` using any of these
methods, you can then load the core rules in your `BUILD` files with:
``` python
@@ -109,35 +119,67 @@
## Using the packaging rules
+The packaging rules create two kinds of repositories: A central repo that holds
+downloaded wheel files, and individual repos for each wheel's extracted
+contents. Users only need to interact with the central repo; the wheel repos
+are essentially an implementation detail. The central repo provides a
+`WORKSPACE` macro to create the wheel repos, as well as a function to call in
+`BUILD` files to translate a pip package name into the label of a `py_library`
+target in the appropriate wheel repo.
+
### Importing `pip` dependencies
-The packaging rules are designed to have developers continue using
-`requirements.txt` to express their dependencies in a Python idiomatic manner.
-These dependencies are imported into the Bazel dependency graph via a
-two-phased process in `WORKSPACE`:
+Adding pip dependencies to your `WORKSPACE` is a two-step process. First you
+declare the central repository using `pip_import`, which invokes pip to read
+a `requirements.txt` file and download the appropriate wheels. Then you load
+the `pip_install` function from the central repo, and call it to create the
+individual wheel repos.
+
+**Important:** If you are using Python 3, load and call `pip3_import` instead.
```python
load("@rules_python//python:pip.bzl", "pip_import")
-# This rule translates the specified requirements.txt into
-# @my_deps//:requirements.bzl, which itself exposes a pip_install method.
-pip_import(
+# Create a central repo that knows about the dependencies needed for
+# requirements.txt.
+pip_import( # or pip3_import
name = "my_deps",
requirements = "//path/to:requirements.txt",
)
-# Load the pip_install symbol for my_deps, and create the dependencies'
-# repositories.
+# Load the central repo's install function from its `//:requirements.bzl` file,
+# and call it.
load("@my_deps//:requirements.bzl", "pip_install")
pip_install()
```
+Note that since pip is executed at WORKSPACE-evaluation time, Bazel has no
+information about the Python toolchain and cannot enforce that the interpreter
+used to invoke pip matches the interpreter used to run `py_binary` targets. By
+default, `pip_import` uses the system command `"python"`, which on most
+platforms is a Python 2 interpreter. This can be overridden by passing the
+`python_interpreter` attribute to `pip_import`. `pip3_import` just acts as a
+wrapper that sets `python_interpreter` to `"python3"`.
+
+You can have multiple `pip_import`s in the same workspace, e.g. for Python 2
+and Python 3. This will create multiple central repos that have no relation to
+one another, and may result in downloading the same wheels multiple times.
+
+As with any repository rule, if you would like to ensure that `pip_import` is
+reexecuted in order to pick up a non-hermetic change to your environment (e.g.,
+updating your system `python` interpreter), you can completely flush out your
+repo cache with `bazel clean --expunge`.
+
### Consuming `pip` dependencies
-Once a set of dependencies has been imported via `pip_import` and `pip_install`
-we can start consuming them in our `py_{binary,library,test}` rules. In support
-of this, the generated `requirements.bzl` also contains a `requirement` method,
-which can be used directly in `deps=[]` to reference an imported `py_library`.
+Each extracted wheel repo contains a `py_library` target representing the
+wheel's contents. Rather than depend on this target's label directly -- which
+would require hardcoding the wheel repo's mangled name into your BUILD files --
+you should instead use the `requirement()` function defined in the central
+repo's `//:requirements.bzl` file. This function maps a pip package name to a
+label. (["Extras"](
+https://packaging.python.org/tutorials/installing-packages/#installing-setuptools-extras)
+can be referenced using the `pkg[extra]` syntax.)
```python
load("@my_deps//:requirements.bzl", "requirement")
@@ -147,38 +189,28 @@
srcs = ["mylib.py"],
deps = [
":myotherlib",
- # This takes the name as specified in requirements.txt
- requirement("importeddep"),
+ requirement("some_pip_dep"),
+ requirement("anohter_pip_dep[some_extra]"),
]
)
```
-### Canonical `whl_library` naming
-
-It is notable that `whl_library` rules imported via `pip_import` are canonically
-named, following the pattern: `pypi__{distribution}_{version}`. Characters in
-these components that are illegal in Bazel label names (e.g. `-`, `.`) are
-replaced with `_`.
-
-This canonical naming helps avoid redundant work to import the same library
-multiple times. It is expected that this naming will remain stable, so folks
-should be able to reliably depend directly on e.g. `@pypi__futures_3_1_1//:pkg`
-for dependencies, however, it is recommended that folks stick with the
-`requirement` pattern in case the need arises for us to make changes to this
-format in the future.
-
-["Extras"](
-https://packaging.python.org/tutorials/installing-packages/#installing-setuptools-extras)
-will have a target of the extra name (in place of `pkg` above).
+For reference, the wheel repos are canonically named following the pattern:
+`@{central_repo_name}_pypi__{distribution}_{version}`. Characters in the
+distribution and version that are illegal in Bazel label names (e.g. `-`, `.`)
+are replaced with `_`. While this naming pattern doesn't change often, it is
+not guaranted to remain stable, so use of the `requirement()` function is
+recommended.
## Migrating from the bundled rules
The core rules are currently available in Bazel as built-in symbols, but this
form is deprecated. Instead, you should depend on rules_python in your
-WORKSPACE file and load the Python rules from `@rules_python//python:defs.bzl`.
+`WORKSPACE` file and load the Python rules from
+`@rules_python//python:defs.bzl`.
A [buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md)
-fix is available to automatically migrate BUILD and .bzl files to add the
+fix is available to automatically migrate `BUILD` and `.bzl` files to add the
appropriate `load()` statements and rewrite uses of `native.py_*`.
```sh
@@ -186,7 +218,7 @@
buildifier --lint=fix --warnings=native-py <files>
```
-Currently the WORKSPACE file needs to be updated manually as per [Getting
+Currently the `WORKSPACE` file needs to be updated manually as per [Getting
started](#Getting-started) above.
Note that Starlark-defined bundled symbols underneath
diff --git a/docs/pip.md b/docs/pip.md
index 54aa476..037bdfa 100755
--- a/docs/pip.md
+++ b/docs/pip.md
@@ -90,6 +90,36 @@
</table>
+<a name="#pip3_import"></a>
+
+## pip3_import
+
+<pre>
+pip3_import(<a href="#pip3_import-kwargs">kwargs</a>)
+</pre>
+
+A wrapper around pip_import that uses the `python3` system command.
+
+Use this for requirements of PY3 programs.
+
+### Parameters
+
+<table class="params-table">
+ <colgroup>
+ <col class="col-param" />
+ <col class="col-description" />
+ </colgroup>
+ <tbody>
+ <tr id="pip3_import-kwargs">
+ <td><code>kwargs</code></td>
+ <td>
+ optional.
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+
<a name="#pip_repositories"></a>
## pip_repositories
diff --git a/packaging/piptool.py b/packaging/piptool.py
index ac78bac..766a674 100644
--- a/packaging/piptool.py
+++ b/packaging/piptool.py
@@ -174,6 +174,9 @@
whls = [Wheel(path) for path in list_whls()]
possible_extras = determine_possible_extras(whls)
+ def repository_name(wheel):
+ return args.name + "_" + wheel.repository_suffix()
+
def whl_library(wheel):
# Indentation here matters. whl_library must be within the scope
# of the function below. We also avoid reimporting an existing WHL.
@@ -185,7 +188,7 @@
whl = "@{name}//:{path}",
requirements = "@{name}//:requirements.bzl",
extras = [{extras}]
- )""".format(name=args.name, repo_name=wheel.repository_name(),
+ )""".format(name=args.name, repo_name=repository_name(wheel),
python_interpreter=args.python_interpreter,
path=wheel.basename(),
extras=','.join([
@@ -195,11 +198,11 @@
whl_targets = ','.join([
','.join([
- '"%s": "@%s//:pkg"' % (whl.distribution().lower(), whl.repository_name())
+ '"%s": "@%s//:pkg"' % (whl.distribution().lower(), repository_name(whl))
] + [
# For every extra that is possible from this requirements.txt
'"%s[%s]": "@%s//:%s"' % (whl.distribution().lower(), extra.lower(),
- whl.repository_name(), extra)
+ repository_name(whl), extra)
for extra in possible_extras.get(whl, [])
])
for whl in whls
diff --git a/packaging/whl.py b/packaging/whl.py
index 63a6d62..c140a13 100644
--- a/packaging/whl.py
+++ b/packaging/whl.py
@@ -42,8 +42,9 @@
parts = self.basename().split('-')
return parts[1]
- def repository_name(self):
- # Returns the canonical name of the Bazel repository for this package.
+ def repository_suffix(self):
+ # Returns a canonical suffix that will form part of the name of the Bazel
+ # repository for this package.
canonical = 'pypi__{}_{}'.format(self.distribution(), self.version())
# Escape any illegal characters with underscore.
return re.sub('[-.+]', '_', canonical)
diff --git a/packaging/whl_test.py b/packaging/whl_test.py
index 2af8b9b..1a29f5b 100644
--- a/packaging/whl_test.py
+++ b/packaging/whl_test.py
@@ -34,7 +34,7 @@
self.assertEqual(wheel.version(), '1.6.0')
self.assertEqual(set(wheel.dependencies()),
set(['enum34', 'futures', 'protobuf', 'six']))
- self.assertEqual('pypi__grpcio_1_6_0', wheel.repository_name())
+ self.assertEqual('pypi__grpcio_1_6_0', wheel.repository_suffix())
self.assertEqual([], wheel.extras())
def test_futures_whl(self):
@@ -44,7 +44,7 @@
self.assertEqual(wheel.distribution(), 'futures')
self.assertEqual(wheel.version(), '3.1.1')
self.assertEqual(set(wheel.dependencies()), set())
- self.assertEqual('pypi__futures_3_1_1', wheel.repository_name())
+ self.assertEqual('pypi__futures_3_1_1', wheel.repository_suffix())
self.assertEqual([], wheel.extras())
def test_whl_with_METADATA_file(self):
@@ -54,7 +54,7 @@
self.assertEqual(wheel.distribution(), 'futures')
self.assertEqual(wheel.version(), '2.2.0')
self.assertEqual(set(wheel.dependencies()), set())
- self.assertEqual('pypi__futures_2_2_0', wheel.repository_name())
+ self.assertEqual('pypi__futures_2_2_0', wheel.repository_suffix())
@patch('platform.python_version', return_value='2.7.13')
def test_mock_whl(self, *args):
@@ -65,7 +65,7 @@
self.assertEqual(wheel.version(), '2.0.0')
self.assertEqual(set(wheel.dependencies()),
set(['funcsigs', 'pbr', 'six']))
- self.assertEqual('pypi__mock_2_0_0', wheel.repository_name())
+ self.assertEqual('pypi__mock_2_0_0', wheel.repository_suffix())
@patch('platform.python_version', return_value='3.3.0')
def test_mock_whl_3_3(self, *args):
@@ -103,7 +103,7 @@
self.assertEqual(set(wheel.dependencies()),
set(expected_deps))
self.assertEqual('pypi__google_cloud_language_0_29_0',
- wheel.repository_name())
+ wheel.repository_suffix())
self.assertEqual([], wheel.extras())
@patch('platform.python_version', return_value='3.4.0')
diff --git a/python/pip.bzl b/python/pip.bzl
index 231c723..37da2a2 100644
--- a/python/pip.bzl
+++ b/python/pip.bzl
@@ -102,6 +102,17 @@
""",
)
+# We don't provide a `pip2_import` that would use the `python2` system command
+# because this command does not exist on all platforms. On most (but not all)
+# systems, `python` means Python 2 anyway. See also #258.
+
+def pip3_import(**kwargs):
+ """A wrapper around pip_import that uses the `python3` system command.
+
+ Use this for requirements of PY3 programs.
+ """
+ pip_import(python_interpreter = "python3", **kwargs)
+
def pip_repositories():
"""Pull in dependencies needed to use the packaging rules."""
diff --git a/tools/piptool.par b/tools/piptool.par
index d5dce0c..57ef98d 100755
--- a/tools/piptool.par
+++ b/tools/piptool.par
Binary files differ
diff --git a/tools/whltool.par b/tools/whltool.par
index c7c0212..964963f 100755
--- a/tools/whltool.par
+++ b/tools/whltool.par
Binary files differ
