refactor(bzlmod)!: simplify pip.parse repository layout (#1395)
Before this PR we would generate extra `alias` repos and the extra `hub`
repo
for the `entry_point` macro usage. This PR removes the extras and
delegates the
creation of version-aware aliases to the `render_pkg_aliases` internal
function. This reduces the number of repositories created by the
`pip.parse`
extension.
Fixes #1255.
BREAKING CHANGE:
Note that this only affects bzlmod support, which is still beta.
* Bzlmod `pip.parse` no longer generates `{hub_name}_{py_version}` hub
repos.
* Bzlmod `pip.parse` no longer generates `{hub_name}_{distribution}` hub
repos.
These repos aren't part of a public API, but were typically used for the
`entry_point`
macros. Instead, use `py_console_script_binary`, which is the supported
replacement
for entry points under bzlmod. Directly referencing the underlying
distribution
repos remains unsupported.diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0e1bf1f..ed3a60d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -58,6 +58,11 @@
* (bzlmod) The `entry_point` macro is no longer supported and has been removed
in favour of the `py_console_script_binary` macro for `bzlmod` users.
+* (bzlmod) The `pip.parse` no longer generates `{hub_name}_{py_version}` hub repos
+ as the `entry_point` macro has been superseded by `py_console_script_binary`.
+
+* (bzlmod) The `pip.parse` no longer generates `{hub_name}_{distribution}` hub repos.
+
### Fixed
* (whl_library) No longer restarts repository rule when fetching external
diff --git a/docs/pip_repository.md b/docs/pip_repository.md
index 8536052..453ca29 100644
--- a/docs/pip_repository.md
+++ b/docs/pip_repository.md
@@ -7,7 +7,7 @@
## pip_hub_repository_bzlmod
<pre>
-pip_hub_repository_bzlmod(<a href="#pip_hub_repository_bzlmod-name">name</a>, <a href="#pip_hub_repository_bzlmod-repo_mapping">repo_mapping</a>, <a href="#pip_hub_repository_bzlmod-repo_name">repo_name</a>, <a href="#pip_hub_repository_bzlmod-whl_library_alias_names">whl_library_alias_names</a>)
+pip_hub_repository_bzlmod(<a href="#pip_hub_repository_bzlmod-name">name</a>, <a href="#pip_hub_repository_bzlmod-default_version">default_version</a>, <a href="#pip_hub_repository_bzlmod-repo_mapping">repo_mapping</a>, <a href="#pip_hub_repository_bzlmod-repo_name">repo_name</a>, <a href="#pip_hub_repository_bzlmod-whl_map">whl_map</a>)
</pre>
A rule for bzlmod mulitple pip repository creation. PRIVATE USE ONLY.
@@ -18,9 +18,10 @@
| Name | Description | Type | Mandatory | Default |
| :------------- | :------------- | :------------- | :------------- | :------------- |
| <a id="pip_hub_repository_bzlmod-name"></a>name | A unique name for this repository. | <a href="https://bazel.build/concepts/labels#target-names">Name</a> | required | |
+| <a id="pip_hub_repository_bzlmod-default_version"></a>default_version | This is the default python version in the format of X.Y.Z. This should match what is setup by the 'python' extension using the 'is_default = True' setting. | String | required | |
| <a id="pip_hub_repository_bzlmod-repo_mapping"></a>repo_mapping | A dictionary from local repository name to global repository name. This allows controls over workspace dependency resolution for dependencies of this repository.<p>For example, an entry <code>"@foo": "@bar"</code> declares that, for any time this repository depends on <code>@foo</code> (such as a dependency on <code>@foo//some:target</code>, it should actually resolve that dependency within globally-declared <code>@bar</code> (<code>@bar//some:target</code>). | <a href="https://bazel.build/rules/lib/dict">Dictionary: String -> String</a> | required | |
| <a id="pip_hub_repository_bzlmod-repo_name"></a>repo_name | The apparent name of the repo. This is needed because in bzlmod, the name attribute becomes the canonical name. | String | required | |
-| <a id="pip_hub_repository_bzlmod-whl_library_alias_names"></a>whl_library_alias_names | The list of whl alias that we use to build aliases and the whl names | List of strings | required | |
+| <a id="pip_hub_repository_bzlmod-whl_map"></a>whl_map | The wheel map where values are python versions | <a href="https://bazel.build/rules/lib/dict">Dictionary: String -> List of strings</a> | required | |
<a id="pip_repository"></a>
@@ -101,31 +102,6 @@
| <a id="pip_repository-timeout"></a>timeout | Timeout (in seconds) on the rule's execution duration. | Integer | optional | <code>600</code> |
-<a id="pip_repository_bzlmod"></a>
-
-## pip_repository_bzlmod
-
-<pre>
-pip_repository_bzlmod(<a href="#pip_repository_bzlmod-name">name</a>, <a href="#pip_repository_bzlmod-repo_mapping">repo_mapping</a>, <a href="#pip_repository_bzlmod-repo_name">repo_name</a>, <a href="#pip_repository_bzlmod-requirements_darwin">requirements_darwin</a>, <a href="#pip_repository_bzlmod-requirements_linux">requirements_linux</a>,
- <a href="#pip_repository_bzlmod-requirements_lock">requirements_lock</a>, <a href="#pip_repository_bzlmod-requirements_windows">requirements_windows</a>)
-</pre>
-
-A rule for bzlmod pip_repository creation. Intended for private use only.
-
-**ATTRIBUTES**
-
-
-| Name | Description | Type | Mandatory | Default |
-| :------------- | :------------- | :------------- | :------------- | :------------- |
-| <a id="pip_repository_bzlmod-name"></a>name | A unique name for this repository. | <a href="https://bazel.build/concepts/labels#target-names">Name</a> | required | |
-| <a id="pip_repository_bzlmod-repo_mapping"></a>repo_mapping | A dictionary from local repository name to global repository name. This allows controls over workspace dependency resolution for dependencies of this repository.<p>For example, an entry <code>"@foo": "@bar"</code> declares that, for any time this repository depends on <code>@foo</code> (such as a dependency on <code>@foo//some:target</code>, it should actually resolve that dependency within globally-declared <code>@bar</code> (<code>@bar//some:target</code>). | <a href="https://bazel.build/rules/lib/dict">Dictionary: String -> String</a> | required | |
-| <a id="pip_repository_bzlmod-repo_name"></a>repo_name | The apparent name of the repo. This is needed because in bzlmod, the name attribute becomes the canonical name | String | required | |
-| <a id="pip_repository_bzlmod-requirements_darwin"></a>requirements_darwin | Override the requirements_lock attribute when the host platform is Mac OS | <a href="https://bazel.build/concepts/labels">Label</a> | optional | <code>None</code> |
-| <a id="pip_repository_bzlmod-requirements_linux"></a>requirements_linux | Override the requirements_lock attribute when the host platform is Linux | <a href="https://bazel.build/concepts/labels">Label</a> | optional | <code>None</code> |
-| <a id="pip_repository_bzlmod-requirements_lock"></a>requirements_lock | A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that wheels are fetched/built only for the targets specified by 'build/run/test'. | <a href="https://bazel.build/concepts/labels">Label</a> | optional | <code>None</code> |
-| <a id="pip_repository_bzlmod-requirements_windows"></a>requirements_windows | Override the requirements_lock attribute when the host platform is Windows | <a href="https://bazel.build/concepts/labels">Label</a> | optional | <code>None</code> |
-
-
<a id="whl_library"></a>
## whl_library
diff --git a/python/extensions/pip.bzl b/python/extensions/pip.bzl
index 3ba0d3e..f94f18c 100644
--- a/python/extensions/pip.bzl
+++ b/python/extensions/pip.bzl
@@ -15,17 +15,16 @@
"pip module extension for use with bzlmod"
load("@pythons_hub//:interpreters.bzl", "DEFAULT_PYTHON_VERSION", "INTERPRETER_LABELS")
-load("//python:pip.bzl", "whl_library_alias")
load(
"//python/pip_install:pip_repository.bzl",
"locked_requirements_label",
"pip_hub_repository_bzlmod",
"pip_repository_attrs",
- "pip_repository_bzlmod",
"use_isolated",
"whl_library",
)
load("//python/pip_install:requirements_parser.bzl", parse_requirements = "parse")
+load("//python/private:full_version.bzl", "full_version")
load("//python/private:normalize_name.bzl", "normalize_name")
load("//python/private:version_label.bzl", "version_label")
@@ -78,11 +77,11 @@
whl_mods = whl_mods,
)
-def _create_versioned_pip_and_whl_repos(module_ctx, pip_attr, whl_map):
+def _create_whl_repos(module_ctx, pip_attr, whl_map):
python_interpreter_target = pip_attr.python_interpreter_target
# if we do not have the python_interpreter set in the attributes
- # we programtically find it.
+ # we programmatically find it.
hub_name = pip_attr.hub_name
if python_interpreter_target == None:
python_name = "python_" + version_label(pip_attr.python_version, sep = "_")
@@ -104,23 +103,12 @@
requrements_lock = locked_requirements_label(module_ctx, pip_attr)
# Parse the requirements file directly in starlark to get the information
- # needed for the whl_libary declarations below. This is needed to contain
- # the pip_repository logic to a single module extension.
+ # needed for the whl_libary declarations below.
requirements_lock_content = module_ctx.read(requrements_lock)
parse_result = parse_requirements(requirements_lock_content)
requirements = parse_result.requirements
extra_pip_args = pip_attr.extra_pip_args + parse_result.options
- # Create the repository where users load the `requirement` macro. Under bzlmod
- # this does not create the install_deps() macro.
- # TODO: we may not need this repository once we have entry points
- # supported. For now a user can access this repository and use
- # the entrypoint functionality.
- pip_repository_bzlmod(
- name = pip_name,
- repo_name = pip_name,
- requirements_lock = pip_attr.requirements_lock,
- )
if hub_name not in whl_map:
whl_map[hub_name] = {}
@@ -157,12 +145,12 @@
if whl_name not in whl_map[hub_name]:
whl_map[hub_name][whl_name] = {}
- whl_map[hub_name][whl_name][pip_attr.python_version] = pip_name + "_"
+ whl_map[hub_name][whl_name][full_version(pip_attr.python_version)] = pip_name + "_"
def _pip_impl(module_ctx):
- """Implementation of a class tag that creates the pip hub(s) and corresponding pip spoke, alias and whl repositories.
+ """Implementation of a class tag that creates the pip hub and corresponding pip spoke whl repositories.
- This implmentation iterates through all of the `pip.parse` calls and creates
+ This implementation iterates through all of the `pip.parse` calls and creates
different pip hub repositories based on the "hub_name". Each of the
pip calls create spoke repos that uses a specific Python interpreter.
@@ -196,52 +184,33 @@
Both of these pip spokes contain requirements files that includes websocket
and its dependencies.
- Two different repositories are created for the two spokes:
-
- - @@rules_python~override~pip~pip_39
- - @@rules_python~override~pip~pip_310
-
- The different spoke names are a combination of the hub_name and the Python version.
- In the future we may remove this repository, but we do not support entry points.
- yet, and that functionality exists in these repos.
-
We also need repositories for the wheels that the different pip spokes contain.
For each Python version a different wheel repository is created. In our example
- each pip spoke had a requirments file that contained websockets. We
+ each pip spoke had a requirements file that contained websockets. We
then create two different wheel repositories that are named the following.
- @@rules_python~override~pip~pip_39_websockets
- @@rules_python~override~pip~pip_310_websockets
- And if the wheel has any other dependies subsequest wheels are created in the same fashion.
+ And if the wheel has any other dependencies subsequent wheels are created in the same fashion.
- We also create a repository for the wheel alias. We want to just use the syntax
- 'requirement("websockets")' we need to have an alias repository that is named:
-
- - @@rules_python~override~pip~pip_websockets
-
- This repository contains alias statements for the different wheel components (pkg, data, etc).
- Each of those aliases has a select that resolves to a spoke repository depending on
- the Python version.
+ The hub repository has aliases for `pkg`, `data`, etc, which have a select that resolves to
+ a spoke repository depending on the Python version.
Also we may have more than one hub as defined in a MODULES.bazel file. So we could have multiple
hubs pointing to various different pip spokes.
- Some other business rules notes. A hub can only have one spoke per Python version. We cannot
+ Some other business rules notes. A hub can only have one spoke per Python version. We cannot
have a hub named "pip" that has two spokes that use the Python 3.9 interpreter. Second
- we cannot have the same hub name used in submodules. The hub name has to be globally
+ we cannot have the same hub name used in sub-modules. The hub name has to be globally
unique.
- This implementation reuses elements of non-bzlmod code and also reuses the first implementation
- of pip bzlmod, but adds the capability to have multiple pip.parse calls.
-
This implementation also handles the creation of whl_modification JSON files that are used
- during the creation of wheel libraries. These JSON files used via the annotations argument
+ during the creation of wheel libraries. These JSON files used via the annotations argument
when calling wheel_installer.py.
Args:
module_ctx: module contents
-
"""
# Build all of the wheel modifications if the tag class is called.
@@ -259,63 +228,46 @@
for mod in module_ctx.modules:
for pip_attr in mod.tags.parse:
hub_name = pip_attr.hub_name
- if hub_name in pip_hub_map:
- # We cannot have two hubs with the same name in different
- # modules.
- if pip_hub_map[hub_name].module_name != mod.name:
- fail((
- "Duplicate cross-module pip hub named '{hub}': pip hub " +
- "names must be unique across modules. First defined " +
- "by module '{first_module}', second attempted by " +
- "module '{second_module}'"
- ).format(
- hub = hub_name,
- first_module = pip_hub_map[hub_name].module_name,
- second_module = mod.name,
- ))
-
- if pip_attr.python_version in pip_hub_map[hub_name].python_versions:
- fail((
- "Duplicate pip python version '{version}' for hub " +
- "'{hub}' in module '{module}': the Python versions " +
- "used for a hub must be unique"
- ).format(
- hub = hub_name,
- module = mod.name,
- version = pip_attr.python_version,
- ))
- else:
- pip_hub_map[pip_attr.hub_name].python_versions.append(pip_attr.python_version)
- else:
+ if hub_name not in pip_hub_map:
pip_hub_map[pip_attr.hub_name] = struct(
module_name = mod.name,
python_versions = [pip_attr.python_version],
)
+ elif pip_hub_map[hub_name].module_name != mod.name:
+ # We cannot have two hubs with the same name in different
+ # modules.
+ fail((
+ "Duplicate cross-module pip hub named '{hub}': pip hub " +
+ "names must be unique across modules. First defined " +
+ "by module '{first_module}', second attempted by " +
+ "module '{second_module}'"
+ ).format(
+ hub = hub_name,
+ first_module = pip_hub_map[hub_name].module_name,
+ second_module = mod.name,
+ ))
- _create_versioned_pip_and_whl_repos(module_ctx, pip_attr, hub_whl_map)
+ elif pip_attr.python_version in pip_hub_map[hub_name].python_versions:
+ fail((
+ "Duplicate pip python version '{version}' for hub " +
+ "'{hub}' in module '{module}': the Python versions " +
+ "used for a hub must be unique"
+ ).format(
+ hub = hub_name,
+ module = mod.name,
+ version = pip_attr.python_version,
+ ))
+ else:
+ pip_hub_map[pip_attr.hub_name].python_versions.append(pip_attr.python_version)
+
+ _create_whl_repos(module_ctx, pip_attr, hub_whl_map)
for hub_name, whl_map in hub_whl_map.items():
- for whl_name, version_map in whl_map.items():
- if DEFAULT_PYTHON_VERSION in version_map:
- whl_default_version = DEFAULT_PYTHON_VERSION
- else:
- whl_default_version = None
-
- # Create the alias repositories which contains different select
- # statements These select statements point to the different pip
- # whls that are based on a specific version of Python.
- whl_library_alias(
- name = hub_name + "_" + whl_name,
- wheel_name = whl_name,
- default_version = whl_default_version,
- version_map = version_map,
- )
-
- # Create the hub repository for pip.
pip_hub_repository_bzlmod(
name = hub_name,
repo_name = hub_name,
- whl_library_alias_names = whl_map.keys(),
+ whl_map = whl_map,
+ default_version = full_version(DEFAULT_PYTHON_VERSION),
)
def _pip_parse_ext_attrs():
diff --git a/python/pip_install/pip_hub_repository_requirements_bzlmod.bzl.tmpl b/python/pip_install/pip_hub_repository_requirements_bzlmod.bzl.tmpl
deleted file mode 100644
index 53d4ee9..0000000
--- a/python/pip_install/pip_hub_repository_requirements_bzlmod.bzl.tmpl
+++ /dev/null
@@ -1,29 +0,0 @@
-"""Starlark representation of locked requirements.
-
-@generated by rules_python pip_parse repository rule
-from %%REQUIREMENTS_LOCK%%.
-
-This file is different from the other bzlmod template
-because we do not support entry_point yet.
-"""
-
-all_requirements = %%ALL_REQUIREMENTS%%
-
-all_whl_requirements = %%ALL_WHL_REQUIREMENTS%%
-
-all_data_requirements = %%ALL_DATA_REQUIREMENTS%%
-
-def _clean_name(name):
- return name.replace("-", "_").replace(".", "_").lower()
-
-def requirement(name):
- return "%%MACRO_TMPL%%".format(_clean_name(name), "pkg")
-
-def whl_requirement(name):
- return "%%MACRO_TMPL%%".format(_clean_name(name), "whl")
-
-def data_requirement(name):
- return "%%MACRO_TMPL%%".format(_clean_name(name), "data")
-
-def dist_info_requirement(name):
- return "%%MACRO_TMPL%%".format(_clean_name(name), "dist_info")
diff --git a/python/pip_install/pip_repository.bzl b/python/pip_install/pip_repository.bzl
index abe3ca7..ea8b9eb 100644
--- a/python/pip_install/pip_repository.bzl
+++ b/python/pip_install/pip_repository.bzl
@@ -267,10 +267,14 @@
""")
return requirements_txt
-def _create_pip_repository_bzlmod(rctx, bzl_packages, requirements):
- repo_name = rctx.attr.repo_name
- build_contents = _BUILD_FILE_CONTENTS
- aliases = render_pkg_aliases(repo_name = repo_name, bzl_packages = bzl_packages)
+def _pip_hub_repository_bzlmod_impl(rctx):
+ bzl_packages = rctx.attr.whl_map.keys()
+ aliases = render_pkg_aliases(
+ repo_name = rctx.attr.repo_name,
+ rules_python = rctx.attr._template.workspace_name,
+ default_version = rctx.attr.default_version,
+ whl_map = rctx.attr.whl_map,
+ )
for path, contents in aliases.items():
rctx.file(path, contents)
@@ -280,7 +284,7 @@
# `requirement`, et al. macros.
macro_tmpl = "@@{name}//{{}}:{{}}".format(name = rctx.attr.name)
- rctx.file("BUILD.bazel", build_contents)
+ rctx.file("BUILD.bazel", _BUILD_FILE_CONTENTS)
rctx.template("requirements.bzl", rctx.attr._template, substitutions = {
"%%ALL_DATA_REQUIREMENTS%%": _format_repr_list([
macro_tmpl.format(p, "data")
@@ -296,24 +300,26 @@
]),
"%%MACRO_TMPL%%": macro_tmpl,
"%%NAME%%": rctx.attr.name,
- "%%REQUIREMENTS_LOCK%%": requirements,
})
-def _pip_hub_repository_bzlmod_impl(rctx):
- bzl_packages = rctx.attr.whl_library_alias_names
- _create_pip_repository_bzlmod(rctx, bzl_packages, "")
-
pip_hub_repository_bzlmod_attrs = {
+ "default_version": attr.string(
+ mandatory = True,
+ doc = """\
+This is the default python version in the format of X.Y.Z. This should match
+what is setup by the 'python' extension using the 'is_default = True'
+setting.""",
+ ),
"repo_name": attr.string(
mandatory = True,
doc = "The apparent name of the repo. This is needed because in bzlmod, the name attribute becomes the canonical name.",
),
- "whl_library_alias_names": attr.string_list(
+ "whl_map": attr.string_list_dict(
mandatory = True,
- doc = "The list of whl alias that we use to build aliases and the whl names",
+ doc = "The wheel map where values are python versions",
),
"_template": attr.label(
- default = ":pip_hub_repository_requirements_bzlmod.bzl.tmpl",
+ default = ":pip_repository_requirements_bzlmod.bzl.tmpl",
),
}
@@ -323,52 +329,6 @@
implementation = _pip_hub_repository_bzlmod_impl,
)
-def _pip_repository_bzlmod_impl(rctx):
- requirements_txt = locked_requirements_label(rctx, rctx.attr)
- content = rctx.read(requirements_txt)
- parsed_requirements_txt = parse_requirements(content)
-
- packages = [(normalize_name(name), requirement) for name, requirement in parsed_requirements_txt.requirements]
-
- bzl_packages = sorted([name for name, _ in packages])
- _create_pip_repository_bzlmod(rctx, bzl_packages, str(requirements_txt))
-
-pip_repository_bzlmod_attrs = {
- "repo_name": attr.string(
- mandatory = True,
- doc = "The apparent name of the repo. This is needed because in bzlmod, the name attribute becomes the canonical name",
- ),
- "requirements_darwin": attr.label(
- allow_single_file = True,
- doc = "Override the requirements_lock attribute when the host platform is Mac OS",
- ),
- "requirements_linux": attr.label(
- allow_single_file = True,
- doc = "Override the requirements_lock attribute when the host platform is Linux",
- ),
- "requirements_lock": attr.label(
- allow_single_file = True,
- doc = """
-A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead
-of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that
-wheels are fetched/built only for the targets specified by 'build/run/test'.
-""",
- ),
- "requirements_windows": attr.label(
- allow_single_file = True,
- doc = "Override the requirements_lock attribute when the host platform is Windows",
- ),
- "_template": attr.label(
- default = ":pip_repository_requirements_bzlmod.bzl.tmpl",
- ),
-}
-
-pip_repository_bzlmod = repository_rule(
- attrs = pip_repository_bzlmod_attrs,
- doc = """A rule for bzlmod pip_repository creation. Intended for private use only.""",
- implementation = _pip_repository_bzlmod_impl,
-)
-
def _pip_repository_impl(rctx):
requirements_txt = locked_requirements_label(rctx, rctx.attr)
content = rctx.read(requirements_txt)
diff --git a/python/pip_install/pip_repository_requirements_bzlmod.bzl.tmpl b/python/pip_install/pip_repository_requirements_bzlmod.bzl.tmpl
index 00580f5..c72187c 100644
--- a/python/pip_install/pip_repository_requirements_bzlmod.bzl.tmpl
+++ b/python/pip_install/pip_repository_requirements_bzlmod.bzl.tmpl
@@ -1,7 +1,6 @@
"""Starlark representation of locked requirements.
-@generated by rules_python pip_parse repository rule
-from %%REQUIREMENTS_LOCK%%.
+@generated by rules_python pip.parse bzlmod extension.
"""
all_requirements = %%ALL_REQUIREMENTS%%
