refactor: starlark reimplementation of pip_repository (#1043)
Previously we were using pip_parse python scripts. This has a few drawbacks:
* Requires system python to be present.
* Usage of a Python script makes it harder to reason as there is an extra layer of abstraction.
* Extending/reusing code between multi_pip_parse and pip_parse is hard.
Now we use Starlark to parse the requirements.txt into requirements.bzl.
diff --git a/docs/pip.md b/docs/pip.md
index 2f5b92e..528abf7 100644
--- a/docs/pip.md
+++ b/docs/pip.md
@@ -168,7 +168,7 @@
## pip_parse
<pre>
-pip_parse(<a href="#pip_parse-requirements">requirements</a>, <a href="#pip_parse-requirements_lock">requirements_lock</a>, <a href="#pip_parse-name">name</a>, <a href="#pip_parse-bzlmod">bzlmod</a>, <a href="#pip_parse-kwargs">kwargs</a>)
+pip_parse(<a href="#pip_parse-requirements">requirements</a>, <a href="#pip_parse-requirements_lock">requirements_lock</a>, <a href="#pip_parse-name">name</a>, <a href="#pip_parse-kwargs">kwargs</a>)
</pre>
Accepts a locked/compiled requirements file and installs the dependencies listed within.
@@ -264,7 +264,6 @@
| <a id="pip_parse-requirements"></a>requirements | Deprecated. See requirements_lock. | <code>None</code> |
| <a id="pip_parse-requirements_lock"></a>requirements_lock | A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that wheels are fetched/built only for the targets specified by 'build/run/test'. Note that if your lockfile is platform-dependent, you can use the <code>requirements_[platform]</code> attributes. | <code>None</code> |
| <a id="pip_parse-name"></a>name | The name of the generated repository. The generated repositories containing each requirement will be of the form <code><name>_<requirement-name></code>. | <code>"pip_parsed_deps"</code> |
-| <a id="pip_parse-bzlmod"></a>bzlmod | Whether this rule is being run under a bzlmod module extension. | <code>False</code> |
| <a id="pip_parse-kwargs"></a>kwargs | Additional arguments to the [<code>pip_repository</code>](./pip_repository.md) repository rule. | none |
diff --git a/docs/pip_repository.md b/docs/pip_repository.md
index 7abb503..2ccdc64 100644
--- a/docs/pip_repository.md
+++ b/docs/pip_repository.md
@@ -7,8 +7,8 @@
## pip_repository
<pre>
-pip_repository(<a href="#pip_repository-name">name</a>, <a href="#pip_repository-annotations">annotations</a>, <a href="#pip_repository-bzlmod">bzlmod</a>, <a href="#pip_repository-download_only">download_only</a>, <a href="#pip_repository-enable_implicit_namespace_pkgs">enable_implicit_namespace_pkgs</a>,
- <a href="#pip_repository-environment">environment</a>, <a href="#pip_repository-extra_pip_args">extra_pip_args</a>, <a href="#pip_repository-isolated">isolated</a>, <a href="#pip_repository-pip_data_exclude">pip_data_exclude</a>, <a href="#pip_repository-python_interpreter">python_interpreter</a>,
+pip_repository(<a href="#pip_repository-name">name</a>, <a href="#pip_repository-annotations">annotations</a>, <a href="#pip_repository-download_only">download_only</a>, <a href="#pip_repository-enable_implicit_namespace_pkgs">enable_implicit_namespace_pkgs</a>, <a href="#pip_repository-environment">environment</a>,
+ <a href="#pip_repository-extra_pip_args">extra_pip_args</a>, <a href="#pip_repository-isolated">isolated</a>, <a href="#pip_repository-pip_data_exclude">pip_data_exclude</a>, <a href="#pip_repository-python_interpreter">python_interpreter</a>,
<a href="#pip_repository-python_interpreter_target">python_interpreter_target</a>, <a href="#pip_repository-quiet">quiet</a>, <a href="#pip_repository-repo_mapping">repo_mapping</a>, <a href="#pip_repository-repo_prefix">repo_prefix</a>, <a href="#pip_repository-requirements_darwin">requirements_darwin</a>,
<a href="#pip_repository-requirements_linux">requirements_linux</a>, <a href="#pip_repository-requirements_lock">requirements_lock</a>, <a href="#pip_repository-requirements_windows">requirements_windows</a>, <a href="#pip_repository-timeout">timeout</a>)
</pre>
@@ -60,7 +60,6 @@
| :------------- | :------------- | :------------- | :------------- | :------------- |
| <a id="pip_repository-name"></a>name | A unique name for this repository. | <a href="https://bazel.build/concepts/labels#target-names">Name</a> | required | |
| <a id="pip_repository-annotations"></a>annotations | Optional annotations to apply to packages | <a href="https://bazel.build/rules/lib/dict">Dictionary: String -> String</a> | optional | <code>{}</code> |
-| <a id="pip_repository-bzlmod"></a>bzlmod | Whether this repository rule is invoked under bzlmod, in which case we do not create the install_deps() macro. | Boolean | optional | <code>False</code> |
| <a id="pip_repository-download_only"></a>download_only | Whether to use "pip download" instead of "pip wheel". Disables building wheels from source, but allows use of --platform, --python-version, --implementation, and --abi in --extra_pip_args to download wheels for a different platform from the host platform. | Boolean | optional | <code>False</code> |
| <a id="pip_repository-enable_implicit_namespace_pkgs"></a>enable_implicit_namespace_pkgs | If true, disables conversion of native namespace packages into pkg-util style namespace packages. When set all py_binary and py_test targets must specify either <code>legacy_create_init=False</code> or the global Bazel option <code>--incompatible_default_to_explicit_init_py</code> to prevent <code>__init__.py</code> being automatically generated in every directory.<br><br>This option is required to support some packages which cannot handle the conversion to pkg-util style. | Boolean | optional | <code>False</code> |
| <a id="pip_repository-environment"></a>environment | Environment variables to set in the pip subprocess. Can be used to set common variables such as <code>http_proxy</code>, <code>https_proxy</code> and <code>no_proxy</code> Note that pip is run with "--isolated" on the CLI so <code>PIP_<VAR>_<NAME></code> style env vars are ignored, but env vars that control requests and urllib3 can be passed. | <a href="https://bazel.build/rules/lib/dict">Dictionary: String -> String</a> | optional | <code>{}</code> |
@@ -79,6 +78,30 @@
| <a id="pip_repository-timeout"></a>timeout | Timeout (in seconds) on the rule's execution duration. | Integer | optional | <code>600</code> |
+<a id="pip_repository_bzlmod"></a>
+
+## pip_repository_bzlmod
+
+<pre>
+pip_repository_bzlmod(<a href="#pip_repository_bzlmod-name">name</a>, <a href="#pip_repository_bzlmod-repo_mapping">repo_mapping</a>, <a href="#pip_repository_bzlmod-requirements_darwin">requirements_darwin</a>, <a href="#pip_repository_bzlmod-requirements_linux">requirements_linux</a>,
+ <a href="#pip_repository_bzlmod-requirements_lock">requirements_lock</a>, <a href="#pip_repository_bzlmod-requirements_windows">requirements_windows</a>)
+</pre>
+
+A rule for bzlmod pip_repository creation. Intended for private use only.
+
+**ATTRIBUTES**
+
+
+| Name | Description | Type | Mandatory | Default |
+| :------------- | :------------- | :------------- | :------------- | :------------- |
+| <a id="pip_repository_bzlmod-name"></a>name | A unique name for this repository. | <a href="https://bazel.build/concepts/labels#target-names">Name</a> | required | |
+| <a id="pip_repository_bzlmod-repo_mapping"></a>repo_mapping | A dictionary from local repository name to global repository name. This allows controls over workspace dependency resolution for dependencies of this repository.<p>For example, an entry <code>"@foo": "@bar"</code> declares that, for any time this repository depends on <code>@foo</code> (such as a dependency on <code>@foo//some:target</code>, it should actually resolve that dependency within globally-declared <code>@bar</code> (<code>@bar//some:target</code>). | <a href="https://bazel.build/rules/lib/dict">Dictionary: String -> String</a> | required | |
+| <a id="pip_repository_bzlmod-requirements_darwin"></a>requirements_darwin | Override the requirements_lock attribute when the host platform is Mac OS | <a href="https://bazel.build/concepts/labels">Label</a> | optional | <code>None</code> |
+| <a id="pip_repository_bzlmod-requirements_linux"></a>requirements_linux | Override the requirements_lock attribute when the host platform is Linux | <a href="https://bazel.build/concepts/labels">Label</a> | optional | <code>None</code> |
+| <a id="pip_repository_bzlmod-requirements_lock"></a>requirements_lock | A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that wheels are fetched/built only for the targets specified by 'build/run/test'. | <a href="https://bazel.build/concepts/labels">Label</a> | optional | <code>None</code> |
+| <a id="pip_repository_bzlmod-requirements_windows"></a>requirements_windows | Override the requirements_lock attribute when the host platform is Windows | <a href="https://bazel.build/concepts/labels">Label</a> | optional | <code>None</code> |
+
+
<a id="whl_library"></a>
## whl_library
diff --git a/examples/pip_parse_vendored/BUILD.bazel b/examples/pip_parse_vendored/BUILD.bazel
index 9585195..56630e5 100644
--- a/examples/pip_parse_vendored/BUILD.bazel
+++ b/examples/pip_parse_vendored/BUILD.bazel
@@ -20,7 +20,6 @@
# Replace the bazel 6.0.0 specific comment with something that bazel 5.4.0 would produce.
# This enables this example to be run as a test under bazel 5.4.0.
"""sed -e 's#@//#//#'""",
- """tr "'" '"' """,
"""sed 's#"@python39_.*//:bin/python3"#interpreter#' >$@""",
]),
)
diff --git a/examples/pip_parse_vendored/requirements.bzl b/examples/pip_parse_vendored/requirements.bzl
index e13503a..015df93 100644
--- a/examples/pip_parse_vendored/requirements.bzl
+++ b/examples/pip_parse_vendored/requirements.bzl
@@ -14,29 +14,20 @@
_packages = [("pip_certifi", "certifi==2022.12.7 --hash=sha256:35824b4c3a97115964b408844d64aa14db1cc518f6562e8d7261699d1350a9e3 --hash=sha256:4ad3232f5e926d6718ec31cfc1fcadfde020920e278684144551c91769c7bc18"), ("pip_charset_normalizer", "charset-normalizer==2.1.1 --hash=sha256:5a3d016c7c547f69d6f81fb0db9449ce888b418b5b9952cc5e6e66843e9dd845 --hash=sha256:83e9a75d1911279afd89352c68b45348559d1fc0506b054b346651b5e7fee29f"), ("pip_idna", "idna==3.4 --hash=sha256:814f528e8dead7d329833b91c5faa87d60bf71824cd12a7530b5526063d02cb4 --hash=sha256:90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2"), ("pip_requests", "requests==2.28.1 --hash=sha256:7c5599b102feddaa661c826c56ab4fee28bfd17f5abca1ebbe3e7f19d7c97983 --hash=sha256:8fefa2a1a1365bf5520aac41836fbee479da67864514bdb821f31ce07ce65349"), ("pip_urllib3", "urllib3==1.26.13 --hash=sha256:47cc05d99aaa09c9e72ed5809b60e7ba354e64b59c9c173ac3018642d8bb41fc --hash=sha256:c083dd0dce68dbfbe1129d5271cb90f9447dea7d52097c6e0126120c521ddea8")]
_config = {"download_only": False, "enable_implicit_namespace_pkgs": False, "environment": {}, "extra_pip_args": [], "isolated": True, "pip_data_exclude": [], "python_interpreter": "python3", "python_interpreter_target": interpreter, "quiet": True, "repo": "pip", "repo_prefix": "pip_", "timeout": 600}
_annotations = {}
-_bzlmod = False
def _clean_name(name):
return name.replace("-", "_").replace(".", "_").lower()
def requirement(name):
- if _bzlmod:
- return "@@pip//:" + _clean_name(name) + "_pkg"
return "@pip_" + _clean_name(name) + "//:pkg"
def whl_requirement(name):
- if _bzlmod:
- return "@@pip//:" + _clean_name(name) + "_whl"
return "@pip_" + _clean_name(name) + "//:whl"
def data_requirement(name):
- if _bzlmod:
- return "@@pip//:" + _clean_name(name) + "_data"
return "@pip_" + _clean_name(name) + "//:data"
def dist_info_requirement(name):
- if _bzlmod:
- return "@@pip//:" + _clean_name(name) + "_dist_info"
return "@pip_" + _clean_name(name) + "//:dist_info"
def entry_point(pkg, script = None):
@@ -46,7 +37,7 @@
def _get_annotation(requirement):
# This expects to parse `setuptools==58.2.0 --hash=sha256:2551203ae6955b9876741a26ab3e767bb3242dafe86a32a749ea0d78b6792f11`
- # down wo `setuptools`.
+ # down to `setuptools`.
name = requirement.split(" ")[0].split("=")[0].split("[")[0]
return _annotations.get(name)
diff --git a/python/extensions.bzl b/python/extensions.bzl
index bc0d570..01f731f 100644
--- a/python/extensions.bzl
+++ b/python/extensions.bzl
@@ -14,9 +14,8 @@
"Module extensions for use with bzlmod"
-load("@rules_python//python:pip.bzl", "pip_parse")
load("@rules_python//python:repositories.bzl", "python_register_toolchains")
-load("@rules_python//python/pip_install:pip_repository.bzl", "locked_requirements_label", "pip_repository_attrs", "use_isolated", "whl_library")
+load("@rules_python//python/pip_install:pip_repository.bzl", "locked_requirements_label", "pip_repository_attrs", "pip_repository_bzlmod", "use_isolated", "whl_library")
load("@rules_python//python/pip_install:repositories.bzl", "pip_install_dependencies")
load("@rules_python//python/pip_install:requirements_parser.bzl", parse_requirements = "parse")
load("@rules_python//python/private:coverage_deps.bzl", "install_coverage_deps")
@@ -68,7 +67,7 @@
# Parse the requirements file directly in starlark to get the information
# needed for the whl_libary declarations below. This is needed to contain
- # the pip_parse logic to a single module extension.
+ # the pip_repository logic to a single module extension.
requirements_lock_content = module_ctx.read(requrements_lock)
parse_result = parse_requirements(requirements_lock_content)
requirements = parse_result.requirements
@@ -76,14 +75,9 @@
# Create the repository where users load the `requirement` macro. Under bzlmod
# this does not create the install_deps() macro.
- pip_parse(
+ pip_repository_bzlmod(
name = attr.name,
requirements_lock = attr.requirements_lock,
- bzlmod = True,
- timeout = attr.timeout,
- python_interpreter = attr.python_interpreter,
- python_interpreter_target = attr.python_interpreter_target,
- quiet = attr.quiet,
)
for name, requirement_line in requirements:
@@ -114,7 +108,7 @@
"name": attr.string(mandatory = True),
}, **pip_repository_attrs)
- # Like the pip_parse macro, we end up setting this manually so
+ # Like the pip_repository rule, we end up setting this manually so
# don't allow users to override it.
attrs.pop("repo_prefix")
diff --git a/python/pip.bzl b/python/pip.bzl
index 3d45aed..3c06301 100644
--- a/python/pip.bzl
+++ b/python/pip.bzl
@@ -47,7 +47,7 @@
print("pip_install is deprecated. Please switch to pip_parse. pip_install will be removed in a future release.")
pip_parse(requirements = requirements, name = name, **kwargs)
-def pip_parse(requirements = None, requirements_lock = None, name = "pip_parsed_deps", bzlmod = False, **kwargs):
+def pip_parse(requirements = None, requirements_lock = None, name = "pip_parsed_deps", **kwargs):
"""Accepts a locked/compiled requirements file and installs the dependencies listed within.
Those dependencies become available in a generated `requirements.bzl` file.
@@ -143,14 +143,9 @@
requirements (Label): Deprecated. See requirements_lock.
name (str, optional): The name of the generated repository. The generated repositories
containing each requirement will be of the form `<name>_<requirement-name>`.
- bzlmod (bool, optional): Whether this rule is being run under a bzlmod module extension.
**kwargs (dict): Additional arguments to the [`pip_repository`](./pip_repository.md) repository rule.
"""
-
- # Don't try to fetch dependencies under bzlmod because they are already fetched via the internal_deps
- # module extention, and because the maybe-install pattern doesn't work under bzlmod.
- if not bzlmod:
- pip_install_dependencies()
+ pip_install_dependencies()
# Temporary compatibility shim.
# pip_install was previously document to use requirements while pip_parse was using requirements_lock.
@@ -160,8 +155,6 @@
pip_repository(
name = name,
requirements_lock = reqs_to_use,
- repo_prefix = "{}_".format(name),
- bzlmod = bzlmod,
**kwargs
)
diff --git a/python/pip_install/BUILD.bazel b/python/pip_install/BUILD.bazel
index 451e7fa..281ccba 100644
--- a/python/pip_install/BUILD.bazel
+++ b/python/pip_install/BUILD.bazel
@@ -4,7 +4,6 @@
"BUILD.bazel",
"//python/pip_install/tools/dependency_resolver:distribution",
"//python/pip_install/tools/lib:distribution",
- "//python/pip_install/tools/lock_file_generator:distribution",
"//python/pip_install/tools/wheel_installer:distribution",
"//python/pip_install/private:distribution",
],
@@ -24,7 +23,6 @@
srcs = [
"//python/pip_install/tools/dependency_resolver:py_srcs",
"//python/pip_install/tools/lib:py_srcs",
- "//python/pip_install/tools/lock_file_generator:py_srcs",
"//python/pip_install/tools/wheel_installer:py_srcs",
],
visibility = ["//python/pip_install/private:__pkg__"],
diff --git a/python/pip_install/pip_repository.bzl b/python/pip_install/pip_repository.bzl
index d5d93f3..982d853 100644
--- a/python/pip_install/pip_repository.bzl
+++ b/python/pip_install/pip_repository.bzl
@@ -257,110 +257,177 @@
""")
return requirements_txt
-# Keep in sync with `_clean_name` in generated requirements.bzl
+# Keep in sync with `_clean_pkg_name` in generated bzlmod requirements.bzl
def _clean_pkg_name(name):
return name.replace("-", "_").replace(".", "_").lower()
-def _bzlmod_pkg_aliases(rctx, requirements_txt):
+def _bzlmod_pkg_aliases(repo_name, bzl_packages):
"""Create alias declarations for each python dependency.
- The aliases should be appended to the pip_parse repo's BUILD.bazel file. These aliases
+ The aliases should be appended to the pip_repository BUILD.bazel file. These aliases
allow users to use requirement() without needed a corresponding `use_repo()` for each dep
when using bzlmod.
Args:
- rctx: the repository context
- requirements_txt: label to the requirements lock file
+ repo_name: the repository name of the parent that is visible to the users.
+ bzl_packages: the list of packages to setup.
"""
- requirements = parse_requirements(rctx.read(requirements_txt)).requirements
-
build_content = ""
- for requirement in requirements:
+ for name in bzl_packages:
build_content += """\
alias(
name = "{name}_pkg",
- actual = "@{repo_prefix}{dep}//:pkg",
+ actual = "@{repo_name}_{dep}//:pkg",
)
alias(
name = "{name}_whl",
- actual = "@{repo_prefix}{dep}//:whl",
+ actual = "@{repo_name}_{dep}//:whl",
)
alias(
name = "{name}_data",
- actual = "@{repo_prefix}{dep}//:data",
+ actual = "@{repo_name}_{dep}//:data",
)
alias(
name = "{name}_dist_info",
- actual = "@{repo_prefix}{dep}//:dist_info",
+ actual = "@{repo_name}_{dep}//:dist_info",
)
""".format(
- name = _clean_pkg_name(requirement[0]),
- repo_prefix = rctx.attr.repo_prefix,
- dep = _clean_pkg_name(requirement[0]),
+ name = name,
+ repo_name = repo_name,
+ dep = name,
)
return build_content
-def _pip_repository_impl(rctx):
- python_interpreter = _resolve_python_interpreter(rctx)
-
- # Write the annotations file to pass to the wheel maker
- annotations = {package: json.decode(data) for (package, data) in rctx.attr.annotations.items()}
- annotations_file = rctx.path("annotations.json")
- rctx.file(annotations_file, json.encode_indent(annotations, indent = " " * 4))
-
+def _pip_repository_bzlmod_impl(rctx):
requirements_txt = locked_requirements_label(rctx, rctx.attr)
- args = [
- python_interpreter,
- "-m",
- "python.pip_install.tools.lock_file_generator.lock_file_generator",
- "--requirements_lock",
- rctx.path(requirements_txt),
- "--requirements_lock_label",
- str(requirements_txt),
- # pass quiet and timeout args through to child repos.
- "--quiet",
- str(rctx.attr.quiet),
- "--timeout",
- str(rctx.attr.timeout),
- "--annotations",
- annotations_file,
- "--bzlmod",
- str(rctx.attr.bzlmod).lower(),
+ content = rctx.read(requirements_txt)
+ parsed_requirements_txt = parse_requirements(content)
+
+ packages = [(_clean_pkg_name(name), requirement) for name, requirement in parsed_requirements_txt.requirements]
+
+ bzl_packages = sorted([name for name, _ in packages])
+
+ repo_name = rctx.attr.name.split("~")[-1]
+
+ build_contents = _BUILD_FILE_CONTENTS + _bzlmod_pkg_aliases(repo_name, bzl_packages)
+
+ rctx.file("BUILD.bazel", build_contents)
+ rctx.template("requirements.bzl", rctx.attr._template, substitutions = {
+ "%%ALL_REQUIREMENTS%%": _format_repr_list([
+ "@{}//:{}_pkg".format(repo_name, p)
+ for p in bzl_packages
+ ]),
+ "%%ALL_WHL_REQUIREMENTS%%": _format_repr_list([
+ "@{}//:{}_whl".format(repo_name, p)
+ for p in bzl_packages
+ ]),
+ "%%NAME%%": rctx.attr.name,
+ "%%REQUIREMENTS_LOCK%%": str(requirements_txt),
+ })
+
+pip_repository_bzlmod_attrs = {
+ "requirements_darwin": attr.label(
+ allow_single_file = True,
+ doc = "Override the requirements_lock attribute when the host platform is Mac OS",
+ ),
+ "requirements_linux": attr.label(
+ allow_single_file = True,
+ doc = "Override the requirements_lock attribute when the host platform is Linux",
+ ),
+ "requirements_lock": attr.label(
+ allow_single_file = True,
+ doc = """
+A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead
+of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that
+wheels are fetched/built only for the targets specified by 'build/run/test'.
+""",
+ ),
+ "requirements_windows": attr.label(
+ allow_single_file = True,
+ doc = "Override the requirements_lock attribute when the host platform is Windows",
+ ),
+ "_template": attr.label(
+ default = ":pip_repository_requirements_bzlmod.bzl.tmpl",
+ ),
+}
+
+pip_repository_bzlmod = repository_rule(
+ attrs = pip_repository_bzlmod_attrs,
+ doc = """A rule for bzlmod pip_repository creation. Intended for private use only.""",
+ implementation = _pip_repository_bzlmod_impl,
+)
+
+def _pip_repository_impl(rctx):
+ requirements_txt = locked_requirements_label(rctx, rctx.attr)
+ content = rctx.read(requirements_txt)
+ parsed_requirements_txt = parse_requirements(content)
+
+ packages = [(_clean_pkg_name(name), requirement) for name, requirement in parsed_requirements_txt.requirements]
+
+ bzl_packages = sorted([name for name, _ in packages])
+
+ imports = [
+ 'load("@rules_python//python/pip_install:pip_repository.bzl", "whl_library")',
]
- args += ["--python_interpreter", _get_python_interpreter_attr(rctx)]
+ annotations = {}
+ for pkg, annotation in rctx.attr.annotations.items():
+ filename = "{}.annotation.json".format(_clean_pkg_name(pkg))
+ rctx.file(filename, json.encode_indent(json.decode(annotation)))
+ annotations[pkg] = "@{name}//:{filename}".format(name = rctx.attr.name, filename = filename)
+
+ tokenized_options = []
+ for opt in parsed_requirements_txt.options:
+ for p in opt.split(" "):
+ tokenized_options.append(p)
+
+ options = tokenized_options + rctx.attr.extra_pip_args
+
+ config = {
+ "download_only": rctx.attr.download_only,
+ "enable_implicit_namespace_pkgs": rctx.attr.enable_implicit_namespace_pkgs,
+ "environment": rctx.attr.environment,
+ "extra_pip_args": options,
+ "isolated": use_isolated(rctx, rctx.attr),
+ "pip_data_exclude": rctx.attr.pip_data_exclude,
+ "python_interpreter": _get_python_interpreter_attr(rctx),
+ "quiet": rctx.attr.quiet,
+ "repo": rctx.attr.name,
+ "repo_prefix": "{}_".format(rctx.attr.name),
+ "timeout": rctx.attr.timeout,
+ }
+
if rctx.attr.python_interpreter_target:
- args += ["--python_interpreter_target", str(rctx.attr.python_interpreter_target)]
- progress_message = "Parsing requirements to starlark"
+ config["python_interpreter_target"] = str(rctx.attr.python_interpreter_target)
- args += ["--repo", rctx.attr.name, "--repo-prefix", rctx.attr.repo_prefix]
- args = _parse_optional_attrs(rctx, args)
-
- rctx.report_progress(progress_message)
-
- result = rctx.execute(
- args,
- # Manually construct the PYTHONPATH since we cannot use the toolchain here
- environment = _create_repository_execution_environment(rctx),
- timeout = rctx.attr.timeout,
- quiet = rctx.attr.quiet,
- )
-
- if result.return_code:
- fail("rules_python failed: %s (%s)" % (result.stdout, result.stderr))
-
- # We need a BUILD file to load the generated requirements.bzl
- build_contents = _BUILD_FILE_CONTENTS
-
- if rctx.attr.bzlmod:
- build_contents += _bzlmod_pkg_aliases(rctx, requirements_txt)
-
- rctx.file("BUILD.bazel", build_contents + "\n# The requirements.bzl file was generated by running:\n# " + " ".join([str(a) for a in args]))
+ rctx.file("BUILD.bazel", _BUILD_FILE_CONTENTS)
+ rctx.template("requirements.bzl", rctx.attr._template, substitutions = {
+ "%%ALL_REQUIREMENTS%%": _format_repr_list([
+ "@{}_{}//:pkg".format(rctx.attr.name, p)
+ for p in bzl_packages
+ ]),
+ "%%ALL_WHL_REQUIREMENTS%%": _format_repr_list([
+ "@{}_{}//:whl".format(rctx.attr.name, p)
+ for p in bzl_packages
+ ]),
+ "%%ANNOTATIONS%%": _format_dict(_repr_dict(annotations)),
+ "%%CONFIG%%": _format_dict(_repr_dict(config)),
+ "%%EXTRA_PIP_ARGS%%": json.encode(options),
+ "%%IMPORTS%%": "\n".join(sorted(imports)),
+ "%%NAME%%": rctx.attr.name,
+ "%%PACKAGES%%": _format_repr_list(
+ [
+ ("{}_{}".format(rctx.attr.name, p), r)
+ for p, r in packages
+ ],
+ ),
+ "%%REQUIREMENTS_LOCK%%": str(requirements_txt),
+ })
return
@@ -453,12 +520,6 @@
"annotations": attr.string_dict(
doc = "Optional annotations to apply to packages",
),
- "bzlmod": attr.bool(
- default = False,
- doc = """Whether this repository rule is invoked under bzlmod, in which case
-we do not create the install_deps() macro.
-""",
- ),
"requirements_darwin": attr.label(
allow_single_file = True,
doc = "Override the requirements_lock attribute when the host platform is Mac OS",
@@ -479,6 +540,9 @@
allow_single_file = True,
doc = "Override the requirements_lock attribute when the host platform is Windows",
),
+ "_template": attr.label(
+ default = ":pip_repository_requirements.bzl.tmpl",
+ ),
}
pip_repository_attrs.update(**common_attrs)
@@ -625,3 +689,19 @@
data_exclude_glob = data_exclude_glob,
srcs_exclude_glob = srcs_exclude_glob,
))
+
+# pip_repository implementation
+
+def _format_list(items):
+ return "[{}]".format(", ".join(items))
+
+def _format_repr_list(strings):
+ return _format_list(
+ [repr(s) for s in strings],
+ )
+
+def _repr_dict(items):
+ return {k: repr(v) for k, v in items.items()}
+
+def _format_dict(items):
+ return "{{{}}}".format(", ".join(sorted(['"{}": {}'.format(k, v) for k, v in items.items()])))
diff --git a/python/pip_install/pip_repository_requirements.bzl.tmpl b/python/pip_install/pip_repository_requirements.bzl.tmpl
new file mode 100644
index 0000000..bf6a053
--- /dev/null
+++ b/python/pip_install/pip_repository_requirements.bzl.tmpl
@@ -0,0 +1,52 @@
+"""Starlark representation of locked requirements.
+
+@generated by rules_python pip_parse repository rule
+from %%REQUIREMENTS_LOCK%%
+"""
+
+%%IMPORTS%%
+
+all_requirements = %%ALL_REQUIREMENTS%%
+
+all_whl_requirements = %%ALL_WHL_REQUIREMENTS%%
+
+_packages = %%PACKAGES%%
+_config = %%CONFIG%%
+_annotations = %%ANNOTATIONS%%
+
+def _clean_name(name):
+ return name.replace("-", "_").replace(".", "_").lower()
+
+def requirement(name):
+ return "@%%NAME%%_" + _clean_name(name) + "//:pkg"
+
+def whl_requirement(name):
+ return "@%%NAME%%_" + _clean_name(name) + "//:whl"
+
+def data_requirement(name):
+ return "@%%NAME%%_" + _clean_name(name) + "//:data"
+
+def dist_info_requirement(name):
+ return "@%%NAME%%_" + _clean_name(name) + "//:dist_info"
+
+def entry_point(pkg, script = None):
+ if not script:
+ script = pkg
+ return "@%%NAME%%_" + _clean_name(pkg) + "//:rules_python_wheel_entry_point_" + script
+
+def _get_annotation(requirement):
+ # This expects to parse `setuptools==58.2.0 --hash=sha256:2551203ae6955b9876741a26ab3e767bb3242dafe86a32a749ea0d78b6792f11`
+ # down to `setuptools`.
+ name = requirement.split(" ")[0].split("=")[0].split("[")[0]
+ return _annotations.get(name)
+
+def install_deps(**whl_library_kwargs):
+ whl_config = dict(_config)
+ whl_config.update(whl_library_kwargs)
+ for name, requirement in _packages:
+ whl_library(
+ name = name,
+ requirement = requirement,
+ annotation = _get_annotation(requirement),
+ **whl_config
+ )
diff --git a/python/pip_install/pip_repository_requirements_bzlmod.bzl.tmpl b/python/pip_install/pip_repository_requirements_bzlmod.bzl.tmpl
new file mode 100644
index 0000000..462829d
--- /dev/null
+++ b/python/pip_install/pip_repository_requirements_bzlmod.bzl.tmpl
@@ -0,0 +1,24 @@
+"""Starlark representation of locked requirements.
+
+@generated by rules_python pip_parse repository rule
+from %%REQUIREMENTS_LOCK%%.
+"""
+
+all_requirements = %%ALL_REQUIREMENTS%%
+
+all_whl_requirements = %%ALL_WHL_REQUIREMENTS%%
+
+def _clean_name(name):
+ return name.replace("-", "_").replace(".", "_").lower()
+
+def requirement(name):
+ return "@@%%NAME%%//:" + _clean_name(name) + "_pkg"
+
+def whl_requirement(name):
+ return "@@%%NAME%%//:" + _clean_name(name) + "_whl"
+
+def data_requirement(name):
+ return "@@%%NAME%%//:" + _clean_name(name) + "_data"
+
+def dist_info_requirement(name):
+ return "@@%%NAME%%//:" + _clean_name(name) + "_dist_info"
diff --git a/python/pip_install/private/srcs.bzl b/python/pip_install/private/srcs.bzl
index 57644f6..f3064a3 100644
--- a/python/pip_install/private/srcs.bzl
+++ b/python/pip_install/private/srcs.bzl
@@ -13,8 +13,6 @@
"@rules_python//python/pip_install/tools/lib:annotation.py",
"@rules_python//python/pip_install/tools/lib:arguments.py",
"@rules_python//python/pip_install/tools/lib:bazel.py",
- "@rules_python//python/pip_install/tools/lock_file_generator:__init__.py",
- "@rules_python//python/pip_install/tools/lock_file_generator:lock_file_generator.py",
"@rules_python//python/pip_install/tools/wheel_installer:namespace_pkgs.py",
"@rules_python//python/pip_install/tools/wheel_installer:wheel.py",
"@rules_python//python/pip_install/tools/wheel_installer:wheel_installer.py",
diff --git a/python/pip_install/tools/lock_file_generator/BUILD.bazel b/python/pip_install/tools/lock_file_generator/BUILD.bazel
deleted file mode 100644
index 804f36a..0000000
--- a/python/pip_install/tools/lock_file_generator/BUILD.bazel
+++ /dev/null
@@ -1,50 +0,0 @@
-load("//python:defs.bzl", "py_binary", "py_library", "py_test")
-load("//python/pip_install:repositories.bzl", "requirement")
-
-py_library(
- name = "lib",
- srcs = [
- "lock_file_generator.py",
- ],
- deps = [
- "//python/pip_install/tools/lib",
- requirement("pip"),
- ],
-)
-
-py_binary(
- name = "lock_file_generator",
- srcs = [
- "lock_file_generator.py",
- ],
- deps = [":lib"],
-)
-
-py_test(
- name = "lock_file_generator_test",
- size = "small",
- srcs = [
- "lock_file_generator_test.py",
- ],
- deps = [
- ":lib",
- ],
-)
-
-filegroup(
- name = "distribution",
- srcs = glob(
- ["*"],
- exclude = ["*_test.py"],
- ),
- visibility = ["//python/pip_install:__subpackages__"],
-)
-
-filegroup(
- name = "py_srcs",
- srcs = glob(
- include = ["**/*.py"],
- exclude = ["**/*_test.py"],
- ),
- visibility = ["//python/pip_install:__subpackages__"],
-)
diff --git a/python/pip_install/tools/lock_file_generator/__init__.py b/python/pip_install/tools/lock_file_generator/__init__.py
deleted file mode 100644
index bbdfb4c..0000000
--- a/python/pip_install/tools/lock_file_generator/__init__.py
+++ /dev/null
@@ -1,14 +0,0 @@
-# Copyright 2023 The Bazel Authors. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
diff --git a/python/pip_install/tools/lock_file_generator/lock_file_generator.py b/python/pip_install/tools/lock_file_generator/lock_file_generator.py
deleted file mode 100644
index ed1488d..0000000
--- a/python/pip_install/tools/lock_file_generator/lock_file_generator.py
+++ /dev/null
@@ -1,336 +0,0 @@
-# Copyright 2023 The Bazel Authors. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import argparse
-import json
-import shlex
-import sys
-import textwrap
-from pathlib import Path
-from typing import Any, Dict, List, TextIO, Tuple
-
-from pip._internal.network.session import PipSession
-from pip._internal.req import constructors
-from pip._internal.req.req_file import (
- RequirementsFileParser,
- get_file_content,
- get_line_parser,
- preprocess,
-)
-from pip._internal.req.req_install import InstallRequirement
-
-from python.pip_install.tools.lib import annotation, arguments, bazel
-
-
-def parse_install_requirements(
- requirements_lock: str, extra_pip_args: List[str]
-) -> List[Tuple[InstallRequirement, str]]:
- ps = PipSession()
- # This is roughly taken from pip._internal.req.req_file.parse_requirements
- # (https://github.com/pypa/pip/blob/21.0.1/src/pip/_internal/req/req_file.py#L127) in order to keep
- # the original line (sort-of, its preprocessed) from the requirements_lock file around, to pass to sub repos
- # as the requirement.
- line_parser = get_line_parser(finder=None)
- parser = RequirementsFileParser(ps, line_parser)
- install_req_and_lines: List[Tuple[InstallRequirement, str]] = []
- _, content = get_file_content(requirements_lock, ps)
- unpinned_reqs = []
- for parsed_line, (_, line) in zip(
- parser.parse(requirements_lock, constraint=False), preprocess(content)
- ):
- if parsed_line.is_requirement:
- install_req = constructors.install_req_from_line(parsed_line.requirement)
- if (
- # PEP-440 direct references are considered pinned
- # See: https://peps.python.org/pep-0440/#direct-references and https://peps.python.org/pep-0508/
- not install_req.link
- and not install_req.is_pinned
- ):
- unpinned_reqs.append(str(install_req))
- install_req_and_lines.append((install_req, line))
-
- else:
- extra_pip_args.extend(shlex.split(line))
-
- if len(unpinned_reqs) > 0:
- unpinned_reqs_str = "\n".join(unpinned_reqs)
- raise RuntimeError(
- f"""\
-The `requirements_lock` file must be fully pinned. See `compile_pip_requirements`.
-Alternatively, use `pip-tools` or a similar mechanism to produce a pinned lockfile.
-
-The following requirements were not pinned:
-{unpinned_reqs_str}"""
- )
-
- return install_req_and_lines
-
-
-def repo_names_and_requirements(
- install_reqs: List[Tuple[InstallRequirement, str]], repo_prefix: str
-) -> List[Tuple[str, str]]:
- return [
- (
- bazel.sanitise_name(ir.name, prefix=repo_prefix),
- line,
- )
- for ir, line in install_reqs
- ]
-
-
-def parse_whl_library_args(args: argparse.Namespace) -> Dict[str, Any]:
- whl_library_args = dict(vars(args))
- whl_library_args = arguments.deserialize_structured_args(whl_library_args)
- whl_library_args.setdefault("python_interpreter", sys.executable)
-
- # These arguments are not used by `whl_library`
- for arg in (
- "requirements_lock",
- "requirements_lock_label",
- "annotations",
- "bzlmod",
- ):
- if arg in whl_library_args:
- whl_library_args.pop(arg)
-
- return whl_library_args
-
-
-def generate_parsed_requirements_contents(
- requirements_lock: Path,
- repo: str,
- repo_prefix: str,
- whl_library_args: Dict[str, Any],
- annotations: Dict[str, str] = dict(),
- bzlmod: bool = False,
-) -> str:
- """
- Parse each requirement from the requirements_lock file, and prepare arguments for each
- repository rule, which will represent the individual requirements.
-
- Generates a requirements.bzl file containing a macro (install_deps()) which instantiates
- a repository rule for each requirement in the lock file.
- """
- install_req_and_lines = parse_install_requirements(
- requirements_lock, whl_library_args["extra_pip_args"]
- )
- repo_names_and_reqs = repo_names_and_requirements(
- install_req_and_lines, repo_prefix
- )
-
- all_requirements = ", ".join(
- [
- bazel.sanitised_repo_library_label(ir.name, repo_prefix=repo_prefix)
- for ir, _ in install_req_and_lines
- ]
- )
- all_whl_requirements = ", ".join(
- [
- bazel.sanitised_repo_file_label(ir.name, repo_prefix=repo_prefix)
- for ir, _ in install_req_and_lines
- ]
- )
-
- install_deps_macro = """
- def install_deps(**whl_library_kwargs):
- whl_config = dict(_config)
- whl_config.update(whl_library_kwargs)
- for name, requirement in _packages:
- whl_library(
- name = name,
- requirement = requirement,
- annotation = _get_annotation(requirement),
- **whl_config
- )
-"""
- return textwrap.dedent(
- (
- """\
-
- load("@rules_python//python/pip_install:pip_repository.bzl", "whl_library")
-
- all_requirements = [{all_requirements}]
-
- all_whl_requirements = [{all_whl_requirements}]
-
- _packages = {repo_names_and_reqs}
- _config = {args}
- _annotations = {annotations}
- _bzlmod = {bzlmod}
-
- def _clean_name(name):
- return name.replace("-", "_").replace(".", "_").lower()
-
- def requirement(name):
- if _bzlmod:
- return "@@{repo}//:" + _clean_name(name) + "_{py_library_label}"
- return "@{repo_prefix}" + _clean_name(name) + "//:{py_library_label}"
-
- def whl_requirement(name):
- if _bzlmod:
- return "@@{repo}//:" + _clean_name(name) + "_{wheel_file_label}"
- return "@{repo_prefix}" + _clean_name(name) + "//:{wheel_file_label}"
-
- def data_requirement(name):
- if _bzlmod:
- return "@@{repo}//:" + _clean_name(name) + "_{data_label}"
- return "@{repo_prefix}" + _clean_name(name) + "//:{data_label}"
-
- def dist_info_requirement(name):
- if _bzlmod:
- return "@@{repo}//:" + _clean_name(name) + "_{dist_info_label}"
- return "@{repo_prefix}" + _clean_name(name) + "//:{dist_info_label}"
-
- def entry_point(pkg, script = None):
- if not script:
- script = pkg
- return "@{repo_prefix}" + _clean_name(pkg) + "//:{entry_point_prefix}_" + script
-
- def _get_annotation(requirement):
- # This expects to parse `setuptools==58.2.0 --hash=sha256:2551203ae6955b9876741a26ab3e767bb3242dafe86a32a749ea0d78b6792f11`
- # down wo `setuptools`.
- name = requirement.split(" ")[0].split("=")[0].split("[")[0]
- return _annotations.get(name)
-"""
- + (install_deps_macro if not bzlmod else "")
- ).format(
- all_requirements=all_requirements,
- all_whl_requirements=all_whl_requirements,
- annotations=json.dumps(annotations),
- args=dict(sorted(whl_library_args.items())),
- data_label=bazel.DATA_LABEL,
- dist_info_label=bazel.DIST_INFO_LABEL,
- entry_point_prefix=bazel.WHEEL_ENTRY_POINT_PREFIX,
- py_library_label=bazel.PY_LIBRARY_LABEL,
- repo_names_and_reqs=repo_names_and_reqs,
- repo=repo,
- repo_prefix=repo_prefix,
- wheel_file_label=bazel.WHEEL_FILE_LABEL,
- bzlmod=bzlmod,
- )
- )
-
-
-def coerce_to_bool(option):
- return str(option).lower() == "true"
-
-
-def main(output: TextIO) -> None:
- """Args:
-
- output: where to write the resulting starlark, such as sys.stdout or an open file
- """
- parser = argparse.ArgumentParser(
- description="Create rules to incrementally fetch needed \
-dependencies from a fully resolved requirements lock file."
- )
- parser.add_argument(
- "--requirements_lock",
- action="store",
- required=True,
- help="Path to fully resolved requirements.txt to use as the source of repos.",
- )
- parser.add_argument(
- "--requirements_lock_label",
- help="Label used to declare the requirements.lock, included in comments in the file.",
- )
- parser.add_argument(
- "--python_interpreter",
- help="The python interpreter that will be used to download and unpack the wheels.",
- )
- parser.add_argument(
- "--python_interpreter_target",
- help="Bazel target of a python interpreter.\
-It will be used in repository rules so it must be an already built interpreter.\
-If set, it will take precedence over python_interpreter.",
- )
- parser.add_argument(
- "--quiet",
- type=coerce_to_bool,
- default=True,
- required=True,
- help="Whether to print stdout / stderr from child repos.",
- )
- parser.add_argument(
- "--timeout",
- type=int,
- action="store",
- required=True,
- help="timeout to use for pip operation.",
- )
- parser.add_argument(
- "--annotations",
- type=annotation.annotations_map_from_str_path,
- help="A json encoded file containing annotations for rendered packages.",
- )
- parser.add_argument(
- "--bzlmod",
- type=coerce_to_bool,
- default=False,
- help="Whether this script is run under bzlmod. Under bzlmod we don't generate the install_deps() macro as it isn't needed.",
- )
- arguments.parse_common_args(parser)
- args = parser.parse_args()
-
- whl_library_args = parse_whl_library_args(args)
-
- # Check for any annotations which match packages in the locked requirements file
- install_requirements = parse_install_requirements(
- args.requirements_lock, whl_library_args["extra_pip_args"]
- )
- req_names = sorted([req.name for req, _ in install_requirements])
- annotations = args.annotations.collect(req_names) if args.annotations else {}
-
- # Write all rendered annotation files and generate a list of the labels to write to the requirements file
- annotated_requirements = dict()
- for name, content in annotations.items():
- annotation_path = Path(name + ".annotation.json")
- annotation_path.write_text(json.dumps(content, indent=4))
- annotated_requirements.update(
- {
- name: "@{}//:{}.annotation.json".format(
- args.repo, name
- )
- }
- )
-
- output.write(
- textwrap.dedent(
- """\
- \"\"\"Starlark representation of locked requirements.
-
- @generated by rules_python pip_parse repository rule
- from {}
- \"\"\"
- """.format(
- args.requirements_lock_label
- )
- )
- )
-
- output.write(
- generate_parsed_requirements_contents(
- requirements_lock=args.requirements_lock,
- repo=args.repo,
- repo_prefix=args.repo_prefix,
- whl_library_args=whl_library_args,
- annotations=annotated_requirements,
- bzlmod=args.bzlmod,
- )
- )
-
-
-if __name__ == "__main__":
- with open("requirements.bzl", "w") as requirement_file:
- main(requirement_file)
diff --git a/python/pip_install/tools/lock_file_generator/lock_file_generator_test.py b/python/pip_install/tools/lock_file_generator/lock_file_generator_test.py
deleted file mode 100644
index be244b1..0000000
--- a/python/pip_install/tools/lock_file_generator/lock_file_generator_test.py
+++ /dev/null
@@ -1,163 +0,0 @@
-# Copyright 2023 The Bazel Authors. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import argparse
-import json
-import tempfile
-import unittest
-from pathlib import Path
-from textwrap import dedent
-
-from pip._internal.req.req_install import InstallRequirement
-
-from python.pip_install.tools.lock_file_generator import lock_file_generator
-
-
-class TestParseRequirementsToBzl(unittest.TestCase):
- maxDiff = None
-
- def test_generated_requirements_bzl(self) -> None:
- with tempfile.TemporaryDirectory() as temp_dir:
- requirements_lock = Path(temp_dir) / "requirements.txt"
- comments_and_flags = "#comment\n--require-hashes True\n"
- requirement_string = "foo==0.0.0 --hash=sha256:hashofFoowhl"
- requirements_lock.write_bytes(
- bytes(comments_and_flags + requirement_string, encoding="utf-8")
- )
- args = argparse.Namespace()
- args.requirements_lock = str(requirements_lock.resolve())
- args.repo = ("pip_parsed_deps_pypi__",)
- args.repo_prefix = "pip_parsed_deps_pypi__"
- extra_pip_args = ["--index-url=pypi.org/simple"]
- pip_data_exclude = ["**.foo"]
- args.extra_pip_args = json.dumps({"arg": extra_pip_args})
- args.pip_data_exclude = json.dumps({"arg": pip_data_exclude})
- args.python_interpreter = "/custom/python3"
- args.python_interpreter_target = "@custom_python//:exec"
- args.environment = json.dumps({"arg": {}})
- whl_library_args = lock_file_generator.parse_whl_library_args(args)
- contents = lock_file_generator.generate_parsed_requirements_contents(
- requirements_lock=args.requirements_lock,
- repo=args.repo,
- repo_prefix=args.repo_prefix,
- whl_library_args=whl_library_args,
- )
- library_target = "@pip_parsed_deps_pypi__foo//:pkg"
- whl_target = "@pip_parsed_deps_pypi__foo//:whl"
- all_requirements = 'all_requirements = ["{library_target}"]'.format(
- library_target=library_target
- )
- all_whl_requirements = 'all_whl_requirements = ["{whl_target}"]'.format(
- whl_target=whl_target
- )
- self.assertIn(all_requirements, contents, contents)
- self.assertIn(all_whl_requirements, contents, contents)
- self.assertIn(requirement_string, contents, contents)
- all_flags = extra_pip_args + ["--require-hashes", "True"]
- self.assertIn(
- "'extra_pip_args': {}".format(repr(all_flags)), contents, contents
- )
- self.assertIn(
- "'pip_data_exclude': {}".format(repr(pip_data_exclude)),
- contents,
- contents,
- )
- self.assertIn("'python_interpreter': '/custom/python3'", contents, contents)
- self.assertIn(
- "'python_interpreter_target': '@custom_python//:exec'",
- contents,
- contents,
- )
- # Assert it gets set to an empty dict by default.
- self.assertIn("'environment': {}", contents, contents)
-
- def test_parse_install_requirements_with_args(self):
- # Test requirements files with varying arguments
- for requirement_args in ("", "--index-url https://index.python.com"):
- with tempfile.TemporaryDirectory() as temp_dir:
- requirements_lock = Path(temp_dir) / "requirements.txt"
- requirements_lock.write_text(
- dedent(
- """\
- {}
-
- wheel==0.37.1 \\
- --hash=sha256:4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a \\
- --hash=sha256:e9a504e793efbca1b8e0e9cb979a249cf4a0a7b5b8c9e8b65a5e39d49529c1c4
- # via -r requirements.in
- setuptools==58.2.0 \\
- --hash=sha256:2551203ae6955b9876741a26ab3e767bb3242dafe86a32a749ea0d78b6792f11 \
- --hash=sha256:2c55bdb85d5bb460bd2e3b12052b677879cffcf46c0c688f2e5bf51d36001145
- # via -r requirements.in
- """.format(
- requirement_args
- )
- )
- )
-
- install_req_and_lines = lock_file_generator.parse_install_requirements(
- str(requirements_lock), ["-v"]
- )
-
- # There should only be two entries for the two requirements
- self.assertEqual(len(install_req_and_lines), 2)
-
- # The first index in each tuple is expected to be an `InstallRequirement` object
- self.assertIsInstance(install_req_and_lines[0][0], InstallRequirement)
- self.assertIsInstance(install_req_and_lines[1][0], InstallRequirement)
-
- # Ensure the requirements text is correctly parsed with the trailing arguments
- self.assertTupleEqual(
- install_req_and_lines[0][1:],
- (
- "wheel==0.37.1 --hash=sha256:4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a --hash=sha256:e9a504e793efbca1b8e0e9cb979a249cf4a0a7b5b8c9e8b65a5e39d49529c1c4",
- ),
- )
- self.assertTupleEqual(
- install_req_and_lines[1][1:],
- (
- "setuptools==58.2.0 --hash=sha256:2551203ae6955b9876741a26ab3e767bb3242dafe86a32a749ea0d78b6792f11 --hash=sha256:2c55bdb85d5bb460bd2e3b12052b677879cffcf46c0c688f2e5bf51d36001145",
- ),
- )
-
- def test_parse_install_requirements_pinned_direct_reference(self):
- # Test PEP-440 direct references
- with tempfile.TemporaryDirectory() as temp_dir:
- requirements_lock = Path(temp_dir) / "requirements.txt"
- requirements_lock.write_text(
- dedent(
- """\
- onnx @ https://files.pythonhosted.org/packages/24/93/f5b001dc0f5de84ce049a34ff382032cd9478e1080aa6ac48470fa810577/onnx-1.11.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl \
- --hash=sha256:67c6d2654c1c203e5c839a47900b51f588fd0de71bbd497fb193d30a0b3ec1e9
- """
- )
- )
-
- install_req_and_lines = lock_file_generator.parse_install_requirements(
- str(requirements_lock), ["-v"]
- )
-
- self.assertEqual(len(install_req_and_lines), 1)
- self.assertEqual(install_req_and_lines[0][0].name, "onnx")
-
- self.assertTupleEqual(
- install_req_and_lines[0][1:],
- (
- "onnx @ https://files.pythonhosted.org/packages/24/93/f5b001dc0f5de84ce049a34ff382032cd9478e1080aa6ac48470fa810577/onnx-1.11.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl --hash=sha256:67c6d2654c1c203e5c839a47900b51f588fd0de71bbd497fb193d30a0b3ec1e9",
- ),
- )
-
-
-if __name__ == "__main__":
- unittest.main()