feat(pypi): support env markers in requirements files (#2059)
Before this change the `all_requirements` and related constants will
have
packages that need to be installed only on specific platforms and will
mean
that users relying on those constants (e.g. `gazelle`) will need to do
extra
work to exclude platform-specific packages. The package managers that
that
support outputting such files now include `uv` and `pdm`. This might be
also
useful in cases where we attempt to handle non-requirements lock files.
Note, that the best way to handle this would be to move all of the
requirements
parsing code to Python, but that may cause regressions as it is a much
bigger
change. This is only changing the code so that we are doing extra
processing
for the requirement lines that have env markers. The lines that have no
markers
will not see any change in the code execution paths and the python
interpreter
will not be downloaded.
We also use the `*_ctx.watch` API where available to correctly
re-evaluate the
markers if the `packaging` Python sources for this change.
Extra changes that are included in this PR:
- Extend the `repo_utils` to have a method for `arch` getting from the
`ctx`.
- Change the `local_runtime_repo` to perform the validation not relying
on the
implementation detail of the `get_platforms_os_name`.
- Add `$(UV)` make variable for the `uv:current_toolchain` so that we
can
generate the requirements for `sphinx` using `uv`.
- Swap the requirement generation using `genrule` and `uv` for `sphinx`
and co
so that we can test the `requirement` marker code. Note, the
`requirement`
markers are not working well with the `requirement_cycles`.
Fixes #1105.
Fixes #1868.
Work towards #260, #1975.
Related #1663.
---------
Co-authored-by: Richard Levasseur <rlevasseur@google.com>
diff --git a/python/private/pypi/extension.bzl b/python/private/pypi/extension.bzl
index 82e580d..1bc8f15 100644
--- a/python/private/pypi/extension.bzl
+++ b/python/private/pypi/extension.bzl
@@ -21,6 +21,7 @@
load("//python/private:repo_utils.bzl", "repo_utils")
load("//python/private:version_label.bzl", "version_label")
load(":attrs.bzl", "use_isolated")
+load(":evaluate_markers.bzl", "evaluate_markers", EVALUATE_MARKERS_SRCS = "SRCS")
load(":hub_repository.bzl", "hub_repository")
load(":parse_requirements.bzl", "host_platform", "parse_requirements", "select_requirement")
load(":parse_whl_name.bzl", "parse_whl_name")
@@ -195,6 +196,28 @@
logger = logger,
),
get_index_urls = get_index_urls,
+ # NOTE @aignas 2024-08-02: , we will execute any interpreter that we find either
+ # in the PATH or if specified as a label. We will configure the env
+ # markers when evaluating the requirement lines based on the output
+ # from the `requirements_files_by_platform` which should have something
+ # similar to:
+ # {
+ # "//:requirements.txt": ["cp311_linux_x86_64", ...]
+ # }
+ #
+ # We know the target python versions that we need to evaluate the
+ # markers for and thus we don't need to use multiple python interpreter
+ # instances to perform this manipulation. This function should be executed
+ # only once by the underlying code to minimize the overhead needed to
+ # spin up a Python interpreter.
+ evaluate_markers = lambda module_ctx, requirements: evaluate_markers(
+ module_ctx,
+ requirements = requirements,
+ python_interpreter = pip_attr.python_interpreter,
+ python_interpreter_target = python_interpreter_target,
+ srcs = pip_attr._evaluate_markers_srcs,
+ logger = logger,
+ ),
logger = logger,
)
@@ -625,6 +648,13 @@
The labels are JSON config files describing the modifications.
""",
),
+ "_evaluate_markers_srcs": attr.label_list(
+ default = EVALUATE_MARKERS_SRCS,
+ doc = """\
+The list of labels to use as SRCS for the marker evaluation code. This ensures that the
+code will be re-evaluated when any of files in the default changes.
+""",
+ ),
}, **ATTRS)
attrs.update(AUTH_ATTRS)
