fix(pypi): allow pip_compile to work with read-only sources (#2712)
The validating `py_test` generated by `compile_pip_requirements` chokes
when the source `requirements.txt` is stored read-only, such as when
managed by the Perforce Helix Core SCM. Though `dependency_resolver`
makes a temporary copy of this file, it does so w/ `shutil.copy` which
preserves the original read-only file mode. To address this, this commit
replaces `shutil.copy` with a `shutil.copyfileobj` such that the
temporary file is created w/ permissions according to the user's umask.
Resolves (#2608).
---------
Co-authored-by: Ignas Anikevicius <240938+aignas@users.noreply.github.com>
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 355f1fe..0a2dc41 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -93,6 +93,8 @@
also retrieved from the URL as opposed to only the `--hash` parameter. Fixes
[#2363](https://github.com/bazel-contrib/rules_python/issues/2363).
* (pypi) `whl_library` now infers file names from its `urls` attribute correctly.
+* (pypi) When running under `bazel test`, be sure that temporary `requirements` file
+ remains writable.
* (py_test, py_binary) Allow external files to be used for main
{#v0-0-0-added}
diff --git a/python/private/pypi/dependency_resolver/dependency_resolver.py b/python/private/pypi/dependency_resolver/dependency_resolver.py
index ada0763..a42821c 100644
--- a/python/private/pypi/dependency_resolver/dependency_resolver.py
+++ b/python/private/pypi/dependency_resolver/dependency_resolver.py
@@ -151,9 +151,16 @@
requirements_out = os.path.join(
os.environ["TEST_TMPDIR"], os.path.basename(requirements_file) + ".out"
)
+ # Why this uses shutil.copyfileobj:
+ #
# Those two files won't necessarily be on the same filesystem, so we can't use os.replace
# or shutil.copyfile, as they will fail with OSError: [Errno 18] Invalid cross-device link.
- shutil.copy(resolved_requirements_file, requirements_out)
+ #
+ # Further, shutil.copy preserves the source file's mode, and so if
+ # our source file is read-only (the default under Perforce Helix),
+ # this scratch file will also be read-only, defeating its purpose.
+ with open(resolved_requirements_file, "rb") as fsrc, open(requirements_out, "wb") as fdst:
+ shutil.copyfileobj(fsrc, fdst)
update_command = (
os.getenv("CUSTOM_COMPILE_COMMAND") or f"bazel run {target_label_prefix}.update"