| commit | afae3f08fe44ab87c6f445b94c647845b22220d0 | [log] [tgz] |
|---|---|---|
| author | Zhongpeng Lin <zplin@uber.com> | Wed Apr 17 13:11:24 2024 -0700 |
| committer | GitHub <noreply@github.com> | Wed Apr 17 20:11:24 2024 +0000 |
| tree | 903332f3334c089fbee25d70dbf6aa92a93484e6 | |
| parent | 4be00a69ae3a0c4e7ef3a5f952254bacd16d0a2a [diff] |
fix: using thread pool on macOS (#1861)
The python parser uses ProcessPoolExecutor, which is problematic on
macOS when it is distributed as a zip file, leading to errors like:
```
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/var/folders/fw/vythc6112ygfsvky8mdb5p580000gn/T/Bazel.runfiles_esxfeg_v/runfiles/python3_aarch64-apple-darwin/lib/python3.9/multiprocessing/resource_tracker.py", line 24, in <module>
from . import spawn
File "/var/folders/fw/vythc6112ygfsvky8mdb5p580000gn/T/Bazel.runfiles_esxfeg_v/runfiles/python3_aarch64-apple-darwin/lib/python3.9/multiprocessing/spawn.py", line 13, in <module>
import runpy
File "/var/folders/fw/vythc6112ygfsvky8mdb5p580000gn/T/Bazel.runfiles_esxfeg_v/runfiles/python3_aarch64-apple-darwin/lib/python3.9/runpy.py", line 19, in <module>
from pkgutil import read_code, get_importer
ModuleNotFoundError: No module named 'pkgutil'
```
According to ["Contexts and start methods"
section](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods)
of the documentation:
> On macOS, the spawn start method is now the default. The fork start
method should be considered unsafe as it can lead to crashes of the
subprocess as macOS system libraries may start threads.
meanwhile:
> The 'spawn' and 'forkserver' start methods generally cannot be used
with “frozen” executables (i.e., binaries produced by packages like
PyInstaller and cx_Freeze) on POSIX systems.
This means there is no way to start a ProcessPoolExecutor when the
Python zip file is running on macOS. This PR switches it to
ThreadPoolExecutor instead.
This repository is the home of the core Python rules -- py_library, py_binary, py_test, py_proto_library, and related symbols that provide the basis for Python support in Bazel. It also contains package installation rules for integrating with PyPI and other indices.
Documentation for rules_python is at https://rules-python.readthedocs.io and in the Bazel Build Encyclopedia.
Examples live in the examples directory.
Currently, the core rules build into the Bazel binary, and the symbols in this repository are simple aliases. However, we are migrating the rules to Starlark and removing them from the Bazel binary. Therefore, the future-proof way to depend on Python rules is via this repository. SeeMigrating from the Bundled Rules below.
The core rules are stable. Their implementation in Bazel is subject to Bazel's backward compatibility policy. Once migrated to rules_python, they may evolve at a different rate, but this repository will still follow semantic versioning.
The Bazel community maintains this repository. Neither Google nor the Bazel team provides support for the code. However, this repository is part of the test suite used to vet new Bazel releases. See How to contribute page for information on our development workflow.
For detailed documentation, see https://rules-python.readthedocs.io
See Bzlmod support for more details.