feat: add persistent worker for sphinxdocs (#2938)

This implements a simple, serialized persistent worker for Sphinxdocs
with several optimizations. It is enabled by default.

* The worker computes what inputs have changed, allowing Sphinx to only
rebuild what
  is necessary.
* Doctrees are written to a separate directory so they are retained
between builds.
* The worker tells Sphinx to write output to an internal directory, then
copies it
to the expected Bazel output directory afterwards. This allows Sphinx to
only
  write output files that need to be updated.

This works by having the worker compute what files have changed and
having a Sphinx
extension use the `get-env-outdated` event to tell Sphinx which files
have changed.
The extension is based on https://pwrev.dev/294057, but re-implemented
to be
in-memory as part of the worker instead of a separate extension projects
must configure.

For rules_python's doc building, this reduces incremental building from
about 8 seconds
to about 0.8 seconds. From what I can tell, about half the time is spent
generating
doctrees, and the other half generating the output files.

Worker mode is enabled by default and can be disabled on the target or
by adjusting
the Bazel flags controlling execution strategy. Docs added to explain
how.

Because `--doctree-dir` is now always specified and outside the output
dir,
non-worker invocations can benefit, too, if run without sandboxing. Docs
added to
explain how to do this.

Along the way:

* Remove `--write-all` and `--fresh-env` from run args. This lets direct
  invocations benefit from the normal caching Sphinx does.
* Change the args formatting to `--foo=bar` so they are a single
element; just
  a bit nicer to see when debugging.


Work towards https://github.com/bazel-contrib/rules_python/issues/2878,
https://github.com/bazel-contrib/rules_python/issues/2879

---------

Co-authored-by: Kayce Basques <kayce@google.com>
Co-authored-by: Richard Levasseur <rlevasseur@google.com>
5 files changed
tree: 5e2ece8e3a3a6ed68f556472a890b4804a8b622a
  1. .bazelci/
  2. .bcr/
  3. .ci/
  4. .github/
  5. docs/
  6. examples/
  7. gazelle/
  8. private/
  9. python/
  10. sphinxdocs/
  11. tests/
  12. tools/
  13. .bazelignore
  14. .bazelrc
  15. .bazelversion
  16. .editorconfig
  17. .git-blame-ignore-revs
  18. .gitattributes
  19. .gitignore
  20. .pre-commit-config.yaml
  21. .python-version
  22. .readthedocs.yml
  23. addlicense.sh
  24. AUTHORS
  25. BUILD.bazel
  26. BZLMOD_SUPPORT.md
  27. CHANGELOG.md
  28. CONTRIBUTING.md
  29. CONTRIBUTORS
  30. internal_dev_deps.bzl
  31. internal_dev_setup.bzl
  32. LICENSE
  33. MODULE.bazel
  34. README.md
  35. RELEASING.md
  36. version.bzl
  37. WORKSPACE
  38. WORKSPACE.bzlmod
README.md

Python Rules for Bazel

Build status

Overview

This repository is the home of the core Python rules -- py_library, py_binary, py_test, py_proto_library, and related symbols that provide the basis for Python support in Bazel. It also contains package installation rules for integrating with PyPI and other indices.

Documentation for rules_python is at https://rules-python.readthedocs.io and in the Bazel Build Encyclopedia.

Examples live in the examples directory.

The core rules are stable. Their implementation is subject to Bazel's backward compatibility policy. This repository aims to follow semantic versioning.

The Bazel community maintains this repository. Neither Google nor the Bazel team provides support for the code. However, this repository is part of the test suite used to vet new Bazel releases. See How to contribute page for information on our development workflow.

Documentation

For detailed documentation, see https://rules-python.readthedocs.io

Bzlmod support

  • Status: Beta
  • Full Feature Parity: No

See Bzlmod support for more details.