docs: generate Starlark domain markup instead of regular markdown (#1919)
This switches our doc generation over to using the Starlark domain
markup from the sphinx_stardoc plugin instead of using regular markdown.
This allows the docs generated from code to better integrate with each
other and other parts of the doc site.
Overview of changes:
* Makes the doc paths under the API directory more directly mirror their
actual location. e.g. moves "defs.md" -> "python/defs.md". This is so
the //tools doc entries have a more natural location, but can also be
used for our other top-level directories.
* Adds API docs for some of the well known targets we have. These aren't
automatically generated, but use the Starlark domain markup, so
integrate nicely with everything.
* Ensures default values are parsable as Python expressions. Stardoc
returns values like "<function foo>" or 'Label(*, "//bar")' in some
cases for the default value of args/attrs.
* Ensures function signatures don't crash doc rendering. Stardoc gives
bad/incomplete information, so reconstructing the original signature of
a function is tricky.
* Allows references flags using leading slashes and a value, e.g.
`--foo=bar`. This makes it more natural to write while cross referencing
to the flag.
* Implements `{any}` xref resolution. It was just totally broken before.
* Adds some additional bzl files that get documented.
* Adds some more Bazel external references.
* Fixes some missing bzl_library dependencies.
* A few minor QoL improvements to the docs dev server:
* Print the serving directory when CTRL+C is received. This makes it
easier to find the raw files that are being generated.
* Fix an error during shutdown about an unterminated generator.
* The `sphinx_stardocs.footer` arg is removed. This was always just a
hack to get extra link targets into the generated bzl docs. It's no
longer needed when the bzl domain is used.
* Using `@repo//pkg:file.bzl%Name` syntax is supported in type
expressions (e.g. `:type:` option or `{type}` role) by quoting the
label. The quoting is necessary because, under the hood, the expressions
are parsed as Python.
* Objects directives support an `:origin-key` directive. This records
the label identity that Bazel sees for an object (as from the Stardoc
origin_key field). The markdown generate doesn't generate this for
everything yet because some things are documented twice (e.g. py_binary
in defs.bzl and py_binary.bzl), which would cause a crash (type things
trying to define the same id).
* Add `*` and `**` to var-args and var-kwargs in signatures.
* Allow providers to be refered to using the `type` role. This allows
providers to be referenced in `:type:` directives (e.g. in a provider
field).diff --git a/docs/sphinx/BUILD.bazel b/docs/sphinx/BUILD.bazel
index e3b9ad3..c2a1690 100644
--- a/docs/sphinx/BUILD.bazel
+++ b/docs/sphinx/BUILD.bazel
@@ -30,7 +30,7 @@
"@platforms//os:linux": [],
"@platforms//os:macos": [],
"//conditions:default": ["@platforms//:incompatible"],
-})
+}) if IS_BAZEL_7_OR_HIGHER else ["@platforms//:incompatible"]
# See README.md for instructions. Short version:
# * `bazel run //docs/sphinx:docs.serve` in a separate terminal
@@ -76,24 +76,34 @@
sphinx_stardocs(
name = "bzl_api_docs",
docs = {
- "api/cc/py_cc_toolchain.md": dict(
+ "api/python/cc/py_cc_toolchain.md": dict(
dep = "//python/private:py_cc_toolchain_bzl",
input = "//python/private:py_cc_toolchain_rule.bzl",
public_load_path = "//python/cc:py_cc_toolchain.bzl",
),
- "api/cc/py_cc_toolchain_info.md": "//python/cc:py_cc_toolchain_info_bzl",
- "api/defs.md": "//python:defs_bzl",
- "api/entry_points/py_console_script_binary.md": "//python/entry_points:py_console_script_binary_bzl",
- "api/packaging.md": "//python:packaging_bzl",
- "api/pip.md": "//python:pip_bzl",
+ "api/python/cc/py_cc_toolchain_info.md": "//python/cc:py_cc_toolchain_info_bzl",
+ "api/python/defs.md": "//python:defs_bzl",
+ "api/python/entry_points/py_console_script_binary.md": "//python/entry_points:py_console_script_binary_bzl",
+ "api/python/packaging.md": "//python:packaging_bzl",
+ "api/python/pip.md": "//python:pip_bzl",
+ "api/python/py_binary.md": "//python:py_binary_bzl",
+ "api/python/py_cc_link_params_info.md": "//python:py_cc_link_params_info_bzl",
+ "api/python/py_library.md": "//python:py_library_bzl",
+ "api/python/py_runtime.md": "//python:py_runtime_bzl",
+ "api/python/py_runtime_info.md": "//python:py_runtime_info_bzl",
+ "api/python/py_runtime_pair.md": dict(
+ dep = "//python/private:py_runtime_pair_rule_bzl",
+ input = "//python/private:py_runtime_pair_rule.bzl",
+ public_load_path = "//python:py_runtime_pair.bzl",
+ ),
+ "api/python/py_test.md": "//python:py_test_bzl",
} | ({
# Bazel 6 + Stardoc isn't able to parse something about the python bzlmod extension
- "api/extensions/python.md": "//python/extensions:python_bzl",
+ "api/python/extensions/python.md": "//python/extensions:python_bzl",
} if IS_BAZEL_7_OR_HIGHER else {}) | ({
# This depends on @pythons_hub, which is only created under bzlmod,
- "api/extensions/pip.md": "//python/extensions:pip_bzl",
+ "api/python/extensions/pip.md": "//python/extensions:pip_bzl",
} if IS_BAZEL_7_OR_HIGHER and BZLMOD_ENABLED else {}),
- footer = "_stardoc_footer.md",
tags = ["docs"],
target_compatible_with = _TARGET_COMPATIBLE_WITH,
)
@@ -112,6 +122,8 @@
requirement("sphinx_rtd_theme"),
requirement("myst_parser"),
requirement("readthedocs_sphinx_ext"),
+ requirement("typing_extensions"),
+ "//sphinxdocs/src/sphinx_stardoc",
],
)
diff --git a/docs/sphinx/api/python/config_settings/index.md b/docs/sphinx/api/python/config_settings/index.md
new file mode 100644
index 0000000..82a5b2a
--- /dev/null
+++ b/docs/sphinx/api/python/config_settings/index.md
@@ -0,0 +1,68 @@
+:::{bzl:currentfile} //python/config_settings:BUILD.bazel
+:::
+
+# //python/config_settings
+
+:::{bzl:flag} precompile
+Determines if Python source files should be compiled at build time.
+
+NOTE: The flag value is overridden by the target level `precompile` attribute,
+except for the case of `force_enabled` and `forced_disabled`.
+
+Values:
+
+* `auto`: Automatically decide the effective value based on environment,
+ target platform, etc.
+* `enabled`: Compile Python source files at build time. Note that
+ {bzl:obj}`--precompile_add_to_runfiles` affects how the compiled files are included into
+ a downstream binary.
+* `disabled`: Don't compile Python source files at build time.
+* `if_generated_source`: Compile Python source files, but only if they're a
+ generated file.
+* `force_enabled`: Like `enabled`, except overrides target-level setting. This
+ is mostly useful for development, testing enabling precompilation more
+ broadly, or as an escape hatch if build-time compiling is not available.
+* `force_disabled`: Like `disabled`, except overrides target-level setting. This
+ is useful useful for development, testing enabling precompilation more
+ broadly, or as an escape hatch if build-time compiling is not available.
+:::
+
+:::{bzl:flag} precompile_source_retention
+Determines, when a source file is compiled, if the source file is kept
+in the resulting output or not.
+
+NOTE: This flag is overridden by the target level `precompile_source_retention`
+attribute.
+
+Values:
+
+* `keep_source`: Include the original Python source.
+* `omit_source`: Don't include the orignal py source.
+* `omit_if_generated_source`: Keep the original source if it's a regular source
+ file, but omit it if it's a generated file.
+:::
+
+:::{bzl:flag} precompile_add_to_runfiles
+Determines if a target adds its compiled files to its runfiles.
+
+When a target compiles its files, but doesn't add them to its own runfiles, it
+relies on a downstream target to retrieve them from
+{bzl:obj}`PyInfo.transitive_pyc_files`
+
+Values:
+* `always`: Always include the compiled files in the target's runfiles.
+* `decided_elsewhere`: Don't include the compiled files in the target's
+ runfiles; they are still added to {bzl:obj}`PyInfo.transitive_pyc_files`. See
+ also: {bzl:obj}`py_binary.pyc_collection` attribute. This is useful for allowing
+ incrementally enabling precompilation on a per-binary basis.
+:::
+
+:::{bzl:flag} pyc_collection
+Determine if `py_binary` collects transitive pyc files.
+
+NOTE: This flag is overridden by the target level `pyc_collection` attribute.
+
+Values:
+* `include_pyc`: Include `PyInfo.transitive_pyc_files` as part of the binary.
+* `disabled`: Don't include `PyInfo.transitive_pyc_files` as part of the binary.
+:::
diff --git a/docs/sphinx/api/python/index.md b/docs/sphinx/api/python/index.md
new file mode 100644
index 0000000..8026a7f
--- /dev/null
+++ b/docs/sphinx/api/python/index.md
@@ -0,0 +1,23 @@
+:::{bzl:currentfile} //python:BUILD.bazel
+:::
+
+# //python
+
+:::{bzl:target} toolchain_type
+
+Identifier for the toolchain type for the target platform.
+:::
+
+:::{bzl:target} exec_tools_toolchain_type
+
+Identifier for the toolchain type for exec tools used to build Python targets.
+:::
+
+:::{bzl:target} current_py_toolchain
+
+Helper target to resolve to the consumer's current Python toolchain. This target
+provides:
+
+* `PyRuntimeInfo`: The consuming target's target toolchain information
+
+:::
diff --git a/docs/sphinx/api/tools/precompiler/index.md b/docs/sphinx/api/tools/precompiler/index.md
new file mode 100644
index 0000000..1a47651
--- /dev/null
+++ b/docs/sphinx/api/tools/precompiler/index.md
@@ -0,0 +1,15 @@
+:::{bzl:currentfile} //tools/precompiler:BUILD.bazel
+:::
+
+# //tools/precompiler
+
+:::{bzl:flag} execution_requirements
+Determines the execution requirements `//tools/precompiler:precompiler` uses.
+
+This is a repeatable string_list flag. The values are `key=value` entries, each
+of which are added to the execution requirements for the `PyCompile` action to
+generate pyc files.
+
+Customizing this flag mostly allows controlling whether Bazel runs the
+precompiler as a regular worker, persistent worker, or regular action.
+:::
diff --git a/docs/sphinx/bazel_inventory.txt b/docs/sphinx/bazel_inventory.txt
index b099f42..62cbdf8 100644
--- a/docs/sphinx/bazel_inventory.txt
+++ b/docs/sphinx/bazel_inventory.txt
@@ -22,3 +22,5 @@
str bzl:type 1 rules/lib/string -
struct bzl:type 1 rules/lib/builtins/struct -
target-name bzl:doc 1 concepts/labels#target-names -
+CcInfo bzl:provider 1 rules/lib/providers/CcInfo -
+CcInfo.linking_context bzl:provider-field 1 rules/lib/providers/CcInfo#linking_context -
diff --git a/docs/sphinx/conf.py b/docs/sphinx/conf.py
index e9af97a..fef083c 100644
--- a/docs/sphinx/conf.py
+++ b/docs/sphinx/conf.py
@@ -27,6 +27,7 @@
"sphinx.ext.intersphinx",
"myst_parser",
"sphinx_rtd_theme", # Necessary to get jquery to make flyout work
+ "sphinx_stardoc.stardoc",
]
# Adapted from the template code:
@@ -89,6 +90,10 @@
myst_substitutions = {}
+# --- sphinx_stardoc configuration
+
+bzl_default_repository_name = "@rules_python"
+
# -- Options for HTML output
# See https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
# For additional html settings
diff --git a/docs/sphinx/precompiling.md b/docs/sphinx/precompiling.md
index e30fc94..52678e6 100644
--- a/docs/sphinx/precompiling.md
+++ b/docs/sphinx/precompiling.md
@@ -29,14 +29,15 @@
To use this approach, the two basic steps are:
1. Disable pyc files from being automatically added to runfiles:
- `--@rules_python//python/config_settings:precompile_add_to_runfiles=decided_elsewhere`,
+ {bzl:obj}`--@rules_python//python/config_settings:precompile_add_to_runfiles=decided_elsewhere`,
2. Set the `pyc_collection` attribute on the binaries/tests that should or should
not use precompiling.
-The default for the `pyc_collection` attribute is controlled by a flag, so you
-can use an opt-in or opt-out approach by setting the flag:
-* targets must opt-out: `--@rules_python//python/config_settings:pyc_collection=include_pyc`,
-* targets must opt-in: `--@rules_python//python/config_settings:pyc_collection=disabled`,
+The default for the `pyc_collection` attribute is controlled by the flag
+{bzl:obj}`--@rules_python//python/config_settings:pyc_collection`, so you
+can use an opt-in or opt-out approach by setting its value:
+* targets must opt-out: `--@rules_python//python/config_settings:pyc_collection=include_pyc`
+* targets must opt-in: `--@rules_python//python/config_settings:pyc_collection=disabled`
## Advanced precompiler customization
