docs: generate Starlark domain markup instead of regular markdown (#1919)
This switches our doc generation over to using the Starlark domain
markup from the sphinx_stardoc plugin instead of using regular markdown.
This allows the docs generated from code to better integrate with each
other and other parts of the doc site.
Overview of changes:
* Makes the doc paths under the API directory more directly mirror their
actual location. e.g. moves "defs.md" -> "python/defs.md". This is so
the //tools doc entries have a more natural location, but can also be
used for our other top-level directories.
* Adds API docs for some of the well known targets we have. These aren't
automatically generated, but use the Starlark domain markup, so
integrate nicely with everything.
* Ensures default values are parsable as Python expressions. Stardoc
returns values like "<function foo>" or 'Label(*, "//bar")' in some
cases for the default value of args/attrs.
* Ensures function signatures don't crash doc rendering. Stardoc gives
bad/incomplete information, so reconstructing the original signature of
a function is tricky.
* Allows references flags using leading slashes and a value, e.g.
`--foo=bar`. This makes it more natural to write while cross referencing
to the flag.
* Implements `{any}` xref resolution. It was just totally broken before.
* Adds some additional bzl files that get documented.
* Adds some more Bazel external references.
* Fixes some missing bzl_library dependencies.
* A few minor QoL improvements to the docs dev server:
* Print the serving directory when CTRL+C is received. This makes it
easier to find the raw files that are being generated.
* Fix an error during shutdown about an unterminated generator.
* The `sphinx_stardocs.footer` arg is removed. This was always just a
hack to get extra link targets into the generated bzl docs. It's no
longer needed when the bzl domain is used.
* Using `@repo//pkg:file.bzl%Name` syntax is supported in type
expressions (e.g. `:type:` option or `{type}` role) by quoting the
label. The quoting is necessary because, under the hood, the expressions
are parsed as Python.
* Objects directives support an `:origin-key` directive. This records
the label identity that Bazel sees for an object (as from the Stardoc
origin_key field). The markdown generate doesn't generate this for
everything yet because some things are documented twice (e.g. py_binary
in defs.bzl and py_binary.bzl), which would cause a crash (type things
trying to define the same id).
* Add `*` and `**` to var-args and var-kwargs in signatures.
* Allow providers to be refered to using the `type` role. This allows
providers to be referenced in `:type:` directives (e.g. in a provider
field).diff --git a/docs/sphinx/BUILD.bazel b/docs/sphinx/BUILD.bazel
index e3b9ad3..c2a1690 100644
--- a/docs/sphinx/BUILD.bazel
+++ b/docs/sphinx/BUILD.bazel
@@ -30,7 +30,7 @@
"@platforms//os:linux": [],
"@platforms//os:macos": [],
"//conditions:default": ["@platforms//:incompatible"],
-})
+}) if IS_BAZEL_7_OR_HIGHER else ["@platforms//:incompatible"]
# See README.md for instructions. Short version:
# * `bazel run //docs/sphinx:docs.serve` in a separate terminal
@@ -76,24 +76,34 @@
sphinx_stardocs(
name = "bzl_api_docs",
docs = {
- "api/cc/py_cc_toolchain.md": dict(
+ "api/python/cc/py_cc_toolchain.md": dict(
dep = "//python/private:py_cc_toolchain_bzl",
input = "//python/private:py_cc_toolchain_rule.bzl",
public_load_path = "//python/cc:py_cc_toolchain.bzl",
),
- "api/cc/py_cc_toolchain_info.md": "//python/cc:py_cc_toolchain_info_bzl",
- "api/defs.md": "//python:defs_bzl",
- "api/entry_points/py_console_script_binary.md": "//python/entry_points:py_console_script_binary_bzl",
- "api/packaging.md": "//python:packaging_bzl",
- "api/pip.md": "//python:pip_bzl",
+ "api/python/cc/py_cc_toolchain_info.md": "//python/cc:py_cc_toolchain_info_bzl",
+ "api/python/defs.md": "//python:defs_bzl",
+ "api/python/entry_points/py_console_script_binary.md": "//python/entry_points:py_console_script_binary_bzl",
+ "api/python/packaging.md": "//python:packaging_bzl",
+ "api/python/pip.md": "//python:pip_bzl",
+ "api/python/py_binary.md": "//python:py_binary_bzl",
+ "api/python/py_cc_link_params_info.md": "//python:py_cc_link_params_info_bzl",
+ "api/python/py_library.md": "//python:py_library_bzl",
+ "api/python/py_runtime.md": "//python:py_runtime_bzl",
+ "api/python/py_runtime_info.md": "//python:py_runtime_info_bzl",
+ "api/python/py_runtime_pair.md": dict(
+ dep = "//python/private:py_runtime_pair_rule_bzl",
+ input = "//python/private:py_runtime_pair_rule.bzl",
+ public_load_path = "//python:py_runtime_pair.bzl",
+ ),
+ "api/python/py_test.md": "//python:py_test_bzl",
} | ({
# Bazel 6 + Stardoc isn't able to parse something about the python bzlmod extension
- "api/extensions/python.md": "//python/extensions:python_bzl",
+ "api/python/extensions/python.md": "//python/extensions:python_bzl",
} if IS_BAZEL_7_OR_HIGHER else {}) | ({
# This depends on @pythons_hub, which is only created under bzlmod,
- "api/extensions/pip.md": "//python/extensions:pip_bzl",
+ "api/python/extensions/pip.md": "//python/extensions:pip_bzl",
} if IS_BAZEL_7_OR_HIGHER and BZLMOD_ENABLED else {}),
- footer = "_stardoc_footer.md",
tags = ["docs"],
target_compatible_with = _TARGET_COMPATIBLE_WITH,
)
@@ -112,6 +122,8 @@
requirement("sphinx_rtd_theme"),
requirement("myst_parser"),
requirement("readthedocs_sphinx_ext"),
+ requirement("typing_extensions"),
+ "//sphinxdocs/src/sphinx_stardoc",
],
)
diff --git a/docs/sphinx/api/python/config_settings/index.md b/docs/sphinx/api/python/config_settings/index.md
new file mode 100644
index 0000000..82a5b2a
--- /dev/null
+++ b/docs/sphinx/api/python/config_settings/index.md
@@ -0,0 +1,68 @@
+:::{bzl:currentfile} //python/config_settings:BUILD.bazel
+:::
+
+# //python/config_settings
+
+:::{bzl:flag} precompile
+Determines if Python source files should be compiled at build time.
+
+NOTE: The flag value is overridden by the target level `precompile` attribute,
+except for the case of `force_enabled` and `forced_disabled`.
+
+Values:
+
+* `auto`: Automatically decide the effective value based on environment,
+ target platform, etc.
+* `enabled`: Compile Python source files at build time. Note that
+ {bzl:obj}`--precompile_add_to_runfiles` affects how the compiled files are included into
+ a downstream binary.
+* `disabled`: Don't compile Python source files at build time.
+* `if_generated_source`: Compile Python source files, but only if they're a
+ generated file.
+* `force_enabled`: Like `enabled`, except overrides target-level setting. This
+ is mostly useful for development, testing enabling precompilation more
+ broadly, or as an escape hatch if build-time compiling is not available.
+* `force_disabled`: Like `disabled`, except overrides target-level setting. This
+ is useful useful for development, testing enabling precompilation more
+ broadly, or as an escape hatch if build-time compiling is not available.
+:::
+
+:::{bzl:flag} precompile_source_retention
+Determines, when a source file is compiled, if the source file is kept
+in the resulting output or not.
+
+NOTE: This flag is overridden by the target level `precompile_source_retention`
+attribute.
+
+Values:
+
+* `keep_source`: Include the original Python source.
+* `omit_source`: Don't include the orignal py source.
+* `omit_if_generated_source`: Keep the original source if it's a regular source
+ file, but omit it if it's a generated file.
+:::
+
+:::{bzl:flag} precompile_add_to_runfiles
+Determines if a target adds its compiled files to its runfiles.
+
+When a target compiles its files, but doesn't add them to its own runfiles, it
+relies on a downstream target to retrieve them from
+{bzl:obj}`PyInfo.transitive_pyc_files`
+
+Values:
+* `always`: Always include the compiled files in the target's runfiles.
+* `decided_elsewhere`: Don't include the compiled files in the target's
+ runfiles; they are still added to {bzl:obj}`PyInfo.transitive_pyc_files`. See
+ also: {bzl:obj}`py_binary.pyc_collection` attribute. This is useful for allowing
+ incrementally enabling precompilation on a per-binary basis.
+:::
+
+:::{bzl:flag} pyc_collection
+Determine if `py_binary` collects transitive pyc files.
+
+NOTE: This flag is overridden by the target level `pyc_collection` attribute.
+
+Values:
+* `include_pyc`: Include `PyInfo.transitive_pyc_files` as part of the binary.
+* `disabled`: Don't include `PyInfo.transitive_pyc_files` as part of the binary.
+:::
diff --git a/docs/sphinx/api/python/index.md b/docs/sphinx/api/python/index.md
new file mode 100644
index 0000000..8026a7f
--- /dev/null
+++ b/docs/sphinx/api/python/index.md
@@ -0,0 +1,23 @@
+:::{bzl:currentfile} //python:BUILD.bazel
+:::
+
+# //python
+
+:::{bzl:target} toolchain_type
+
+Identifier for the toolchain type for the target platform.
+:::
+
+:::{bzl:target} exec_tools_toolchain_type
+
+Identifier for the toolchain type for exec tools used to build Python targets.
+:::
+
+:::{bzl:target} current_py_toolchain
+
+Helper target to resolve to the consumer's current Python toolchain. This target
+provides:
+
+* `PyRuntimeInfo`: The consuming target's target toolchain information
+
+:::
diff --git a/docs/sphinx/api/tools/precompiler/index.md b/docs/sphinx/api/tools/precompiler/index.md
new file mode 100644
index 0000000..1a47651
--- /dev/null
+++ b/docs/sphinx/api/tools/precompiler/index.md
@@ -0,0 +1,15 @@
+:::{bzl:currentfile} //tools/precompiler:BUILD.bazel
+:::
+
+# //tools/precompiler
+
+:::{bzl:flag} execution_requirements
+Determines the execution requirements `//tools/precompiler:precompiler` uses.
+
+This is a repeatable string_list flag. The values are `key=value` entries, each
+of which are added to the execution requirements for the `PyCompile` action to
+generate pyc files.
+
+Customizing this flag mostly allows controlling whether Bazel runs the
+precompiler as a regular worker, persistent worker, or regular action.
+:::
diff --git a/docs/sphinx/bazel_inventory.txt b/docs/sphinx/bazel_inventory.txt
index b099f42..62cbdf8 100644
--- a/docs/sphinx/bazel_inventory.txt
+++ b/docs/sphinx/bazel_inventory.txt
@@ -22,3 +22,5 @@
str bzl:type 1 rules/lib/string -
struct bzl:type 1 rules/lib/builtins/struct -
target-name bzl:doc 1 concepts/labels#target-names -
+CcInfo bzl:provider 1 rules/lib/providers/CcInfo -
+CcInfo.linking_context bzl:provider-field 1 rules/lib/providers/CcInfo#linking_context -
diff --git a/docs/sphinx/conf.py b/docs/sphinx/conf.py
index e9af97a..fef083c 100644
--- a/docs/sphinx/conf.py
+++ b/docs/sphinx/conf.py
@@ -27,6 +27,7 @@
"sphinx.ext.intersphinx",
"myst_parser",
"sphinx_rtd_theme", # Necessary to get jquery to make flyout work
+ "sphinx_stardoc.stardoc",
]
# Adapted from the template code:
@@ -89,6 +90,10 @@
myst_substitutions = {}
+# --- sphinx_stardoc configuration
+
+bzl_default_repository_name = "@rules_python"
+
# -- Options for HTML output
# See https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
# For additional html settings
diff --git a/docs/sphinx/precompiling.md b/docs/sphinx/precompiling.md
index e30fc94..52678e6 100644
--- a/docs/sphinx/precompiling.md
+++ b/docs/sphinx/precompiling.md
@@ -29,14 +29,15 @@
To use this approach, the two basic steps are:
1. Disable pyc files from being automatically added to runfiles:
- `--@rules_python//python/config_settings:precompile_add_to_runfiles=decided_elsewhere`,
+ {bzl:obj}`--@rules_python//python/config_settings:precompile_add_to_runfiles=decided_elsewhere`,
2. Set the `pyc_collection` attribute on the binaries/tests that should or should
not use precompiling.
-The default for the `pyc_collection` attribute is controlled by a flag, so you
-can use an opt-in or opt-out approach by setting the flag:
-* targets must opt-out: `--@rules_python//python/config_settings:pyc_collection=include_pyc`,
-* targets must opt-in: `--@rules_python//python/config_settings:pyc_collection=disabled`,
+The default for the `pyc_collection` attribute is controlled by the flag
+{bzl:obj}`--@rules_python//python/config_settings:pyc_collection`, so you
+can use an opt-in or opt-out approach by setting its value:
+* targets must opt-out: `--@rules_python//python/config_settings:pyc_collection=include_pyc`
+* targets must opt-in: `--@rules_python//python/config_settings:pyc_collection=disabled`
## Advanced precompiler customization
diff --git a/python/BUILD.bazel b/python/BUILD.bazel
index 3ab390d..5d31df5 100644
--- a/python/BUILD.bazel
+++ b/python/BUILD.bazel
@@ -130,6 +130,10 @@
bzl_library(
name = "py_cc_link_params_info_bzl",
srcs = ["py_cc_link_params_info.bzl"],
+ deps = [
+ "//python/private/common:providers_bzl",
+ "@rules_python_internal//:rules_python_config_bzl",
+ ],
)
bzl_library(
@@ -185,6 +189,7 @@
"//python/private:reexports_bzl",
"//python/private:util_bzl",
"//python/private/common:providers_bzl",
+ "@rules_python_internal//:rules_python_config_bzl",
],
)
diff --git a/python/private/common/providers.bzl b/python/private/common/providers.bzl
index ab56fbe..5b84549 100644
--- a/python/private/common/providers.bzl
+++ b/python/private/common/providers.bzl
@@ -144,63 +144,86 @@
""",
init = _PyRuntimeInfo_init,
fields = {
- "bootstrap_template": (
- "See py_runtime_rule.bzl%py_runtime.bootstrap_template for docs."
- ),
- "coverage_files": (
- "The files required at runtime for using `coverage_tool`. " +
- "Will be `None` if no `coverage_tool` was provided."
- ),
- "coverage_tool": (
- "If set, this field is a `File` representing tool used for collecting code coverage information from python tests. Otherwise, this is `None`."
- ),
- "files": (
- "If this is an in-build runtime, this field is a `depset` of `File`s" +
- "that need to be added to the runfiles of an executable target that " +
- "uses this runtime (in particular, files needed by `interpreter`). " +
- "The value of `interpreter` need not be included in this field. If " +
- "this is a platform runtime then this field is `None`."
- ),
- "implementation_name": "Optional string; the Python implementation name (`sys.implementation.name`)",
- "interpreter": (
- "If this is an in-build runtime, this field is a `File` representing " +
- "the interpreter. Otherwise, this is `None`. Note that an in-build " +
- "runtime can use either a prebuilt, checked-in interpreter or an " +
- "interpreter built from source."
- ),
- "interpreter_path": (
- "If this is a platform runtime, this field is the absolute " +
- "filesystem path to the interpreter on the target platform. " +
- "Otherwise, this is `None`."
- ),
- "interpreter_version_info": (
- "Version information about the interpreter this runtime provides. " +
- "It should match the format given by `sys.version_info`, however " +
- "for simplicity, the micro, releaselevel, and serial values are " +
- "optional." +
- "A struct with the following fields:\n" +
- " * major: int, the major version number\n" +
- " * minor: int, the minor version number\n" +
- " * micro: optional int, the micro version number\n" +
- " * releaselevel: optional str, the release level\n" +
- " * serial: optional int, the serial number of the release"
- ),
- "pyc_tag": """
-Optional string; the tag portion of a pyc filename, e.g. the `cpython-39` infix
-of `foo.cpython-39.pyc`. See PEP 3147. If not specified, it will be computed
-from `implementation_name` and `interpreter_version_info`. If no pyc_tag is
-available, then only source-less pyc generation will function correctly.
+ "bootstrap_template": """
+:type: File
+
+See py_runtime_rule.bzl%py_runtime.bootstrap_template for docs.
""",
- "python_version": (
- "Indicates whether this runtime uses Python major version 2 or 3. " +
- "Valid values are (only) `\"PY2\"` and " +
- "`\"PY3\"`."
- ),
- "stub_shebang": (
- "\"Shebang\" expression prepended to the bootstrapping Python stub " +
- "script used when executing `py_binary` targets. Does not " +
- "apply to Windows."
- ),
+ "coverage_files": """
+:type: depset[File] | None
+
+The files required at runtime for using `coverage_tool`. Will be `None` if no
+`coverage_tool` was provided.
+""",
+ "coverage_tool": """
+:type: File | None
+
+If set, this field is a `File` representing tool used for collecting code
+coverage information from python tests. Otherwise, this is `None`.
+""",
+ "files": """
+:type: depset[File] | None
+
+If this is an in-build runtime, this field is a `depset` of `File`s that need to
+be added to the runfiles of an executable target that uses this runtime (in
+particular, files needed by `interpreter`). The value of `interpreter` need not
+be included in this field. If this is a platform runtime then this field is
+`None`.
+""",
+ "implementation_name": """
+:type: str | None
+
+The Python implementation name (`sys.implementation.name`)
+""",
+ "interpreter": """
+:type: File | None
+
+If this is an in-build runtime, this field is a `File` representing the
+interpreter. Otherwise, this is `None`. Note that an in-build runtime can use
+either a prebuilt, checked-in interpreter or an interpreter built from source.
+""",
+ "interpreter_path": """
+:type: str | None
+
+If this is a platform runtime, this field is the absolute filesystem path to the
+interpreter on the target platform. Otherwise, this is `None`.
+""",
+ "interpreter_version_info": """
+:type: struct
+
+Version information about the interpreter this runtime provides.
+It should match the format given by `sys.version_info`, however
+for simplicity, the micro, releaselevel, and serial values are
+optional.
+A struct with the following fields:
+* `major`: {type}`int`, the major version number
+* `minor`: {type}`int`, the minor version number
+* `micro`: {type}`int | None`, the micro version number
+* `releaselevel`: {type}`str | None`, the release level
+* `serial`: {type}`int | None`, the serial number of the release
+""",
+ "pyc_tag": """
+:type: str | None
+
+The tag portion of a pyc filename, e.g. the `cpython-39` infix
+of `foo.cpython-39.pyc`. See PEP 3147. If not specified, it will be computed
+from {obj}`implementation_name` and {obj}`interpreter_version_info`. If no
+pyc_tag is available, then only source-less pyc generation will function
+correctly.
+""",
+ "python_version": """
+:type: str
+
+Indicates whether this runtime uses Python major version 2 or 3. Valid values
+are (only) `"PY2"` and `"PY3"`.
+""",
+ "stub_shebang": """
+:type: str
+
+"Shebang" expression prepended to the bootstrapping Python stub
+script used when executing {obj}`py_binary` targets. Does not
+apply to Windows.
+""",
},
)
@@ -248,26 +271,43 @@
init = _PyInfo_init,
fields = {
"direct_pyc_files": """
-depset[File] of precompiled Python files that are considered directly provided
+:type: depset[File]
+
+Precompiled Python files that are considered directly provided
by the target.
""",
- "has_py2_only_sources": "Whether any of this target's transitive sources requires a Python 2 runtime.",
- "has_py3_only_sources": "Whether any of this target's transitive sources requires a Python 3 runtime.",
+ "has_py2_only_sources": """
+:type: bool
+
+Whether any of this target's transitive sources requires a Python 2 runtime.
+""",
+ "has_py3_only_sources": """
+:type: bool
+
+Whether any of this target's transitive sources requires a Python 3 runtime.
+""",
"imports": """\
+:type: depset[str]
+
A depset of import path strings to be added to the `PYTHONPATH` of executable
Python targets. These are accumulated from the transitive `deps`.
The order of the depset is not guaranteed and may be changed in the future. It
is recommended to use `default` order (the default).
""",
"transitive_pyc_files": """
-depset[File] of direct and transitive precompiled Python files that are provied
-by the target.
+:type: depset[File]
+
+Direct and transitive precompiled Python files that are provided by the target.
""",
"transitive_sources": """\
+:type: depset[File]
+
A (`postorder`-compatible) depset of `.py` files appearing in the target's
`srcs` and the `srcs` of the target's transitive `deps`.
""",
"uses_shared_libraries": """
+:type: bool
+
Whether any of this target's transitive `deps` has a shared library file (such
as a `.so` file).
@@ -283,11 +323,15 @@
# buildifier: disable=name-conventions
PyCcLinkParamsProvider, _unused_raw_py_cc_link_params_provider_ctor = _define_provider(
- doc = ("Python-wrapper to forward CcInfo.linking_context. This is to " +
+ doc = ("Python-wrapper to forward {obj}`CcInfo.linking_context`. This is to " +
"allow Python targets to propagate C++ linking information, but " +
"without the Python target appearing to be a valid C++ rule dependency"),
init = _PyCcLinkParamsProvider_init,
fields = {
- "cc_info": "A CcInfo instance; it has only linking_context set",
+ "cc_info": """
+:type: CcInfo
+
+Linking information; it has only {obj}`CcInfo.linking_context` set.
+""",
},
)
diff --git a/python/private/py_cc_toolchain_info.bzl b/python/private/py_cc_toolchain_info.bzl
index a47a6a5..ae46bf4 100644
--- a/python/private/py_cc_toolchain_info.bzl
+++ b/python/private/py_cc_toolchain_info.bzl
@@ -18,7 +18,9 @@
doc = "C/C++ information about the Python runtime.",
fields = {
"headers": """\
-(struct) Information about the header files, with fields:
+:type: struct
+
+Information about the header files, struct with fields:
* providers_map: a dict of string to provider instances. The key should be
a fully qualified name (e.g. `@rules_foo//bar:baz.bzl#MyInfo`) of the
provider to uniquely identify its type.
@@ -39,7 +41,9 @@
represents).
""",
"libs": """\
-(struct) Information about C libraries, with fields:
+:type: struct
+
+Information about C libraries, struct with fields:
* providers_map: A dict of string to provider instances. The key should be
a fully qualified name (e.g. `@rules_foo//bar:baz.bzl#MyInfo`) of the
provider to uniquely identify its type.
@@ -59,6 +63,10 @@
e.g. `:current_py_cc_headers` to act as the underlying headers target it
represents).
""",
- "python_version": "(str) The Python Major.Minor version.",
+ "python_version": """
+:type: str
+
+The Python Major.Minor version.
+""",
},
)
diff --git a/python/private/py_runtime_pair_rule.bzl b/python/private/py_runtime_pair_rule.bzl
index d17b008..02f9a5b 100644
--- a/python/private/py_runtime_pair_rule.bzl
+++ b/python/private/py_runtime_pair_rule.bzl
@@ -98,15 +98,12 @@
Usually the wrapped runtimes are declared using the `py_runtime` rule, but any
rule returning a `PyRuntimeInfo` provider may be used.
-This rule returns a `platform_common.ToolchainInfo` provider with the following
-schema:
+This rule returns a {obj}`ToolchainInfo` provider with fields:
-```python
-platform_common.ToolchainInfo(
- py2_runtime = <PyRuntimeInfo or None>,
- py3_runtime = <PyRuntimeInfo or None>,
-)
-```
+* `py2_runtime`: {type}`PyRuntimeInfo | None`, runtime information for a
+ Python 2 runtime.
+* `py3_runtime`: {type}`PyRuntimeInfo | None`. runtime information for a
+ Python 3 runtime.
Example usage:
diff --git a/sphinxdocs/private/proto_to_markdown.py b/sphinxdocs/private/proto_to_markdown.py
index 18d4e1e..d667eec 100644
--- a/sphinxdocs/private/proto_to_markdown.py
+++ b/sphinxdocs/private/proto_to_markdown.py
@@ -80,6 +80,12 @@
yield i == 0, i == len(values) - 1, value
+def _sort_attributes_inplace(attributes):
+ # Sort attributes so the iteration order results in a Python-syntax
+ # valid signature. Keep name first because that's convention.
+ attributes.sort(key=lambda a: (a.name != "name", bool(a.default_value), a.name))
+
+
class _MySTRenderer:
def __init__(
self,
@@ -99,6 +105,9 @@
bzl_path = self._public_load_path
else:
bzl_path = "//" + self._module.file.split("//")[1]
+
+ self._write(":::{default-domain} bzl\n:::\n")
+ self._write(":::{bzl:currentfile} ", bzl_path, "\n:::\n\n")
self._write(
f"# {bzl_path}\n",
"\n",
@@ -129,320 +138,344 @@
self._write("\n")
def _render_aspect(self, aspect: stardoc_output_pb2.AspectInfo):
- aspect_anchor = _anchor_id(aspect.aspect_name)
- self._write(
- _block_attrs(".starlark-object"),
- f"## {aspect.aspect_name}\n\n",
- "_Propagates on attributes:_ ", # todo add link here
- ", ".join(sorted(f"`{attr}`" for attr in aspect.aspect_attribute)),
- "\n\n",
- aspect.doc_string.strip(),
- "\n\n",
- )
+ _sort_attributes_inplace(aspect.attribute)
+ self._write("::::::{bzl:aspect} ", aspect.aspect_name, "\n\n")
+ edges = ", ".join(sorted(f"`{attr}`" for attr in aspect.aspect_attribute))
+ self._write(":aspect-attributes: ", edges, "\n\n")
+ self._write(aspect.doc_string.strip(), "\n\n")
if aspect.attribute:
- self._render_attributes(aspect_anchor, aspect.attribute)
- self._write("\n")
+ self._render_attributes(aspect.attribute)
+ self._write("\n")
+ self._write("::::::\n")
def _render_module_extension(self, mod_ext: stardoc_output_pb2.ModuleExtensionInfo):
- self._write(
- _block_attrs(".starlark-object"),
- f"## {mod_ext.extension_name}\n\n",
- )
-
+ self._write("::::::{bzl:module-extension} ", mod_ext.extension_name, "\n\n")
self._write(mod_ext.doc_string.strip(), "\n\n")
- mod_ext_anchor = _anchor_id(mod_ext.extension_name)
for tag in mod_ext.tag_class:
tag_name = f"{mod_ext.extension_name}.{tag.tag_name}"
- tag_anchor = f"{mod_ext_anchor}_{tag.tag_name}"
- self._write(
- _block_attrs(".starlark-module-extension-tag-class"),
- f"### {tag_name}\n\n",
- )
+ tag_name = f"{tag.tag_name}"
+ self._write(":::::{bzl:tag-class} ", tag_name, "\n\n")
+
+ _sort_attributes_inplace(tag.attribute)
self._render_signature(
tag_name,
- tag_anchor,
tag.attribute,
get_name=lambda a: a.name,
get_default=lambda a: a.default_value,
)
self._write(tag.doc_string.strip(), "\n\n")
- self._render_attributes(tag_anchor, tag.attribute)
- self._write("\n")
+ self._render_attributes(tag.attribute)
+ self._write(":::::\n")
+ self._write("::::::\n")
def _render_repository_rule(self, repo_rule: stardoc_output_pb2.RepositoryRuleInfo):
- self._write(
- _block_attrs(".starlark-object"),
- f"## {repo_rule.rule_name}\n\n",
- )
- repo_anchor = _anchor_id(repo_rule.rule_name)
+ self._write("::::::{bzl:repo-rule} ")
+ _sort_attributes_inplace(repo_rule.attribute)
self._render_signature(
repo_rule.rule_name,
- repo_anchor,
repo_rule.attribute,
get_name=lambda a: a.name,
get_default=lambda a: a.default_value,
)
self._write(repo_rule.doc_string.strip(), "\n\n")
if repo_rule.attribute:
- self._render_attributes(repo_anchor, repo_rule.attribute)
+ self._render_attributes(repo_rule.attribute)
if repo_rule.environ:
- self._write(
- "**ENVIRONMENT VARIABLES** ",
- _link_here_icon(repo_anchor + "_env"),
- "\n",
- )
- for name in sorted(repo_rule.environ):
- self._write(f"* `{name}`\n")
+ self._write(":envvars: ", ", ".join(sorted(repo_rule.environ)))
self._write("\n")
def _render_rule(self, rule: stardoc_output_pb2.RuleInfo):
rule_name = rule.rule_name
- rule_anchor = _anchor_id(rule_name)
- self._write(
- _block_attrs(".starlark-object"),
- f"## {rule_name}\n\n",
- )
-
+ _sort_attributes_inplace(rule.attribute)
+ self._write("::::{bzl:rule} ")
self._render_signature(
rule_name,
- rule_anchor,
rule.attribute,
get_name=lambda r: r.name,
get_default=lambda r: r.default_value,
)
-
self._write(rule.doc_string.strip(), "\n\n")
- if len(rule.advertised_providers.provider_name) == 0:
- self._write("_Provides_: no providers advertised.")
- else:
- self._write(
- "_Provides_: ",
- ", ".join(rule.advertised_providers.provider_name),
- )
- self._write("\n\n")
+ if rule.advertised_providers.provider_name:
+ self._write(":provides: ")
+ self._write(" | ".join(rule.advertised_providers.provider_name))
+ self._write("\n")
+ self._write("\n")
if rule.attribute:
- self._render_attributes(rule_anchor, rule.attribute)
+ self._render_attributes(rule.attribute)
+ self._write("\n")
+ self._write("::::\n")
def _rule_attr_type_string(self, attr: stardoc_output_pb2.AttributeInfo) -> str:
if attr.type == _AttributeType.NAME:
- return _link("Name", ref="target-name")
+ return "Name"
elif attr.type == _AttributeType.INT:
- return _link("int", ref="int")
+ return "int"
elif attr.type == _AttributeType.LABEL:
- return _link("label", ref="attr-label")
+ return "label"
elif attr.type == _AttributeType.STRING:
- return _link("string", ref="str")
+ return "str"
elif attr.type == _AttributeType.STRING_LIST:
- return "list of " + _link("string", ref="str")
+ return "list[str]"
elif attr.type == _AttributeType.INT_LIST:
- return "list of " + _link("int", ref="int")
+ return "list[int]"
elif attr.type == _AttributeType.LABEL_LIST:
- return "list of " + _link("label", ref="attr-label") + "s"
+ return "list[label]"
elif attr.type == _AttributeType.BOOLEAN:
- return _link("bool", ref="bool")
+ return "bool"
elif attr.type == _AttributeType.LABEL_STRING_DICT:
- return "dict of {key} to {value}".format(
- key=_link("label", ref="attr-label"), value=_link("string", ref="str")
- )
+ return "dict[label, str]"
elif attr.type == _AttributeType.STRING_DICT:
- return "dict of {key} to {value}".format(
- key=_link("string", ref="str"), value=_link("string", ref="str")
- )
+ return "dict[str, str]"
elif attr.type == _AttributeType.STRING_LIST_DICT:
- return "dict of {key} to list of {value}".format(
- key=_link("string", ref="str"), value=_link("string", ref="str")
- )
+ return "dict[str, list[str]]"
elif attr.type == _AttributeType.OUTPUT:
- return _link("label", ref="attr-label")
+ return "label"
elif attr.type == _AttributeType.OUTPUT_LIST:
- return "list of " + _link("label", ref="attr-label")
+ return "list[label]"
else:
# If we get here, it means the value was unknown for some reason.
# Rather than error, give some somewhat understandable value.
return _AttributeType.Name(attr.type)
def _render_func(self, func: stardoc_output_pb2.StarlarkFunctionInfo):
- func_name = func.function_name
- func_anchor = _anchor_id(func_name)
- self._write(
- _block_attrs(".starlark-object"),
- f"## {func_name}\n\n",
- )
+ self._write("::::::{bzl:function} ")
- parameters = [param for param in func.parameter if param.name != "self"]
-
- self._render_signature(
- func_name,
- func_anchor,
- parameters,
- get_name=lambda p: p.name,
- get_default=lambda p: p.default_value,
- )
+ parameters = self._render_func_signature(func)
self._write(func.doc_string.strip(), "\n\n")
if parameters:
- self._write(
- _block_attrs(f"{func_anchor}_parameters"),
- "**PARAMETERS** ",
- _link_here_icon(f"{func_anchor}_parameters"),
- "\n\n",
- )
- entries = []
for param in parameters:
- entries.append(
- [
- f"{func_anchor}_{param.name}",
- param.name,
- f"(_default `{param.default_value}`_) "
- if param.default_value
- else "",
- param.doc_string if param.doc_string else "_undocumented_",
- ]
- )
- self._render_field_list(entries)
+ self._write(f":arg {param.name}:\n")
+ if param.default_value:
+ default_value = self._format_default_value(param.default_value)
+ self._write(" {default-value}`", default_value, "`\n")
+ if param.doc_string:
+ self._write(" ", _indent_block_text(param.doc_string), "\n")
+ else:
+ self._write(" _undocumented_\n")
+ self._write("\n")
- if getattr(func, "return").doc_string:
- return_doc = _indent_block_text(getattr(func, "return").doc_string)
- self._write(
- _block_attrs(f"{func_anchor}_returns"),
- "RETURNS",
- _link_here_icon(func_anchor + "_returns"),
- "\n",
- ": ",
- return_doc,
- "\n",
- )
+ if return_doc := getattr(func, "return").doc_string:
+ self._write(":returns:\n")
+ self._write(" ", _indent_block_text(return_doc), "\n")
if func.deprecated.doc_string:
- self._write(
- "\n\n**DEPRECATED**\n\n", func.deprecated.doc_string.strip(), "\n"
- )
+ self._write(":::::{deprecated}: unknown\n")
+ self._write(" ", _indent_block_text(func.deprecated.doc_string), "\n")
+ self._write(":::::\n")
+ self._write("::::::\n")
+
+ def _render_func_signature(self, func):
+ self._write(f"{func.function_name}(")
+ # TODO: Have an "is method" directive in the docstring to decide if
+ # the self parameter should be removed.
+ parameters = [param for param in func.parameter if param.name != "self"]
+
+ # Unfortunately, the stardoc info is incomplete and inaccurate:
+ # * The position of the `*args` param is wrong; it'll always
+ # be last (or second to last, if kwargs is present).
+ # * Stardoc doesn't explicitly tell us if an arg is `*args` or
+ # `**kwargs`. Hence f(*args) or f(**kwargs) is ambigiguous.
+ # See these issues:
+ # https://github.com/bazelbuild/stardoc/issues/226
+ # https://github.com/bazelbuild/stardoc/issues/225
+ #
+ # Below, we try to take what info we have and infer what the original
+ # signature was. In short:
+ # * A default=empty, mandatory=false arg is either *args or **kwargs
+ # * If two of those are seen, the first is *args and the second is
+ # **kwargs. Recall, however, the position of *args is mis-represented.
+ # * If a single default=empty, mandatory=false arg is found, then
+ # it's ambiguous as to whether its *args or **kwargs. To figure
+ # that out, we:
+ # * If it's not the last arg, then it must be *args. In practice,
+ # this never occurs due to #226 above.
+ # * If we saw a mandatory arg after an optional arg, then *args
+ # was supposed to be between them (otherwise it wouldn't be
+ # valid syntax).
+ # * Otherwise, it's ambiguous. We just guess by looking at the
+ # parameter name.
+ var_args = None
+ var_kwargs = None
+ saw_mandatory_after_optional = False
+ first_mandatory_after_optional_index = None
+ optionals_started = False
+ for i, p in enumerate(parameters):
+ optionals_started = optionals_started or not p.mandatory
+ if p.mandatory and optionals_started:
+ saw_mandatory_after_optional = True
+ if first_mandatory_after_optional_index is None:
+ first_mandatory_after_optional_index = i
+
+ if not p.default_value and not p.mandatory:
+ if var_args is None:
+ var_args = (i, p)
+ else:
+ var_kwargs = p
+
+ if var_args and not var_kwargs:
+ if var_args[0] != len(parameters) - 1:
+ pass
+ elif saw_mandatory_after_optional:
+ var_kwargs = var_args[1]
+ var_args = None
+ elif var_args[1].name in ("kwargs", "attrs"):
+ var_kwargs = var_args[1]
+ var_args = None
+
+ # Partial workaround for
+ # https://github.com/bazelbuild/stardoc/issues/226: `*args` renders last
+ if var_args and var_kwargs and first_mandatory_after_optional_index is not None:
+ parameters.pop(var_args[0])
+ parameters.insert(first_mandatory_after_optional_index, var_args[1])
+
+ # The only way a mandatory-after-optional can occur is
+ # if there was `*args` before it. But if we didn't see it,
+ # it must have been the unbound `*` symbol, which stardoc doesn't
+ # tell us exists.
+ if saw_mandatory_after_optional and not var_args:
+ self._write("*, ")
+ for _, is_last, p in _position_iter(parameters):
+ if var_args and p.name == var_args[1].name:
+ self._write("*")
+ elif var_kwargs and p.name == var_kwargs.name:
+ self._write("**")
+ self._write(p.name)
+ if p.default_value:
+ self._write("=", self._format_default_value(p.default_value))
+ if not is_last:
+ self._write(", ")
+ self._write(")\n")
+ return parameters
def _render_provider(self, provider: stardoc_output_pb2.ProviderInfo):
- self._write(
- _block_attrs(".starlark-object"),
- f"## {provider.provider_name}\n\n",
- )
-
- provider_anchor = _anchor_id(provider.provider_name)
- self._render_signature(
- provider.provider_name,
- provider_anchor,
- provider.field_info,
- get_name=lambda f: f.name,
- )
+ self._write("::::::{bzl:provider} ", provider.provider_name, "\n")
+ if provider.origin_key:
+ self._render_origin_key_option(provider.origin_key)
+ self._write("\n")
self._write(provider.doc_string.strip(), "\n\n")
- if provider.field_info:
- self._write(
- _block_attrs(provider_anchor),
- "**FIELDS** ",
- _link_here_icon(provider_anchor + "_fields"),
- "\n",
- "\n",
- )
- entries = []
- for field in provider.field_info:
- entries.append(
- [
- f"{provider_anchor}_{field.name}",
- field.name,
- field.doc_string,
- ]
- )
- self._render_field_list(entries)
-
- def _render_attributes(
- self, base_anchor: str, attributes: list[stardoc_output_pb2.AttributeInfo]
- ):
- self._write(
- _block_attrs(f"{base_anchor}_attributes"),
- "**ATTRIBUTES** ",
- _link_here_icon(f"{base_anchor}_attributes"),
- "\n",
+ self._write(":::::{bzl:function} ")
+ provider.field_info.sort(key=lambda f: f.name)
+ self._render_signature(
+ "<init>",
+ provider.field_info,
+ get_name=lambda f: f.name,
)
- entries = []
- for attr in attributes:
- anchor = f"{base_anchor}_{attr.name}"
- required = "required" if attr.mandatory else "optional"
- attr_type = self._rule_attr_type_string(attr)
- default = f", default `{attr.default_value}`" if attr.default_value else ""
- providers_parts = []
- if attr.provider_name_group:
- providers_parts.append("\n\n_Required providers_: ")
- if len(attr.provider_name_group) == 1:
- provider_group = attr.provider_name_group[0]
- if len(provider_group.provider_name) == 1:
- providers_parts.append(provider_group.provider_name[0])
- else:
- providers_parts.extend(
- ["all of ", _join_csv_and(provider_group.provider_name)]
- )
- elif len(attr.provider_name_group) > 1:
- providers_parts.append("any of \n")
- for group in attr.provider_name_group:
- providers_parts.extend(["* ", _join_csv_and(group.provider_name)])
- if providers_parts:
- providers_parts.append("\n")
+ # TODO: Add support for provider.init once our Bazel version supports
+ # that field
+ self._write(":::::\n")
- entries.append(
- [
- anchor,
- attr.name,
- f"_({required} {attr_type}{default})_\n",
- attr.doc_string,
- *providers_parts,
- ]
- )
- self._render_field_list(entries)
+ for field in provider.field_info:
+ self._write(":::::{bzl:provider-field} ", field.name, "\n")
+ self._write(field.doc_string.strip())
+ self._write("\n")
+ self._write(":::::\n")
+ self._write("::::::\n")
+
+ def _render_attributes(self, attributes: list[stardoc_output_pb2.AttributeInfo]):
+ for attr in attributes:
+ attr_type = self._rule_attr_type_string(attr)
+ self._write(f":attr {attr.name}:\n")
+ if attr.default_value:
+ self._write(" {bzl:default-value}`%s`\n" % attr.default_value)
+ self._write(" {type}`%s`\n" % attr_type)
+ self._write(" ", _indent_block_text(attr.doc_string), "\n")
+ self._write(" :::{bzl:attr-info} Info\n")
+ if attr.mandatory:
+ self._write(" :mandatory:\n")
+ self._write(" :::\n")
+ self._write("\n")
+
+ if attr.provider_name_group:
+ self._write(" {required-providers}`")
+ for _, outer_is_last, provider_group in _position_iter(
+ attr.provider_name_group
+ ):
+ pairs = list(
+ zip(
+ provider_group.origin_key,
+ provider_group.provider_name,
+ strict=True,
+ )
+ )
+ if len(pairs) > 1:
+ self._write("[")
+ for _, inner_is_last, (origin_key, name) in _position_iter(pairs):
+ if origin_key.file == "<native>":
+ origin = origin_key.name
+ else:
+ origin = f"{origin_key.file}%{origin_key.name}"
+ # We have to use "title <ref>" syntax because the same
+ # name might map to different origins. Stardoc gives us
+ # the provider's actual name, not the name of the symbol
+ # used in the source.
+ self._write(f"'{name} <{origin}>'")
+ if not inner_is_last:
+ self._write(", ")
+
+ if len(pairs) > 1:
+ self._write("]")
+
+ if not outer_is_last:
+ self._write(" | ")
+ self._write("`\n")
+
+ self._write("\n")
def _render_signature(
self,
name: str,
- base_anchor: str,
parameters: list[_T],
*,
get_name: Callable[_T, str],
get_default: Callable[_T, str] = lambda v: None,
):
- self._write(_block_attrs(".starlark-signature"), name, "(")
+ self._write(name, "(")
for _, is_last, param in _position_iter(parameters):
param_name = get_name(param)
- self._write(_link(param_name, f"{base_anchor}_{param_name}"))
+ self._write(f"{param_name}")
default_value = get_default(param)
if default_value:
+ default_value = self._format_default_value(default_value)
self._write(f"={default_value}")
if not is_last:
- self._write(",\n")
+ self._write(", ")
self._write(")\n\n")
- def _render_field_list(self, entries: list[list[str]]):
- """Render a list of field lists.
+ def _render_origin_key_option(self, origin_key, indent=""):
+ self._write(
+ indent,
+ ":origin-key: ",
+ self._format_option_value(f"{origin_key.file}%{origin_key.name}"),
+ "\n",
+ )
- Args:
- entries: list of field list entries. Each element is 3
- pieces: an anchor, field description, and one or more
- text strings for the body of the field list entry.
- """
- for anchor, description, *body_pieces in entries:
- body_pieces = [_block_attrs(anchor), *body_pieces]
- self._write(
- ":",
- _span(description + _link_here_icon(anchor)),
- ":\n ",
- # The text has to be indented to be associated with the block correctly.
- "".join(body_pieces).strip().replace("\n", "\n "),
- "\n",
- )
- # Ensure there is an empty line after the field list, otherwise
- # the next line of content will fold into the field list
- self._write("\n")
+ def _format_default_value(self, default_value):
+ # Handle <function foo from //baz:bar.bzl>
+ # For now, just use quotes for lack of a better option
+ if default_value.startswith("<"):
+ return f"'{default_value}'"
+ elif default_value.startswith("Label("):
+ # Handle Label(*, "@some//label:target")
+ start_quote = default_value.find('"')
+ end_quote = default_value.rfind('"')
+ return default_value[start_quote : end_quote + 1]
+ else:
+ return default_value
+
+ def _format_option_value(self, value):
+ # Leading @ symbols are special markup; escape them.
+ if value.startswith("@"):
+ return "\\" + value
+ else:
+ return value
def _write(self, *lines: str):
self._out_stream.writelines(lines)
@@ -452,21 +485,15 @@
*,
proto: pathlib.Path,
output: pathlib.Path,
- footer: pathlib.Path,
public_load_path: str,
):
- if footer:
- footer_content = footer.read_text()
-
module = stardoc_output_pb2.ModuleInfo.FromString(proto.read_bytes())
with output.open("wt", encoding="utf8") as out_stream:
_MySTRenderer(module, out_stream, public_load_path).render()
- out_stream.write(footer_content)
def _create_parser():
parser = argparse.ArgumentParser(fromfile_prefix_chars="@")
- parser.add_argument("--footer", dest="footer", type=pathlib.Path)
parser.add_argument("--proto", dest="proto", type=pathlib.Path)
parser.add_argument("--output", dest="output", type=pathlib.Path)
parser.add_argument("--public-load-path", dest="public_load_path")
@@ -478,7 +505,6 @@
_convert(
proto=options.proto,
output=options.output,
- footer=options.footer,
public_load_path=options.public_load_path,
)
return 0
diff --git a/sphinxdocs/private/sphinx_server.py b/sphinxdocs/private/sphinx_server.py
index e71889a..1f4fae8 100644
--- a/sphinxdocs/private/sphinx_server.py
+++ b/sphinxdocs/private/sphinx_server.py
@@ -2,6 +2,7 @@
import errno
import os
import sys
+import time
from http import server
@@ -17,17 +18,33 @@
address = ("0.0.0.0", 8000)
# with server.ThreadingHTTPServer(address, DirectoryHandler) as (ip, port, httpd):
with _start_server(DirectoryHandler, "0.0.0.0", 8000) as (ip, port, httpd):
- print(f"Serving...")
- print(f" Address: http://{ip}:{port}")
- print(f" Serving directory: {serve_directory}")
- print(f" CWD: {os.getcwd()}")
- print()
- print("*** You do not need to restart this server to see changes ***")
- print()
- try:
- httpd.serve_forever()
- except KeyboardInterrupt:
- pass
+
+ def _print_server_info():
+ print(f"Serving...")
+ print(f" Address: http://{ip}:{port}")
+ print(f" Serving directory: {serve_directory}")
+ print(f" url: file://{serve_directory}")
+ print(f" Server CWD: {os.getcwd()}")
+ print()
+ print("*** You do not need to restart this server to see changes ***")
+ print("*** CTRL+C once to reprint this info ***")
+ print("*** CTRL+C twice to exit ***")
+ print()
+
+ while True:
+ _print_server_info()
+ try:
+ httpd.serve_forever()
+ except KeyboardInterrupt:
+ _print_server_info()
+ print(
+ "*** KeyboardInterrupt received: CTRL+C again to terminate server ***"
+ )
+ try:
+ time.sleep(1)
+ print("Restarting serving ...")
+ except KeyboardInterrupt:
+ break
return 0
@@ -37,6 +54,7 @@
try:
with server.ThreadingHTTPServer((ip, port), handler) as httpd:
yield ip, port, httpd
+ return
except OSError as e:
if e.errno == errno.EADDRINUSE:
pass
diff --git a/sphinxdocs/private/sphinx_stardoc.bzl b/sphinxdocs/private/sphinx_stardoc.bzl
index 810dca3..e2b1756 100644
--- a/sphinxdocs/private/sphinx_stardoc.bzl
+++ b/sphinxdocs/private/sphinx_stardoc.bzl
@@ -19,7 +19,7 @@
load("@io_bazel_stardoc//stardoc:stardoc.bzl", "stardoc")
load("//python/private:util.bzl", "add_tag", "copy_propagating_kwargs") # buildifier: disable=bzl-visibility
-def sphinx_stardocs(name, docs, footer = None, **kwargs):
+def sphinx_stardocs(name, docs, **kwargs):
"""Generate Sphinx-friendly Markdown docs using Stardoc for bzl libraries.
A `build_test` for the docs is also generated to ensure Stardoc is able
@@ -39,7 +39,6 @@
* A `dict` with keys `input` and `dep`. The `input` key is a string
label to the bzl file to generate docs for. The `dep` key is a
string label to a `bzl_library` providing the necessary dependencies.
- footer: optional [`label`] File to append to generated docs.
**kwargs: Additional kwargs to pass onto each `sphinx_stardoc` target
"""
add_tag(kwargs, "@rules_python//sphinxdocs:sphinx_stardocs")
@@ -60,7 +59,6 @@
doc_name = "_{}_{}".format(name.lstrip("_"), out_name.replace("/", "_"))
_sphinx_stardoc(
name = doc_name,
- footer = footer,
out = out_name,
**stardoc_kwargs
)
@@ -77,7 +75,7 @@
**common_kwargs
)
-def _sphinx_stardoc(*, name, out, footer = None, public_load_path = None, **kwargs):
+def _sphinx_stardoc(*, name, out, public_load_path = None, **kwargs):
stardoc_name = "_{}_stardoc".format(name.lstrip("_"))
stardoc_pb = stardoc_name + ".binaryproto"
@@ -95,7 +93,6 @@
name = name,
src = stardoc_pb,
output = out,
- footer = footer,
public_load_path = public_load_path,
)
@@ -108,9 +105,6 @@
args.add("--proto", ctx.file.src)
args.add("--output", ctx.outputs.output)
- if ctx.file.footer:
- args.add("--footer", ctx.file.footer)
- inputs.append(ctx.file.footer)
if ctx.attr.public_load_path:
args.add("--public-load-path={}".format(ctx.attr.public_load_path))
@@ -126,7 +120,6 @@
_stardoc_proto_to_markdown = rule(
implementation = _stardoc_proto_to_markdown_impl,
attrs = {
- "footer": attr.label(allow_single_file = True),
"output": attr.output(mandatory = True),
"public_load_path": attr.string(),
"src": attr.label(allow_single_file = True, mandatory = True),
diff --git a/sphinxdocs/src/sphinx_stardoc/stardoc.py b/sphinxdocs/src/sphinx_stardoc/stardoc.py
index 283fb67..be38d8a 100644
--- a/sphinxdocs/src/sphinx_stardoc/stardoc.py
+++ b/sphinxdocs/src/sphinx_stardoc/stardoc.py
@@ -133,6 +133,9 @@
self.search_priority,
)
+ def __repr__(self):
+ return f"ObjectEntry({self.full_id=}, {self.object_type=}, {self.display_name=}, {self.index_entry.docname=})"
+
# A simple helper just to document what the index tuple nodes are.
def _index_node_tuple(
@@ -241,8 +244,12 @@
def visit_Constant(self, node: ast.Constant):
if node.value is None:
self._append(self.make_xref("None"))
+ elif isinstance(node.value, str):
+ self._append(self.make_xref(node.value))
else:
- raise InvalidValueError(f"Unexpected Constant node value: {node.value}")
+ raise InvalidValueError(
+ f"Unexpected Constant node value: ({type(node.value)}) {node.value=}"
+ )
def visit_Name(self, node: ast.Name):
xref_node = self.make_xref(node.id)
@@ -470,8 +477,9 @@
def run(self):
content_node = docutils_nodes.paragraph("", "")
- if "mandatory" in self.options:
- content_node += docutils_nodes.paragraph("", "mandatory (must be non-None)")
+ content_node += docutils_nodes.paragraph(
+ "", "mandatory" if "mandatory" in self.options else "optional"
+ )
if "executable" in self.options:
content_node += docutils_nodes.paragraph("", "Must be an executable")
@@ -496,6 +504,10 @@
* `foo(arg1, arg2=default) -> returntype`
"""
+ option_spec = sphinx_directives.ObjectDescription.option_spec | {
+ "origin-key": docutils_directives.unchanged,
+ }
+
@override
def before_content(self) -> None:
symbol_name = self.names[-1].symbol
@@ -588,7 +600,7 @@
if type_expr := self.options.get("type"):
- def make_xref(name):
+ def make_xref(name, title=None):
content_node = addnodes.desc_type(name, name)
return addnodes.pending_xref(
"",
@@ -608,25 +620,53 @@
sig_node += attr_annotation_node
if params_text:
- signature = inspect.signature_from_str(params_text)
+ try:
+ signature = inspect.signature_from_str(params_text)
+ except SyntaxError:
+ # Stardoc doesn't provide accurate info, so the reconstructed
+ # signature might not be valid syntax. Rather than fail, just
+ # provide a plain-text description of the approximate signature.
+ # See https://github.com/bazelbuild/stardoc/issues/225
+ sig_node += addnodes.desc_parameterlist(
+ # Offset by 1 to remove the surrounding parentheses
+ params_text[1:-1],
+ params_text[1:-1],
+ )
+ else:
+ last_kind = None
+ paramlist_node = addnodes.desc_parameterlist()
+ for param in signature.parameters.values():
+ if param.kind == param.KEYWORD_ONLY and last_kind in (
+ param.POSITIONAL_OR_KEYWORD,
+ param.POSITIONAL_ONLY,
+ None,
+ ):
+ # Add separator for keyword only parameter: *
+ paramlist_node += addnodes.desc_parameter(
+ "", "", addnodes.desc_sig_operator("", "*")
+ )
- paramlist_node = addnodes.desc_parameterlist()
- for param in signature.parameters.values():
- node = addnodes.desc_parameter()
- node += addnodes.desc_sig_name(rawsource="", text=param.name)
- if param.default is not param.empty:
- node += addnodes.desc_sig_operator("", "=")
- node += docutils_nodes.inline(
- "",
- param.default,
- classes=["default_value"],
- support_smartquotes=False,
- )
- paramlist_node += node
- sig_node += paramlist_node
+ last_kind = param.kind
+ node = addnodes.desc_parameter()
+ if param.kind == param.VAR_POSITIONAL:
+ node += addnodes.desc_sig_operator("", "*")
+ elif param.kind == param.VAR_KEYWORD:
+ node += addnodes.desc_sig_operator("", "**")
- if signature.return_annotation is not signature.empty:
- sig_node += addnodes.desc_returns("", signature.return_annotation)
+ node += addnodes.desc_sig_name(rawsource="", text=param.name)
+ if param.default is not param.empty:
+ node += addnodes.desc_sig_operator("", "=")
+ node += docutils_nodes.inline(
+ "",
+ param.default,
+ classes=["default_value"],
+ support_smartquotes=False,
+ )
+ paramlist_node += node
+ sig_node += paramlist_node
+
+ if signature.return_annotation is not signature.empty:
+ sig_node += addnodes.desc_returns("", signature.return_annotation)
obj_id = _BzlObjectId.from_env(self.env, relative_name)
@@ -685,9 +725,22 @@
),
)
- self.env.get_domain(self.domain).add_object(
- object_entry, alt_names=self._get_alt_names(object_entry)
- )
+ alt_names = []
+ if origin_key := self.options.get("origin-key"):
+ alt_names.append(
+ origin_key
+ # Options require \@ for leading @, but don't
+ # remove the escaping slash, so we have to do it manually
+ .lstrip("\\")
+ .lstrip("@")
+ .replace("//", "/")
+ .replace(".bzl%", ".")
+ .replace("/", ".")
+ .replace(":", ".")
+ )
+ alt_names.extend(self._get_alt_names(object_entry))
+
+ self.env.get_domain(self.domain).add_object(object_entry, alt_names=alt_names)
def _get_additional_index_types(self):
return []
@@ -1079,6 +1132,8 @@
return ""
+# TODO: Integrate with the option directive, since flags are options, afterall.
+# https://www.sphinx-doc.org/en/master/usage/domains/standard.html#directive-option
class _BzlFlag(_BzlTarget):
"""Documents a flag"""
@@ -1319,10 +1374,11 @@
label = "Bzl"
# NOTE: Most every object type has "obj" as one of the roles because
- # an object type's role determine what reftypes can refer to it. By having
- # "obj" for all of them, it allows writing :bzl:obj`foo` to restrict
- # object searching to the bzl domain. Under the hood, this domain translates
- # requests for the :any: role as lookups for :obj:
+ # an object type's role determine what reftypes (cross referencing) can
+ # refer to it. By having "obj" for all of them, it allows writing
+ # :bzl:obj`foo` to restrict object searching to the bzl domain. Under the
+ # hood, this domain translates requests for the :any: role as lookups for
+ # :obj:.
# NOTE: We also use these object types for categorizing things in the
# generated index page.
object_types = {
@@ -1334,17 +1390,16 @@
"module-extension": domains.ObjType(
"module extension", "module_extension", "obj"
),
- "provider": domains.ObjType("provider", "provider", "obj"),
- "provider-field": domains.ObjType(
- "provider field", "field", "obj"
- ), # provider field
+ # Providers are close enough to types that we include "type". This
+ # also makes :type: Foo work in directive options.
+ "provider": domains.ObjType("provider", "provider", "type", "obj"),
+ "provider-field": domains.ObjType("provider field", "field", "obj"),
"repo-rule": domains.ObjType("repository rule", "repo_rule", "obj"),
"rule": domains.ObjType("rule", "rule", "obj"),
"tag-class": domains.ObjType("tag class", "tag_class", "obj"),
"target": domains.ObjType("target", "target", "obj"), # target in a build file
- "flag": domains.ObjType(
- "flag", "flag", "target", "obj"
- ), # flag-target in a build file
+ # Flags are also targets, so include "target" for xref'ing
+ "flag": domains.ObjType("flag", "flag", "target", "obj"),
# types are objects that have a constructor and methods/attrs
"type": domains.ObjType("type", "type", "obj"),
}
@@ -1393,7 +1448,7 @@
# Objects within each doc
# dict[str, dict[str, _ObjectEntry]]
"doc_names": {},
- # Objects by a shorter name
+ # Objects by a shorter or alternative name
# dict[str, _ObjectEntry]
"alt_names": {},
}
@@ -1420,9 +1475,16 @@
node: addnodes.pending_xref,
contnode: docutils_nodes.Element,
) -> list[tuple[str, docutils_nodes.Element]]:
- ref_node = self.resolve_xref(
- env, fromdocname, builder, "obj", target, node, contnode
+ del env, node # Unused
+ entry = self._find_entry_for_xref(fromdocname, "obj", target)
+ if not entry:
+ return []
+ to_docname = entry.index_entry.docname
+ to_anchor = entry.index_entry.anchor
+ ref_node = sphinx_nodes.make_refnode(
+ builder, fromdocname, to_docname, to_anchor, contnode, title=to_anchor
)
+
matches = [(f"bzl:{entry.object_type}", ref_node)]
return matches
@@ -1454,14 +1516,21 @@
def _find_entry_for_xref(
self, fromdocname: str, object_type: str, target: str
) -> _ObjectEntry | None:
- # Normalize labels to dotted notation
+ # Normalize a variety of formats to the dotted format used internally.
+ # --@foo//:bar flags
+ # --@foo//:bar=value labels
+ # //foo:bar.bzl labels
target = (
- target.lstrip("@/:")
+ target.lstrip("@/:-")
.replace("//", "/")
.replace(".bzl%", ".")
.replace("/", ".")
.replace(":", ".")
)
+ # Elide the value part of --foo=bar flags
+ # Note that the flag value could contain `=`
+ if "=" in target:
+ target = target[: target.find("=")]
if target in self.data["doc_names"].get(fromdocname, {}):
return self.data["doc_names"][fromdocname][target]
@@ -1486,7 +1555,11 @@
alt_names,
)
if entry.full_id in self.data["objects"]:
- raise Exception(f"Object {entry.full_id} already registered")
+ existing = self.data["objects"][entry.full_id]
+ raise Exception(
+ f"Object {entry.full_id} already registered: "
+ + f"existing={existing}, incoming={entry}"
+ )
self.data["objects"][entry.full_id] = entry
self.data["objects_by_type"].setdefault(entry.object_type, {})
self.data["objects_by_type"][entry.object_type][entry.full_id] = entry
diff --git a/sphinxdocs/tests/proto_to_markdown/BUILD.bazel b/sphinxdocs/tests/proto_to_markdown/BUILD.bazel
index 2964785..09f5374 100644
--- a/sphinxdocs/tests/proto_to_markdown/BUILD.bazel
+++ b/sphinxdocs/tests/proto_to_markdown/BUILD.bazel
@@ -13,10 +13,12 @@
# limitations under the License.
load("//python:py_test.bzl", "py_test")
+load("//python/private:util.bzl", "IS_BAZEL_7_OR_HIGHER") # buildifier: disable=bzl-visibility
py_test(
name = "proto_to_markdown_test",
srcs = ["proto_to_markdown_test.py"],
+ target_compatible_with = [] if IS_BAZEL_7_OR_HIGHER else ["@platforms//:incompatible"],
deps = [
"//sphinxdocs/private:proto_to_markdown_lib",
"@dev_pip//absl_py",
diff --git a/sphinxdocs/tests/proto_to_markdown/proto_to_markdown_test.py b/sphinxdocs/tests/proto_to_markdown/proto_to_markdown_test.py
index 2f5b22e..3b664a5 100644
--- a/sphinxdocs/tests/proto_to_markdown/proto_to_markdown_test.py
+++ b/sphinxdocs/tests/proto_to_markdown/proto_to_markdown_test.py
@@ -114,21 +114,22 @@
def test_basic_rendering_everything(self):
actual = self._render(_EVERYTHING_MODULE)
+ self.assertIn("{bzl:currentfile} //pkg:foo.bzl", actual)
self.assertRegex(actual, "# //pkg:foo.bzl")
self.assertRegex(actual, "MODULE_DOC_STRING")
- self.assertRegex(actual, "## rule_1.*")
+ self.assertRegex(actual, "{bzl:rule} rule_1.*")
self.assertRegex(actual, "RULE_1_DOC_STRING")
self.assertRegex(actual, "rule_1_attr_1")
self.assertRegex(actual, "RULE_1_ATTR_1_DOC_STRING")
self.assertRegex(actual, "RULE_1_ATTR_1_DEFAULT_VALUE")
- self.assertRegex(actual, "## ProviderAlpha")
+ self.assertRegex(actual, "{bzl:provider} ProviderAlpha")
self.assertRegex(actual, "PROVIDER_ALPHA_DOC_STRING")
self.assertRegex(actual, "ProviderAlpha_field_a")
self.assertRegex(actual, "PROVIDER_ALPHA_FIELD_A_DOC_STRING")
- self.assertRegex(actual, "## function_1")
+ self.assertRegex(actual, "{bzl:function} function_1")
self.assertRegex(actual, "FUNCTION_1_DOC_STRING")
self.assertRegex(actual, "function_1_param_a")
self.assertRegex(actual, "FUNCTION_1_PARAM_A_DOC_STRING")
@@ -136,22 +137,22 @@
self.assertRegex(actual, "FUNCTION_1_RETURN_DOC_STRING")
self.assertRegex(actual, "FUNCTION_1_DEPRECATED_DOC_STRING")
- self.assertRegex(actual, "## aspect_1")
+ self.assertRegex(actual, "{bzl:aspect} aspect_1")
self.assertRegex(actual, "ASPECT_1_DOC_STRING")
self.assertRegex(actual, "aspect_1_aspect_attribute_a")
self.assertRegex(actual, "aspect_1_attribute_a")
self.assertRegex(actual, "ASPECT_1_ATTRIBUTE_A_DOC_STRING")
self.assertRegex(actual, "694638")
- self.assertRegex(actual, "## bzlmod_ext")
+ self.assertRegex(actual, "{bzl:module-extension} bzlmod_ext")
self.assertRegex(actual, "BZLMOD_EXT_DOC_STRING")
- self.assertRegex(actual, "### bzlmod_ext.bzlmod_ext_tag_a")
+ self.assertRegex(actual, "{bzl:tag-class} bzlmod_ext_tag_a")
self.assertRegex(actual, "BZLMOD_EXT_TAG_A_DOC_STRING")
self.assertRegex(actual, "bzlmod_ext_tag_a_attribute_1")
self.assertRegex(actual, "BZLMOD_EXT_TAG_A_ATTRIBUTE_1_DOC_STRING")
self.assertRegex(actual, "BZLMOD_EXT_TAG_A_ATTRIBUTE_1_DEFAULT_VALUE")
- self.assertRegex(actual, "## repository_rule")
+ self.assertRegex(actual, "{bzl:repo-rule} repository_rule")
self.assertRegex(actual, "REPOSITORY_RULE_DOC_STRING")
self.assertRegex(actual, "repository_rule_attribute_a")
self.assertRegex(actual, "REPOSITORY_RULE_ATTRIBUTE_A_DOC_STRING")
@@ -172,31 +173,25 @@
name: "param_without_default"
}
parameter: {
+ name: "param_with_function_default",
+ default_value: "<function foo from //bar:baz.bzl>"
+ }
+ parameter: {
+ name: "param_with_label_default",
+ default_value: 'Label(*, "@repo//pkg:file.bzl")'
+ }
+ parameter: {
name: "last_param"
}
}
"""
)
- self.assertIn("[param_with_default](#func_param_with_default)=DEFAULT,", actual)
- self.assertIn("[param_without_default](#func_param_without_default),", actual)
-
- def test_render_field_list(self):
- actual = self._render(
- """\
-file: "@repo//pkg:foo.bzl"
-func_info: {
- function_name: "func"
- parameter: {
- name: "param"
- default_value: "DEFAULT"
- }
-}
-"""
- )
- self.assertRegex(
- actual, re.compile("^:.*param.*¶.*headerlink.*:\n", re.MULTILINE)
- )
- self.assertRegex(actual, re.compile("^ .*#func_param", re.MULTILINE))
+ self.assertIn("param_with_default=DEFAULT,", actual)
+ self.assertIn("{default-value}`DEFAULT`", actual)
+ self.assertIn(":arg param_with_default:", actual)
+ self.assertIn("param_without_default,", actual)
+ self.assertIn('{default-value}`"@repo//pkg:file.bzl"`', actual)
+ self.assertIn("{default-value}`'<function foo from //bar:baz.bzl>'", actual)
if __name__ == "__main__":
diff --git a/sphinxdocs/tests/sphinx_stardoc/BUILD.bazel b/sphinxdocs/tests/sphinx_stardoc/BUILD.bazel
index 5cf5736..63e34fb 100644
--- a/sphinxdocs/tests/sphinx_stardoc/BUILD.bazel
+++ b/sphinxdocs/tests/sphinx_stardoc/BUILD.bazel
@@ -1,4 +1,7 @@
+load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
+load("//python/private:util.bzl", "IS_BAZEL_7_OR_HIGHER") # buildifier: disable=bzl-visibility
load("//sphinxdocs:sphinx.bzl", "sphinx_build_binary", "sphinx_docs")
+load("//sphinxdocs:sphinx_stardoc.bzl", "sphinx_stardocs")
sphinx_docs(
name = "docs",
@@ -6,7 +9,7 @@
include = [
"*.md",
],
- ),
+ ) + [":bzl_docs"],
config = "conf.py",
formats = [
"html",
@@ -22,7 +25,31 @@
"@platforms//os:linux": [],
"@platforms//os:macos": [],
"//conditions:default": ["@platforms//:incompatible"],
- }),
+ }) if IS_BAZEL_7_OR_HIGHER else ["@platforms//:incompatible"],
+)
+
+sphinx_stardocs(
+ name = "bzl_docs",
+ docs = {
+ "bzl_function.md": dict(
+ dep = ":all_bzl",
+ input = "//sphinxdocs/tests/sphinx_stardoc:bzl_function.bzl",
+ ),
+ "bzl_providers.md": dict(
+ dep = ":all_bzl",
+ input = "//sphinxdocs/tests/sphinx_stardoc:bzl_providers.bzl",
+ ),
+ "bzl_rule.md": dict(
+ dep = ":all_bzl",
+ input = "//sphinxdocs/tests/sphinx_stardoc:bzl_rule.bzl",
+ ),
+ },
+ target_compatible_with = [] if IS_BAZEL_7_OR_HIGHER else ["@platforms//:incompatible"],
+)
+
+bzl_library(
+ name = "all_bzl",
+ srcs = glob(["*.bzl"]),
)
sphinx_build_binary(
diff --git a/sphinxdocs/tests/sphinx_stardoc/bzl_function.bzl b/sphinxdocs/tests/sphinx_stardoc/bzl_function.bzl
new file mode 100644
index 0000000..822ff26
--- /dev/null
+++ b/sphinxdocs/tests/sphinx_stardoc/bzl_function.bzl
@@ -0,0 +1,34 @@
+"""Tests for plain functions."""
+
+def middle_varargs(a, *args, b):
+ """Expect: `middle_varargs(a, *args, b)`
+
+ NOTE: https://github.com/bazelbuild/stardoc/issues/226: `*args` renders last
+
+ Args:
+ a: {type}`str` doc for a
+ *args: {type}`varags` doc for *args
+ b: {type}`list[str]` doc for c
+
+ """
+ _ = a, args, b # @unused
+
+def mixture(a, b = 1, *args, c, d = 2, **kwargs):
+ """Expect: `mixture(a, b=1, *args, c, d=2, **kwargs)`"""
+ _ = a, b, args, c, d, kwargs # @unused
+
+def only_varargs(*args):
+ """Expect: `only_varargs(*args)`"""
+ _ = args # @unused
+
+def only_varkwargs(**kwargs):
+ """Expect: `only_varkwargs(**kwargs)`"""
+ _ = kwargs # @unused
+
+def unnamed_varargs(*, a = 1, b):
+ """Expect: unnamed_varargs(*, a=1, b)"""
+ _ = a, b # @unused
+
+def varargs_and_varkwargs(*args, **kwargs):
+ """Expect: `varargs_and_varkwargs(*args, **kwargs)`"""
+ _ = args, kwargs # @unused
diff --git a/sphinxdocs/tests/sphinx_stardoc/bzl_providers.bzl b/sphinxdocs/tests/sphinx_stardoc/bzl_providers.bzl
new file mode 100644
index 0000000..189d975
--- /dev/null
+++ b/sphinxdocs/tests/sphinx_stardoc/bzl_providers.bzl
@@ -0,0 +1,4 @@
+"""Providers"""
+
+# buildifier: disable=provider-params
+GenericInfo = provider()
diff --git a/sphinxdocs/tests/sphinx_stardoc/bzl_rule.bzl b/sphinxdocs/tests/sphinx_stardoc/bzl_rule.bzl
new file mode 100644
index 0000000..d17c8bc
--- /dev/null
+++ b/sphinxdocs/tests/sphinx_stardoc/bzl_rule.bzl
@@ -0,0 +1,24 @@
+"""Tests for rules."""
+
+load(":bzl_providers.bzl", OtherGenericInfo = "GenericInfo")
+
+# buildifier: disable=provider-params
+GenericInfo = provider()
+
+# buildifier: disable=provider-params
+P1 = provider()
+
+# buildifier: disable=provider-params
+P2 = provider()
+
+def _impl(ctx):
+ _ = ctx # @unused
+
+my_rule = rule(
+ implementation = _impl,
+ attrs = {
+ "srcs": attr.label(
+ providers = [[GenericInfo], [OtherGenericInfo], [P1, P2], [platform_common.ToolchainInfo]],
+ ),
+ },
+)
diff --git a/sphinxdocs/tests/sphinx_stardoc/function.md b/sphinxdocs/tests/sphinx_stardoc/function.md
index b8cbd37..de7d16a 100644
--- a/sphinxdocs/tests/sphinx_stardoc/function.md
+++ b/sphinxdocs/tests/sphinx_stardoc/function.md
@@ -9,7 +9,7 @@
Module documentation
-:::{bzl:function} myfunc(foo, bar=False, baz=[]) -> FooObj
+::::::{bzl:function} myfunc(foo, bar=False, baz=[]) -> FooObj
This is a bazel function.
@@ -34,8 +34,13 @@
{bzl:return-type}`list | int`
description
+:::{deprecated} unspecified
+
+Some doc about the deprecation
:::
+::::::
+
:::{bzl:function} mylongfunc(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9)
:::
diff --git a/sphinxdocs/tests/sphinx_stardoc/rule.md b/sphinxdocs/tests/sphinx_stardoc/rule.md
index a6f3a56..0f90ed3 100644
--- a/sphinxdocs/tests/sphinx_stardoc/rule.md
+++ b/sphinxdocs/tests/sphinx_stardoc/rule.md
@@ -23,7 +23,7 @@
:mandatory: true
:::
- {required-providers}`LangInfo | [OtherLangInfo, AnotherLangInfo]`
+ {required-providers}`"Display <//lang:provider.bzl%LangInfo>"`
:attr ra2:
{type}`attr.label`
diff --git a/sphinxdocs/tests/sphinx_stardoc/xrefs.md b/sphinxdocs/tests/sphinx_stardoc/xrefs.md
index f0ea038..9eb7b81 100644
--- a/sphinxdocs/tests/sphinx_stardoc/xrefs.md
+++ b/sphinxdocs/tests/sphinx_stardoc/xrefs.md
@@ -48,3 +48,11 @@
* rule: {obj}`lang.rule.my_rule`
* rule attr: {obj}`lang.rule.my_rule.ra1`
* provider: {obj}`lang.provider.LangInfo`
+
+## Using origin keys
+
+* provider using `{type}`: {type}`"@rules_python//sphinxdocs/tests/sphinx_stardoc:bzl_rule.bzl%GenericInfo"`
+
+## Any xref
+
+* {any}`LangInfo`
