[feat/precompile-tooling] docs: document the opt-in precompiled mode (#6138)
* docs: document the opt-in precompiled mode
Assisted-by: ClaudeCode:claude-fable-5
* chore: lint the -inl.h invariants of the precompiled mode
tools/check_inl_headers.py checks that each namespace-scope definition
in an -inl.h file has PYBIND11_INLINE, and that each macro in a
preprocessor condition is known: either the same for the library and
the module, or encoded in the precompiled config guard. Also document
that a custom PYBIND11_NAMESPACE must reach the library.
Assisted-by: ClaudeCode:claude-opus-5-5
* docs: note the call overhead of a non-LTO precompiled library
Release build, trivial bound functions, against an LTO header-only
module: AppleClang arm64 adds 2-4 ns per call (7-11%), GCC 15 Linux
aarch64 adds 1-2 ns (1-7%). With LTO on the library, both are within a
few percent.
Assisted-by: ClaudeCode:claude-opus-5-5
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 6251152..d61e642 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -141,6 +141,11 @@
language: pygrep
entry: PyBind|\bNumpy\b|Cmake|CCache|PyTest
exclude: ^\.pre-commit-config.yaml$
+ - id: check-inl-headers
+ name: Check -inl.h files for the precompiled mode
+ language: python
+ entry: python tools/check_inl_headers.py
+ files: -inl\.h$
# PyLint has native support - not always usable, but works for us
- repo: https://github.com/PyCQA/pylint
diff --git a/docs/compiling.rst b/docs/compiling.rst
index a6bee86..a81b16f 100644
--- a/docs/compiling.rst
+++ b/docs/compiling.rst
@@ -348,7 +348,8 @@
.. code-block:: cmake
pybind11_add_module(<name> [MODULE | SHARED] [EXCLUDE_FROM_ALL]
- [NO_EXTRAS] [THIN_LTO] [OPT_SIZE] source1 [source2 ...])
+ [NO_EXTRAS] [THIN_LTO] [OPT_SIZE] [PRECOMPILE | NO_PRECOMPILE]
+ source1 [source2 ...])
This function behaves very much like CMake's builtin ``add_library`` (in fact,
it's a wrapper function around that command). It will add a library target
@@ -404,6 +405,104 @@
.. _ThinLTO: http://clang.llvm.org/docs/ThinLTO.html
+.. _precompile-mode:
+
+Pre-compiling part of pybind11
+------------------------------
+
+pybind11 is header-only by default: every translation unit compiles its own
+copy of the non-template implementation. The opt-in *precompiled* mode
+compiles that implementation once, into a static library built inside your
+own project with your own flags. This reduces the build time, most of all for
+projects with many translation units or many modules in one build.
+
+.. code-block:: cmake
+
+ pybind11_add_module(example PRECOMPILE example.cpp)
+
+The first ``PRECOMPILE`` target creates the library target
+``pybind11::precompiled``; further targets reuse it. Set the CMake variable
+``PYBIND11_PRECOMPILE`` to make it the default for all
+``pybind11_add_module`` calls; use ``NO_PRECOMPILE`` on a target to opt back
+out. For targets you create yourself, call the ``pybind11_precompile()``
+function and link ``pybind11::precompiled`` PRIVATE; the target carries the
+required ``PYBIND11_PRECOMPILED`` compile definition PUBLIC, so your sources
+also get it.
+
+Requirements and caveats:
+
+* The library and every module linking it must agree on the configuration
+ macros ``PYBIND11_INTERNALS_VERSION``, ``Py_GIL_DISABLED``,
+ ``PYBIND11_SIMPLE_GIL_MANAGEMENT``,
+ ``PYBIND11_DETAILED_ERROR_MESSAGES`` (defaults on in debug builds),
+ ``PYBIND11_HAS_SUBINTERPRETER_SUPPORT``, and
+ ``PYBIND11_BACKWARD_COMPATIBILITY_TP_DICTOFFSET``. A mismatch produces one
+ readable undefined symbol at link time referencing
+ ``pybind11_precompiled_config``.
+* Configuration macros that only change code inside the library (for example
+ ``PYBIND11_DISABLE_NEW_STYLE_INIT_WARNING``) must be defined when the
+ library is compiled; a definition only on your module has no effect. This
+ also applies to a custom ``PYBIND11_NAMESPACE`` visibility attribute: set
+ it at the directory level (``add_compile_definitions``) so that the library
+ gets it too.
+* The library picks up your directory-level flags and C++ standard when it is
+ first created, so set those before the first ``PRECOMPILE`` target. A
+ status message reports the directory that created the library.
+* The library is not compiled with link-time optimization, and the per-target
+ ``THIN_LTO`` and ``OPT_SIZE`` options of ``pybind11_add_module`` do not
+ apply to it. In a Release build this adds a few nanoseconds to each call of
+ a bound function (up to about 10% for a function that does nothing,
+ depending on the compiler; a few percent with LTO on the library). To change this, call ``pybind11_precompile()``
+ yourself and
+ set the properties on the created target, ``pybind11_precompiled`` (the
+ real target behind the ``pybind11::precompiled`` alias; CMake does not let
+ you set properties through an alias):
+
+ .. code-block:: cmake
+
+ pybind11_precompile()
+ set_target_properties(pybind11_precompiled PROPERTIES
+ INTERPROCEDURAL_OPTIMIZATION ON)
+
+* The library is static and per-build-tree; it is never installed or shared
+ between projects. Each extension module links its own copy, which keeps
+ pybind11's per-module state the same as in header-only mode.
+* Not available with ``PYBIND11_NOPYTHON`` (the library needs Python
+ headers).
+
+For build systems other than CMake, the same sources ship with the pybind11
+package: compile ``pybind11_combined.cpp`` from the directory reported by
+``python -m pybind11 --srcdir`` (also available as
+``pybind11.get_source_dir()`` and the ``srcdir`` pkg-config variable) into
+a static library or into your extension, and define
+``PYBIND11_PRECOMPILED`` for every translation unit.
+
+With Meson, build the library once per build tree and link it into each
+extension module, the same as the CMake path:
+
+.. code-block:: meson
+
+ pybind11_dep = dependency('pybind11')
+ pybind11_src = run_command(py, ['-m', 'pybind11', '--srcdir'],
+ check : true).stdout().strip()
+
+ pybind11_precompiled = static_library('pybind11_precompiled',
+ pybind11_src / 'pybind11_combined.cpp',
+ cpp_args : ['-DPYBIND11_PRECOMPILED'],
+ gnu_symbol_visibility : 'hidden',
+ dependencies : [pybind11_dep, py.dependency()])
+
+ py.extension_module('example', 'example.cpp',
+ cpp_args : ['-DPYBIND11_PRECOMPILED'],
+ link_with : pybind11_precompiled,
+ dependencies : [pybind11_dep])
+
+The configuration-macro rules above apply here too: the static library and
+every module that links it must be compiled with the same configuration
+macros, and ``-DPYBIND11_PRECOMPILED`` must appear in both ``cpp_args``
+lists. (``pybind11_dep.get_variable('srcdir')`` also reports the source
+directory when Meson finds pybind11 through pkg-config.)
+
Configuration variables
-----------------------
diff --git a/docs/faq.rst b/docs/faq.rst
index 2b89d20..8a061c8 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -79,7 +79,12 @@
How can I reduce the build time?
================================
-It's good practice to split binding code over multiple files, as in the
+First, consider the opt-in precompiled mode: it compiles the non-template
+part of pybind11 once per project instead of once for each translation unit.
+In CMake, this is one keyword on ``pybind11_add_module``. See
+:ref:`precompile-mode`.
+
+It's also good practice to split binding code over multiple files, as in the
following example:
:file:`example.cpp`:
diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h
index 224198c..164da39 100644
--- a/include/pybind11/detail/internals.h
+++ b/include/pybind11/detail/internals.h
@@ -52,7 +52,8 @@
// the library and the modules linking it. PYBIND11_MODULE calls it, so a mismatch (or a
// missing library) surfaces as one readable undefined symbol at link time instead of many
// unrelated ones at run time. If you add a configuration macro that changes the code in the
-// -inl.h files, encode it here and add it to the list in docs/compiling.rst.
+// -inl.h files, encode it here, add it to the list in docs/compiling.rst, and add it to
+// tools/check_inl_headers.py.
# if defined(Py_GIL_DISABLED)
# define PYBIND11_PRECOMPILED_CFG_GD 1
# else
diff --git a/tools/check_inl_headers.py b/tools/check_inl_headers.py
new file mode 100755
index 0000000..6787fb7
--- /dev/null
+++ b/tools/check_inl_headers.py
@@ -0,0 +1,143 @@
+#!/usr/bin/env python3
+"""
+Check the invariants of the -inl.h files used by the precompiled mode:
+
+* Every namespace-scope function definition is marked PYBIND11_INLINE.
+* Every macro in a preprocessor condition is known. A macro that changes the
+ code must either be the same for the library and the modules (platform,
+ compiler, Python version), or be encoded in PYBIND11_PRECOMPILED_CONFIG_CHECK
+ (detail/internals.h) and listed in docs/compiling.rst.
+"""
+
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+# Same in the library and in the modules that link it.
+ENVIRONMENT_MACROS = {
+ "GRAALVM_PYTHON",
+ "NDEBUG",
+ "PYBIND11_BUILTIN_QUALNAME",
+ "PYBIND11_HAS_STRING_VIEW",
+ "PYBIND11_PRECOMPILED",
+ "PYPY_VERSION",
+ "PY_MAJOR_VERSION",
+ "PY_MINOR_VERSION",
+ "PY_VERSION_HEX",
+ "Py_REF_DEBUG",
+ "_MSC_VER",
+ "_WIN32",
+ "__GLIBCXX__",
+ "__GNUC__",
+ "__cpp_lib_unordered_map_try_emplace",
+ "__clang__",
+}
+
+# Encoded in PYBIND11_PRECOMPILED_CONFIG_CHECK.
+GUARDED_MACROS = {
+ "PYBIND11_BACKWARD_COMPATIBILITY_TP_DICTOFFSET",
+ "PYBIND11_DETAILED_ERROR_MESSAGES",
+ "PYBIND11_HAS_SUBINTERPRETER_SUPPORT",
+ "PYBIND11_INTERNALS_VERSION",
+ "PYBIND11_SIMPLE_GIL_MANAGEMENT",
+ "Py_GIL_DISABLED",
+}
+
+# Only change code inside the library; documented in docs/compiling.rst.
+LIBRARY_ONLY_MACROS = {
+ "PYBIND11_DISABLE_NEW_STYLE_INIT_WARNING",
+}
+
+KNOWN_MACROS = ENVIRONMENT_MACROS | GUARDED_MACROS | LIBRARY_ONLY_MACROS
+
+TOKENS = re.compile(
+ r"""
+ (?P<raw>R"(?P<delim>[^(\s]*)\(.*?\)(?P=delim)")
+ | (?P<str>"(?:\\.|[^"\\\n])*")
+ | (?P<chr>'(?:\\.|[^'\\\n])*')
+ | (?P<line_comment>//[^\n]*)
+ | (?P<block_comment>/\*.*?\*/)
+ """,
+ re.VERBOSE | re.DOTALL,
+)
+PP_CONDITION = re.compile(r"^[ \t]*#\s*(?:if|elif|ifdef|ifndef)\b(.*)$", re.MULTILINE)
+PP_LINE = re.compile(r"^[ \t]*#.*$", re.MULTILINE)
+NAMESPACE_MACRO = re.compile(
+ r"^[ \t]*PYBIND11_(?:NAMESPACE_BEGIN|NAMESPACE_END|WARNING_\w+)\(.*\)[ \t]*$",
+ re.MULTILINE,
+)
+IDENTIFIER = re.compile(r"\b[A-Za-z_]\w*\b")
+NON_FUNCTION_BLOCK = re.compile(r"^(?:struct|class|union|enum)\b|=$")
+
+
+def strip(text: str) -> str:
+ """Blank out comments and string literals, keeping the line numbers."""
+
+ def blank(match: re.Match[str]) -> str:
+ if match.group("line_comment") or match.group("block_comment"):
+ return re.sub(r"[^\n]", " ", match.group())
+ return '""' + "\n" * match.group().count("\n")
+
+ return TOKENS.sub(blank, text)
+
+
+def check(path: Path) -> list[str]:
+ text = strip(path.read_text(encoding="utf-8"))
+ errors: list[str] = []
+
+ for match in PP_CONDITION.finditer(text):
+ line = text.count("\n", 0, match.start()) + 1
+ errors.extend(
+ f"{path}:{line}: unknown configuration macro {name}; see {Path(__file__).name}"
+ for name in IDENTIFIER.findall(match.group(1))
+ if name != "defined" and name not in KNOWN_MACROS
+ )
+
+ code = PP_LINE.sub(lambda m: " " * len(m.group()), text)
+ code = NAMESPACE_MACRO.sub(lambda m: " " * len(m.group()), code)
+
+ # One entry per open brace: True for namespace and extern "C" blocks, whose
+ # contents are still at namespace scope.
+ scopes: list[bool] = []
+ start = 0
+ for pos, char in enumerate(code):
+ at_namespace_scope = all(scopes)
+ if char == "{":
+ if not at_namespace_scope:
+ scopes.append(False)
+ continue
+ header = " ".join(code[start:pos].split())
+ transparent = header == 'extern "C"' or header.startswith("namespace")
+ if (
+ not transparent
+ and "(" in header
+ and not NON_FUNCTION_BLOCK.search(header)
+ and "PYBIND11_INLINE" not in header.split()
+ ):
+ offset = start + len(code[start:pos]) - len(code[start:pos].lstrip())
+ line = code.count("\n", 0, offset) + 1
+ errors.append(f"{path}:{line}: definition without PYBIND11_INLINE")
+ scopes.append(transparent)
+ if transparent:
+ start = pos + 1
+ elif char == "}":
+ scopes.pop()
+ if all(scopes):
+ start = pos + 1
+ elif char == ";" and at_namespace_scope:
+ start = pos + 1
+
+ return errors
+
+
+def main(argv: list[str]) -> int:
+ errors = [error for arg in argv for error in check(Path(arg))]
+ for error in errors:
+ print(error)
+ return 1 if errors else 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main(sys.argv[1:]))
diff --git a/tools/pybind11Config.cmake.in b/tools/pybind11Config.cmake.in
index abcd43e..d666e9c 100644
--- a/tools/pybind11Config.cmake.in
+++ b/tools/pybind11Config.cmake.in
@@ -18,6 +18,9 @@
Directories where pybind11 and python headers are located.
``pybind11_INCLUDE_DIR``
Directory where pybind11 headers are located.
+``pybind11_SRC_DIR``
+ Directory where the library sources for the opt-in precompiled mode are
+ located (used by ``pybind11_precompile``).
``pybind11_DEFINITIONS``
Definitions necessary to use pybind11, namely USING_pybind11.
``pybind11_LIBRARIES``
@@ -147,6 +150,7 @@
pybind11_add_module(<target>
[STATIC|SHARED|MODULE]
[THIN_LTO] [OPT_SIZE] [NO_EXTRAS] [WITHOUT_SOABI]
+ [PRECOMPILE|NO_PRECOMPILE]
<files>...
)
@@ -162,6 +166,22 @@
Disable the SOABI component (``PYBIND11_FINDPYTHON`` mode only).
``NO_EXTRAS``
Disable all extras, exit immediately after making the module.
+``PRECOMPILE``
+ Link the target against the ``pybind11::precompiled`` static library
+ (created on first use); ``NO_PRECOMPILE`` opts a target out when the
+ ``PYBIND11_PRECOMPILE`` variable enables it globally.
+
+pybind11_precompile
+^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: cmake
+
+ pybind11_precompile()
+
+Create the ``pybind11::precompiled`` static library from the shipped sources
+(once per build tree). ``pybind11_add_module(... PRECOMPILE)`` calls this for
+you; call it directly to link ``pybind11::precompiled`` into your own
+targets.
pybind11_strip
^^^^^^^^^^^^^^
