fix: only add string_view life support for transient sources (#6096)
* revert: "revert: add life support to handles cast to string_view (#6092)"
This re-applies #6092 (reverting #6097) so the follow-up fixes in this PR can build on it.
Assisted-by: ClaudeCode:claude-opus-4.8
* fix: don't throw from string_view life support outside a bound function
PR #6092 added loader_life_support::add_patient(src) to keep the source
object alive when loading a string view, fixing a real use-after-free when
a container of views is built from a non-sequence iterable (e.g. a
generator): list_caster materializes a temporary tuple that owns the
strings and destroys it when load() returns, before the bound function
body runs.
add_patient throws when there is no life support frame, so casting to a
view outside a bound function (e.g. a manual py::cast<std::string_view>)
now raises instead of relying on the caller-owned source, a regression
from #6092.
For these view-into-src cases registration is best effort: inside a bound
function it keeps src alive (fixing the UAF), and outside one the caller
owns src's lifetime as before. Add try_add_patient(), which returns false
instead of throwing when there is no frame, and use it at the three view
load sites. add_patient() keeps its strict contract for value-creating
conversions.
Assisted-by: ClaudeCode:claude-opus-4.8
* fix: only add string_view life support for transient sources
Refine the previous commit. Best-effort registration (try_add_patient)
silently produces a dangling view when a container of views is built from
a generator outside a bound function: there the materialized temporary is
released before the view is used, and with no frame nothing keeps it
alive. Such a cast cannot be made safe, so it should fail loudly, while a
view into a durable, caller-owned object needs no life support at all.
The view caster cannot tell a durable source from a pybind11-managed
transient one; that provenance lives in the container caster. Introduce an
ambient transient_source_guard that the list, set, map, and array casters
set around their generator/materialized paths, and have the string caster
keep the source alive only when loading from a transient source (via the
throwing add_patient, so try_add_patient is no longer needed). This means:
- views into durable sources (direct arguments, sequences, manual casts)
add no life support and no longer throw outside a bound function, and
- a generator used outside a frame throws, rather than silently dangling.
The guard restores (rather than clears) the previous value, so a durable
container nested in a transient one is correctly treated as transient.
Verified with AddressSanitizer: the in-frame generator case is clean, the
out-of-frame durable cases succeed, and the out-of-frame generator case
throws.
Assisted-by: ClaudeCode:claude-opus-4.8
* Revert "fix: only add string_view life support for transient sources"
This reverts commit e18b8346a28c125964fcb9d192e6643d5ef3b420.
* test: cover string_view argument life support
* test: cover generated and nested string_view lifetimes
* test: cover temporary-backed string_view casts
* test: explain string_view lifetime regression tests
* docs: clarify string_view lifetime requirements
* docs: explain life support for custom view casters
---------
Co-authored-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
diff --git a/docs/advanced/cast/custom.rst b/docs/advanced/cast/custom.rst
index 786192b..de673b2 100644
--- a/docs/advanced/cast/custom.rst
+++ b/docs/advanced/cast/custom.rst
@@ -120,6 +120,24 @@
For further information on the ``return_value_policy`` argument of ``cast`` refer to :ref:`return_value_policies`.
To learn about the ``convert`` argument of ``load`` see :ref:`nonconverting_arguments`.
+.. note::
+
+ If a custom ``load()`` produces a non-owning view of storage owned by a
+ Python object, and keeping that object alive is sufficient to keep the
+ storage valid, register the owner (often ``src``) with
+ ``pybind11::detail::loader_life_support::try_add_patient(owner)``. During a
+ bound-function call, this keeps the owner alive until the call returns. With
+ no active call frame, it returns ``false`` and the caller remains
+ responsible for the lifetime.
+
+ If ``load()`` instead creates a temporary Python object to own the view's
+ storage, use the strict ``loader_life_support::add_patient(owner)``. Outside
+ a bound-function call, it raises :class:`cast_error` rather than allowing a
+ dangling view. Neither form permits C++ code to retain the view after the
+ call without separate lifetime management. These ``detail`` helpers are
+ intended only for type-caster implementations. See
+ :ref:`string_view_lifetime` for a concrete example.
+
.. warning::
When using custom type casters, it's important to declare them consistently
diff --git a/docs/advanced/cast/stl.rst b/docs/advanced/cast/stl.rst
index 1e17bc3..6abe8b4 100644
--- a/docs/advanced/cast/stl.rst
+++ b/docs/advanced/cast/stl.rst
@@ -17,6 +17,10 @@
can have implications on the program semantics and performance. Please read the
next sections for more details and alternative approaches that avoid this.
+Copying the container does not make non-owning element types own their data.
+In particular, containers of C++ string views have additional
+:ref:`string_view_lifetime` requirements.
+
.. note::
Arbitrary nesting of any of these types is possible.
diff --git a/docs/advanced/cast/strings.rst b/docs/advanced/cast/strings.rst
index 271716b..2b24d7c 100644
--- a/docs/advanced/cast/strings.rst
+++ b/docs/advanced/cast/strings.rst
@@ -280,6 +280,8 @@
no way to capture them in a C++ character type.
+.. _string_view_lifetime:
+
C++17 string views
==================
@@ -289,6 +291,30 @@
UTF-16-encoded data, and a returned ``std::string_view`` will be decoded as
UTF-8).
+A string view does not own its character data. When a view is loaded as an
+argument to a pybind11-bound function, pybind11 keeps the Python object that
+provides the data alive until the function returns. This also applies to views
+nested in automatically converted STL containers. The C++ function must not
+retain any such view after it returns unless it separately guarantees that the
+backing storage remains alive and valid.
+
+Lifetime support keeps the Python object alive, but does not prevent its storage
+from being invalidated. For example, if C++ releases the GIL or calls back into
+Python, resizing a backing ``bytearray`` while the view is in use can invalidate
+the view.
+
+A direct Python-to-C++ :func:`py::cast` made when no bound-function call is
+active has no such lifetime support. When a cast to a non-owning view succeeds,
+the caller must keep the backing Python object alive, with its storage
+unchanged, for as long as the view is used. For a container of views, this
+requirement applies to every element: retain the elements directly or through
+an unmodified owning container, and do not cast an iterable that creates
+temporary elements.
+
+Some view conversions require temporary backing storage, for example to encode
+text. Outside a bound-function call, such conversions raise
+:class:`cast_error` instead of returning a dangling view.
+
References
==========
diff --git a/docs/advanced/pycpp/object.rst b/docs/advanced/pycpp/object.rst
index 93e1a94..d8661c2 100644
--- a/docs/advanced/pycpp/object.rst
+++ b/docs/advanced/pycpp/object.rst
@@ -76,6 +76,10 @@
When conversion fails, both directions throw the exception :class:`cast_error`.
+When casting to a non-owning type such as ``std::string_view``, the Python
+source may need to remain alive after a successful cast. See
+:ref:`string_view_lifetime` for details.
+
.. _python_libs:
Accessing Python libraries from C++
diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h
index 62ca45a..9ab6e33 100644
--- a/include/pybind11/cast.h
+++ b/include/pybind11/cast.h
@@ -525,6 +525,11 @@
return false;
}
value = StringType(buffer, static_cast<size_t>(size));
+ if (IsView) {
+ // `src` owns the buffer; keep it alive if inside a bound function,
+ // otherwise the caller is responsible for its lifetime.
+ loader_life_support::try_add_patient(src);
+ }
return true;
}
@@ -602,6 +607,9 @@
pybind11_fail("Unexpected PYBIND11_BYTES_AS_STRING() failure.");
}
value = StringType(bytes, (size_t) PYBIND11_BYTES_SIZE(src.ptr()));
+ if (IsView) {
+ loader_life_support::try_add_patient(src);
+ }
return true;
}
if (PyByteArray_Check(src.ptr())) {
@@ -612,6 +620,9 @@
pybind11_fail("Unexpected PyByteArray_AsString() failure.");
}
value = StringType(bytearray, (size_t) PyByteArray_Size(src.ptr()));
+ if (IsView) {
+ loader_life_support::try_add_patient(src);
+ }
return true;
}
diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h
index 8fbf700..b6d03ca 100644
--- a/include/pybind11/detail/type_caster_base.h
+++ b/include/pybind11/detail/type_caster_base.h
@@ -81,11 +81,26 @@
}
}
+ /// Keep `h` alive until the current patient frame is destroyed, if there is one.
+ /// Returns false when called outside a bound function (no frame). Use this, rather
+ /// than `add_patient`, when failing to register is acceptable because the caller
+ /// owns the source's lifetime outside the call framework (e.g. a view that points
+ /// into an existing Python object, as opposed to a freshly created temporary).
+ PYBIND11_NOINLINE static bool try_add_patient(handle h) {
+ loader_life_support *frame = tls_current_frame();
+ if (!frame) {
+ return false;
+ }
+ if (frame->keep_alive.insert(h.ptr()).second) {
+ Py_INCREF(h.ptr());
+ }
+ return true;
+ }
+
/// This can only be used inside a pybind11-bound function, either by `argument_loader`
/// at argument preparation time or by `py::cast()` at execution time.
PYBIND11_NOINLINE static void add_patient(handle h) {
- loader_life_support *frame = tls_current_frame();
- if (!frame) {
+ if (!try_add_patient(h)) {
// NOTE: It would be nice to include the stack frames here, as this indicates
// use of pybind11::cast<> outside the normal call framework, finding such
// a location is challenging. Developers could consider printing out
@@ -94,10 +109,6 @@
"do Python -> C++ conversions which require the creation "
"of temporary values");
}
-
- if (frame->keep_alive.insert(h.ptr()).second) {
- Py_INCREF(h.ptr());
- }
}
};
diff --git a/tests/test_stl.cpp b/tests/test_stl.cpp
index 8bddbb1..b5d1979 100644
--- a/tests/test_stl.cpp
+++ b/tests/test_stl.cpp
@@ -582,6 +582,24 @@
[](const std::list<std::string> &) { return 2; });
m.def("func_with_string_or_vector_string_arg_overload", [](const std::string &) { return 3; });
+#ifdef PYBIND11_HAS_STRING_VIEW
+ m.def("func_with_string_views", [](const std::vector<std::string_view> &svs) {
+ py::list l;
+ for (std::string_view sv : svs) {
+ l.append(sv);
+ }
+ return l;
+ });
+ m.def("string_view_life_support_check",
+ [](const std::vector<std::string_view> &, int, const py::list &destroyed) {
+ return destroyed.size();
+ });
+ m.def("nested_string_view_life_support_check",
+ [](const std::vector<std::vector<std::string_view>> &, int, const py::list &destroyed) {
+ return destroyed.size();
+ });
+#endif
+
class Placeholder {
public:
Placeholder() { print_created(this); }
diff --git a/tests/test_stl.py b/tests/test_stl.py
index b04f55c..c75ccb8 100644
--- a/tests/test_stl.py
+++ b/tests/test_stl.py
@@ -1,5 +1,7 @@
from __future__ import annotations
+import weakref
+
import pytest
import env # noqa: F401
@@ -28,6 +30,132 @@
# Test regression caused by 936: pointers to stl containers weren't castable
assert m.cast_ptr_vector() == ["lvalue", "lvalue"]
+ if hasattr(m, "func_with_string_views"):
+
+ def gen():
+ return ("a" + str(x) for x in range(10000, 10010))
+
+ expected = list(gen())
+ assert m.func_with_string_views(gen()) == expected
+ assert m.func_with_string_views(x.encode() for x in gen()) == expected
+ assert (
+ m.func_with_string_views(bytearray(x.encode()) for x in gen()) == expected
+ )
+
+
+@pytest.mark.skipif(
+ not hasattr(m, "string_view_life_support_check"), reason="no <string_view>"
+)
+@pytest.mark.skipif("env.GRAALPY", reason="Cannot reliably trigger GC")
+def test_string_view_life_support_during_argument_conversion():
+ # Design background: PR #6096, "Why these lifetime tests are deliberately
+ # implementation-aware".
+ # This test uses conversion of a later argument as a checkpoint between
+ # loading the string views and entering the C++ function. Argument casters
+ # run left-to-right: after the first argument creates the views, the second
+ # argument's __index__ clears the list that owned their Python strings and
+ # forces GC. The C++ probe checks only whether those strings were destroyed,
+ # never the potentially dangling views. Zero during the call proves that
+ # loader life support worked; dead weakrefs afterward prove that it did not
+ # keep the strings alive too long.
+ destroyed = []
+
+ class TrackedString(str):
+ pass
+
+ source = [TrackedString("first"), TrackedString("second")]
+ weakrefs = [weakref.ref(item, lambda _: destroyed.append(None)) for item in source]
+
+ # Clear the only Python owners after the views have loaded, but before the
+ # bound function is called.
+ class ClearSourceOnIndex:
+ def __index__(self):
+ source.clear()
+ pytest.gc_collect()
+ return 0
+
+ assert (
+ m.string_view_life_support_check(source, ClearSourceOnIndex(), destroyed) == 0
+ )
+ assert source == []
+ pytest.gc_collect()
+ assert len(destroyed) == 2
+ assert all(ref() is None for ref in weakrefs)
+
+
+@pytest.mark.skipif(
+ not hasattr(m, "string_view_life_support_check"), reason="no <string_view>"
+)
+@pytest.mark.skipif("env.GRAALPY", reason="Cannot reliably trigger GC")
+@pytest.mark.parametrize(
+ ("element_type", "values"),
+ [
+ pytest.param(str, ("first", "second"), id="str"),
+ pytest.param(bytes, (b"first", b"second"), id="bytes"),
+ pytest.param(bytearray, (b"first", b"second"), id="bytearray"),
+ ],
+)
+def test_string_view_life_support_for_generator(element_type, values):
+ destroyed = []
+
+ class Tracked(element_type):
+ def __del__(self):
+ destroyed.append(None)
+
+ def source():
+ for value in values:
+ yield Tracked(value)
+
+ class CollectGarbageOnIndex:
+ def __index__(self):
+ pytest.gc_collect()
+ return 0
+
+ assert (
+ m.string_view_life_support_check(source(), CollectGarbageOnIndex(), destroyed)
+ == 0
+ )
+ pytest.gc_collect()
+ assert len(destroyed) == len(values)
+
+
+@pytest.mark.skipif(
+ not hasattr(m, "nested_string_view_life_support_check"),
+ reason="no <string_view>",
+)
+@pytest.mark.skipif("env.GRAALPY", reason="Cannot reliably trigger GC")
+def test_string_view_life_support_for_nested_containers():
+ # Design background: PR #6096, "Why these lifetime tests are deliberately
+ # implementation-aware". This uses the later-argument checkpoint described
+ # in test_string_view_life_support_during_argument_conversion. Here,
+ # clearing the outer list also releases the inner lists, verifying that life
+ # support reaches every Python string backing a view in the recursively
+ # converted std::vector<std::vector<std::string_view>>.
+ destroyed = []
+
+ class TrackedString(str):
+ def __del__(self):
+ destroyed.append(None)
+
+ source = [
+ [TrackedString("first"), TrackedString("second")],
+ [TrackedString("third"), TrackedString("fourth")],
+ ]
+
+ class ClearSourceOnIndex:
+ def __index__(self):
+ source.clear()
+ pytest.gc_collect()
+ return 0
+
+ assert (
+ m.nested_string_view_life_support_check(source, ClearSourceOnIndex(), destroyed)
+ == 0
+ )
+ assert source == []
+ pytest.gc_collect()
+ assert len(destroyed) == 4
+
def test_deque():
"""std::deque <-> list"""
diff --git a/tests/test_with_catch/test_interpreter.cpp b/tests/test_with_catch/test_interpreter.cpp
index 4103c0f..daa1041 100644
--- a/tests/test_with_catch/test_interpreter.cpp
+++ b/tests/test_with_catch/test_interpreter.cpp
@@ -509,3 +509,32 @@
py::initialize_interpreter();
}
+
+#ifdef PYBIND11_HAS_STRING_VIEW
+TEST_CASE("Casting to a string_view outside a bound function") {
+ // Regression for PR #6092: view casters add the source to loader_life_support, but
+ // outside a bound function there is no frame. The caller owns the source's lifetime
+ // here, so the cast must succeed rather than throw.
+ py::str unicode("hello");
+ py::bytes bytes_obj("world", 5);
+ auto bytearray_obj
+ = py::reinterpret_steal<py::object>(PyByteArray_FromStringAndSize("bytes", 5));
+
+ REQUIRE(py::cast<std::string_view>(unicode) == "hello");
+ REQUIRE(py::cast<std::string_view>(bytes_obj) == "world");
+ REQUIRE(py::cast<std::string_view>(bytearray_obj) == "bytes");
+
+ // Wide string views require an encoded temporary. With no loader life-support
+ // frame, returning a view into that temporary must fail.
+ REQUIRE_THROWS_AS(py::cast<std::u16string_view>(unicode), py::cast_error);
+ REQUIRE_THROWS_AS(py::cast<std::u32string_view>(unicode), py::cast_error);
+
+ // Bound-function dispatch provides a frame that keeps both temporaries alive.
+ auto accepts_wide_views
+ = py::cpp_function([](std::u16string_view value16, std::u32string_view value32) {
+ return value16 == std::u16string_view(u"hello")
+ && value32 == std::u32string_view(U"hello");
+ });
+ REQUIRE(accepts_wide_views(unicode, unicode).cast<bool>());
+}
+#endif
