pw_log_rpc/docs.rst - pigweed/pigweed - Git at Google

 .. _module-pw_log_rpc:

 ----------
 pw_log_rpc
 ----------
 An RPC-based logging backend for Pigweed.

 .. warning::
   This module is under construction and might change in the future.

 How to use
 ==========
 1. Set up RPC
 -------------
 Set up RPC for your target device. Basic deployments run RPC over a UART, with
 HDLC on top for framing. See :ref:`module-pw_rpc` for details on how to enable
 ``pw_rpc``.

 2. Set up tokenized logging (optional)
 --------------------------------------
 Set up the :ref:`module-pw_log_tokenized` log backend.

 3. Connect the tokenized logging handler to the MultiSink
 ---------------------------------------------------------
 Create a :ref:`MultiSink <module-pw_multisink>` instance to buffer log entries.
 Then, make the log backend handler,
 ``pw_tokenizer_HandleEncodedMessageWithPayload``, encode log entries in the
 ``log::LogEntry`` format, and add them to the ``MultiSink``.

 4. Create log drains
 --------------------
 Create an ``RpcLogDrainMap`` with one ``RpcLogDrain`` for each RPC channel used
 to stream logs. Provide this map to the ``LogService`` and register the latter
 with the application's RPC service. The ``RpcLogDrainMap`` provides a convenient
 way to access and maintain each ``RpcLogDrain``. Attach each ``RpcLogDrain`` to
 the ``MultiSink``.

 5. Flush the log drains in the background
 -----------------------------------------
 Depending on the product's requirements, create a thread to flush all
 ``RpcLogDrain``\s or one thread per drain. The thread(s) must continuously call
 ``RpcLogDrain::Flush()`` to pull entries from the ``MultiSink`` and send them to
 the log listeners.

 Logging over RPC diagrams
 =========================

 Sample RPC logs request
 -----------------------
 The log listener, e.g. a computer, requests logs via RPC. The log service
 receives the request and sets up the corresponding ``RpcLogDrain`` to start the
 log stream.

 .. mermaid::

   graph TD
     computer[Computer]-->pw_rpc;
     pw_rpc-->log_service[LogService];
     log_service-->rpc_log_drain_pc[RpcLogDrain<br>streams to<br>computer];;

 Sample logging over RPC
 ------------------------
 Logs are streamed via RPC to a computer, and to another log listener. There can
 also be internal log readers, i.e. ``MultiSink::Drain``\s, attached to the
 ``MultiSink``, such as a writer to persistent memory, for example.

 .. mermaid::

   graph TD
     source1[Source 1]-->log_api[pw_log API];
     source2[Source 2]-->log_api;
     log_api-->log_backend[Log backend];
     log_backend-->multisink[MultiSink];
     multisink-->drain[MultiSink::Drain];
     multisink-->rpc_log_drain_pc[RpcLogDrain<br>streams to<br>computer];
     multisink-->rpc_log_drain_other[RpcLogDrain<br>streams to<br>other log listener];
     drain-->other_consumer[Other log consumer<br>e.g. persistent memory];
     rpc_log_drain_pc-->pw_rpc;
     rpc_log_drain_other-->pw_rpc;
     pw_rpc-->computer[Computer];
     pw_rpc-->other_listener[Other log<br>listener];

 RPC log service
 ===============
 The ``LogService`` class is an RPC service that provides a way to request a log
 stream sent via RPC. Thus, it helps avoid using a different protocol for logs
 and RPCs over the same interface(s). It requires a map of ``RpcLogDrains`` to
 assign stream writers and delegate the log stream flushing to the user's
 preferred method.

 RpcLogDrain
 ===========
 An ``RpcLogDrain`` reads from the ``MultiSink`` instance that buffers logs, then
 packs, and sends the retrieved log entries to the log listener. One
 ``RpcLogDrain`` is needed for each log listener. An ``RpcLogDrain`` needs a
 thread to continuously call ``Flush()`` to maintain the log stream. A thread can
 maintain multiple log streams, but it must not be the same thread used by the
 RPC server, to avoid blocking it.

 Each ``RpcLogDrain`` is identified by a known RPC channel ID and requires a
 ``rpc::RawServerWriter`` to write the packed multiple log entries. This writer
 is assigned by the ``LogService::Listen`` RPC. Future work will allow
 ``RpcLogDrain``\s to have an open RPC writer, to constantly stream logs without
 the need to request them. This is useful in cases where the connection to the
 client is dropped silently because the log stream can continue when reconnected
 without the client requesting it.

 An ``RpcLogDrain`` must be attached to a ``MultiSink`` containing multiple
 ``log::LogEntry``\s. When ``Flush`` is called, the drain acquires the
 ``rpc::RawServerWriter`` 's write buffer, grabs one ``log::LogEntry`` from the
 multisink, encodes it into a ``log::LogEntries`` stream, and repeats the process
 until the write buffer is full. Then the drain calls
 ``rpc::RawServerWriter::Write`` to flush the write buffer and repeats the
 process until all the entries in the ``MultiSink`` are read or an error is
 found.

 The user must provide a buffer large enough for the largest entry in the
 ``MultiSink`` while also accounting for the interface's Maximum Transmission
 Unit (MTU). If the ``RpcLogDrain`` finds a drop message count as it reads the
 ``MultiSink`` it will insert a message in the stream with the drop message
 count.

 RpcLogDrainMap
 ==============
 Provides a convenient way to access all or a single ``RpcLogDrain`` by its RPC
 channel ID.

 RpcLogDrainThread
 =================
 The module includes a sample thread that flushes each drain sequentially. Future
 work might replace this with enqueueing the flush work on a work queue. The user
 can also choose to have different threads flushing individual ``RpcLogDrain``\s
 with different priorities.

 Logging example
 ===============
 The following code shows a sample setup to defer the log handling to the
 ``RpcLogDrainThread`` to avoid having the log streaming block at the log
 callsite.

 main.cc
 -------
 .. code-block:: cpp

   #include "foo/foo_log.h"
   #include "pw_log/log.h"
   #include "pw_thread/detached_thread.h"
   #include "pw_thread_stl/options.h"

   namespace {

   void RegisterServices() {
     pw::rpc::system_server::Server().RegisterService(foo_log::log_service);
   }
   }  // namespace

   int main() {
     PW_LOG_INFO("Deferred logging over RPC example");
     pw::rpc::system_server::Init();
     RegisterServices();
     pw::thread::DetachedThread(pw::thread::stl::Options(), foo_log::log_thread);
     pw::rpc::system_server::Start();
     return 0;
   }

 foo_log.cc
 ----------
 Example of a log backend implementation, where logs enter the ``MultiSink`` and
 log drains are set up.

 .. code-block:: cpp

   #include "foo/foo_log.h"

   #include <array>
   #include <cstdint>

   #include "pw_chrono/system_clock.h"
   #include "pw_log/proto_utils.h"
   #include "pw_log_rpc/log_service.h"
   #include "pw_log_rpc/rpc_log_drain.h"
   #include "pw_log_rpc/rpc_log_drain_map.h"
   #include "pw_log_rpc/rpc_log_drain_thread.h"
   #include "pw_rpc_system_server/rpc_server.h"
   #include "pw_sync/interrupt_spin_lock.h"
   #include "pw_sync/lock_annotations.h"
   #include "pw_sync/mutex.h"
   #include "pw_tokenizer/tokenize_to_global_handler_with_payload.h"

   namespace foo_log {
   namespace {
   constexpr size_t kLogBufferSize = 5000;
   // Tokenized logs are typically 12-24 bytes.
   constexpr size_t kMaxMessageSize = 32;
   // kMaxLogEntrySize should be less than the MTU of the RPC channel output used
   // by the provided server writer.
   constexpr size_t kMaxLogEntrySize =
       pw::log_rpc::RpcLogDrain::kMinEntrySizeWithoutPayload + kMaxMessageSize;
   std::array<std::byte, kLogBufferSize> multisink_buffer;

   // To save RAM, share the mutex, since drains will be managed sequentially.
   pw::sync::Mutex shared_mutex;
   std::array<std::byte, kMaxEntrySize> client1_buffer
       PW_GUARDED_BY(shared_mutex);
   std::array<std::byte, kMaxEntrySize> client2_buffer
       PW_GUARDED_BY(shared_mutex);
   std::array<pw::log_rpc::RpcLogDrain, 2> drains = {
       pw::log_rpc::RpcLogDrain(
           1,
           client1_buffer,
           shared_mutex,
           RpcLogDrain::LogDrainErrorHandling::kIgnoreWriterErrors),
       pw::log_rpc::RpcLogDrain(
           2,
           client2_buffer,
           shared_mutex,
           RpcLogDrain::LogDrainErrorHandling::kIgnoreWriterErrors),
   };

   pw::sync::InterruptSpinLock log_encode_lock;
   std::array<std::byte, kMaxLogEntrySize> log_encode_buffer
       PW_GUARDED_BY(log_encode_lock);

   extern "C" void pw_tokenizer_HandleEncodedMessageWithPayload(
       pw_tokenizer_Payload metadata, const uint8_t message[], size_t size_bytes) {
     int64_t timestamp =
         pw::chrono::SystemClock::now().time_since_epoch().count();
     std::lock_guard lock(log_encode_lock);
     pw::Result<pw::ConstByteSpan> encoded_log_result =
       pw::log::EncodeTokenizedLog(
           metadata, message, size_bytes, timestamp, log_encode_buffer);

     if (!encoded_log_result.ok()) {
       GetMultiSink().HandleDropped();
       return;
     }
     GetMultiSink().HandleEntry(encoded_log_result.value());
   }
   }  // namespace

   pw::log_rpc::RpcLogDrainMap drain_map(drains);
   pw::log_rpc::RpcLogDrainThread log_thread(GetMultiSink(), drain_map);
   pw::log_rpc::LogService log_service(drain_map);

   pw::multisink::MultiSink& GetMultiSink() {
     static pw::multisink::MultiSink multisink(multisink_buffer);
     return multisink;
   }
   }  // namespace foo_log

 Logging in other source files
 -----------------------------
 To defer logging, other source files must simply include ``pw_log/log.h`` and
 use the :ref:`module-pw_log` APIs, as long as the source set that includes
 ``foo_log.cc`` is setup as the log backend.
	.. _module-pw_log_rpc:

	----------
	pw_log_rpc
	----------
	An RPC-based logging backend for Pigweed.

	.. warning::
	This module is under construction and might change in the future.

	How to use
	==========
	1. Set up RPC
	-------------
	Set up RPC for your target device. Basic deployments run RPC over a UART, with
	HDLC on top for framing. See :ref:`module-pw_rpc` for details on how to enable
	``pw_rpc``.

	2. Set up tokenized logging (optional)
	--------------------------------------
	Set up the :ref:`module-pw_log_tokenized` log backend.

	3. Connect the tokenized logging handler to the MultiSink
	---------------------------------------------------------
	Create a :ref:`MultiSink <module-pw_multisink>` instance to buffer log entries.
	Then, make the log backend handler,
	``pw_tokenizer_HandleEncodedMessageWithPayload``, encode log entries in the
	``log::LogEntry`` format, and add them to the ``MultiSink``.

	4. Create log drains
	--------------------
	Create an ``RpcLogDrainMap`` with one ``RpcLogDrain`` for each RPC channel used
	to stream logs. Provide this map to the ``LogService`` and register the latter
	with the application's RPC service. The ``RpcLogDrainMap`` provides a convenient
	way to access and maintain each ``RpcLogDrain``. Attach each ``RpcLogDrain`` to
	the ``MultiSink``.

	5. Flush the log drains in the background
	-----------------------------------------
	Depending on the product's requirements, create a thread to flush all
	``RpcLogDrain``\s or one thread per drain. The thread(s) must continuously call
	``RpcLogDrain::Flush()`` to pull entries from the ``MultiSink`` and send them to
	the log listeners.

	Logging over RPC diagrams
	=========================

	Sample RPC logs request
	-----------------------
	The log listener, e.g. a computer, requests logs via RPC. The log service
	receives the request and sets up the corresponding ``RpcLogDrain`` to start the
	log stream.

	.. mermaid::

	graph TD
	computer[Computer]-->pw_rpc;
	pw_rpc-->log_service[LogService];
	log_service-->rpc_log_drain_pc[RpcLogDrain<br>streams to<br>computer];;

	Sample logging over RPC
	------------------------
	Logs are streamed via RPC to a computer, and to another log listener. There can
	also be internal log readers, i.e. ``MultiSink::Drain``\s, attached to the
	``MultiSink``, such as a writer to persistent memory, for example.

	.. mermaid::

	graph TD
	source1[Source 1]-->log_api[pw_log API];
	source2[Source 2]-->log_api;
	log_api-->log_backend[Log backend];
	log_backend-->multisink[MultiSink];
	multisink-->drain[MultiSink::Drain];
	multisink-->rpc_log_drain_pc[RpcLogDrain<br>streams to<br>computer];
	multisink-->rpc_log_drain_other[RpcLogDrain<br>streams to<br>other log listener];
	drain-->other_consumer[Other log consumer<br>e.g. persistent memory];
	rpc_log_drain_pc-->pw_rpc;
	rpc_log_drain_other-->pw_rpc;
	pw_rpc-->computer[Computer];
	pw_rpc-->other_listener[Other log<br>listener];

	RPC log service
	===============
	The ``LogService`` class is an RPC service that provides a way to request a log
	stream sent via RPC. Thus, it helps avoid using a different protocol for logs
	and RPCs over the same interface(s). It requires a map of ``RpcLogDrains`` to
	assign stream writers and delegate the log stream flushing to the user's
	preferred method.

	RpcLogDrain
	===========
	An ``RpcLogDrain`` reads from the ``MultiSink`` instance that buffers logs, then
	packs, and sends the retrieved log entries to the log listener. One
	``RpcLogDrain`` is needed for each log listener. An ``RpcLogDrain`` needs a
	thread to continuously call ``Flush()`` to maintain the log stream. A thread can
	maintain multiple log streams, but it must not be the same thread used by the
	RPC server, to avoid blocking it.

	Each ``RpcLogDrain`` is identified by a known RPC channel ID and requires a
	``rpc::RawServerWriter`` to write the packed multiple log entries. This writer
	is assigned by the ``LogService::Listen`` RPC. Future work will allow
	``RpcLogDrain``\s to have an open RPC writer, to constantly stream logs without
	the need to request them. This is useful in cases where the connection to the
	client is dropped silently because the log stream can continue when reconnected
	without the client requesting it.

	An ``RpcLogDrain`` must be attached to a ``MultiSink`` containing multiple
	``log::LogEntry``\s. When ``Flush`` is called, the drain acquires the
	``rpc::RawServerWriter`` 's write buffer, grabs one ``log::LogEntry`` from the
	multisink, encodes it into a ``log::LogEntries`` stream, and repeats the process
	until the write buffer is full. Then the drain calls
	``rpc::RawServerWriter::Write`` to flush the write buffer and repeats the
	process until all the entries in the ``MultiSink`` are read or an error is
	found.

	The user must provide a buffer large enough for the largest entry in the
	``MultiSink`` while also accounting for the interface's Maximum Transmission
	Unit (MTU). If the ``RpcLogDrain`` finds a drop message count as it reads the
	``MultiSink`` it will insert a message in the stream with the drop message
	count.

	RpcLogDrainMap
	==============
	Provides a convenient way to access all or a single ``RpcLogDrain`` by its RPC
	channel ID.

	RpcLogDrainThread
	=================
	The module includes a sample thread that flushes each drain sequentially. Future
	work might replace this with enqueueing the flush work on a work queue. The user
	can also choose to have different threads flushing individual ``RpcLogDrain``\s
	with different priorities.

	Logging example
	===============
	The following code shows a sample setup to defer the log handling to the
	``RpcLogDrainThread`` to avoid having the log streaming block at the log
	callsite.

	main.cc
	-------
	.. code-block:: cpp

	#include "foo/foo_log.h"
	#include "pw_log/log.h"
	#include "pw_thread/detached_thread.h"
	#include "pw_thread_stl/options.h"

	namespace {

	void RegisterServices() {
	pw::rpc::system_server::Server().RegisterService(foo_log::log_service);
	}
	} // namespace

	int main() {
	PW_LOG_INFO("Deferred logging over RPC example");
	pw::rpc::system_server::Init();
	RegisterServices();
	pw::thread::DetachedThread(pw::thread::stl::Options(), foo_log::log_thread);
	pw::rpc::system_server::Start();
	return 0;
	}

	foo_log.cc
	----------
	Example of a log backend implementation, where logs enter the ``MultiSink`` and
	log drains are set up.

	.. code-block:: cpp

	#include "foo/foo_log.h"

	#include <array>
	#include <cstdint>

	#include "pw_chrono/system_clock.h"
	#include "pw_log/proto_utils.h"
	#include "pw_log_rpc/log_service.h"
	#include "pw_log_rpc/rpc_log_drain.h"
	#include "pw_log_rpc/rpc_log_drain_map.h"
	#include "pw_log_rpc/rpc_log_drain_thread.h"
	#include "pw_rpc_system_server/rpc_server.h"
	#include "pw_sync/interrupt_spin_lock.h"
	#include "pw_sync/lock_annotations.h"
	#include "pw_sync/mutex.h"
	#include "pw_tokenizer/tokenize_to_global_handler_with_payload.h"

	namespace foo_log {
	namespace {
	constexpr size_t kLogBufferSize = 5000;
	// Tokenized logs are typically 12-24 bytes.
	constexpr size_t kMaxMessageSize = 32;
	// kMaxLogEntrySize should be less than the MTU of the RPC channel output used
	// by the provided server writer.
	constexpr size_t kMaxLogEntrySize =
	pw::log_rpc::RpcLogDrain::kMinEntrySizeWithoutPayload + kMaxMessageSize;
	std::array<std::byte, kLogBufferSize> multisink_buffer;

	// To save RAM, share the mutex, since drains will be managed sequentially.
	pw::sync::Mutex shared_mutex;
	std::array<std::byte, kMaxEntrySize> client1_buffer
	PW_GUARDED_BY(shared_mutex);
	std::array<std::byte, kMaxEntrySize> client2_buffer
	PW_GUARDED_BY(shared_mutex);
	std::array<pw::log_rpc::RpcLogDrain, 2> drains = {
	pw::log_rpc::RpcLogDrain(
	1,
	client1_buffer,
	shared_mutex,
	RpcLogDrain::LogDrainErrorHandling::kIgnoreWriterErrors),
	pw::log_rpc::RpcLogDrain(
	2,
	client2_buffer,
	shared_mutex,
	RpcLogDrain::LogDrainErrorHandling::kIgnoreWriterErrors),
	};

	pw::sync::InterruptSpinLock log_encode_lock;
	std::array<std::byte, kMaxLogEntrySize> log_encode_buffer
	PW_GUARDED_BY(log_encode_lock);

	extern "C" void pw_tokenizer_HandleEncodedMessageWithPayload(
	pw_tokenizer_Payload metadata, const uint8_t message[], size_t size_bytes) {
	int64_t timestamp =
	pw::chrono::SystemClock::now().time_since_epoch().count();
	std::lock_guard lock(log_encode_lock);
	pw::Result<pw::ConstByteSpan> encoded_log_result =
	pw::log::EncodeTokenizedLog(
	metadata, message, size_bytes, timestamp, log_encode_buffer);

	if (!encoded_log_result.ok()) {
	GetMultiSink().HandleDropped();
	return;
	}
	GetMultiSink().HandleEntry(encoded_log_result.value());
	}
	} // namespace

	pw::log_rpc::RpcLogDrainMap drain_map(drains);
	pw::log_rpc::RpcLogDrainThread log_thread(GetMultiSink(), drain_map);
	pw::log_rpc::LogService log_service(drain_map);

	pw::multisink::MultiSink& GetMultiSink() {
	static pw::multisink::MultiSink multisink(multisink_buffer);
	return multisink;
	}
	} // namespace foo_log

	Logging in other source files
	-----------------------------
	To defer logging, other source files must simply include ``pw_log/log.h`` and
	use the :ref:`module-pw_log` APIs, as long as the source set that includes
	``foo_log.cc`` is setup as the log backend.