pw_protobuf/encoding.rst

.. _module-pw_protobuf-encoding:

--------------------
pw_protobuf Encoding
--------------------

Usage
=====
Pigweed's protobuf encoders encode directly to the wire format of a proto rather
than staging information to a mutable datastructure. This means any writes of a
value are final, and can't be referenced or modified as a later step in the
encode process.

MemoryEncoder
-------------
A MemoryEncoder directly encodes a proto to an in-memory buffer.

.. Code:: cpp

  // Writes a proto response to the provided buffer, returning the encode
  // status and number of bytes written.
  StatusWithSize WriteProtoResponse(ByteSpan response) {
    // All proto writes are directly written to the `response` buffer.
    MemoryEncoder encoder(response);
    encoder.WriteUint32(kMagicNumberField, 0x1a1a2b2b);
    encoder.WriteString(kFavoriteFood, "cookies");
    return StatusWithSize(encoder.status(), encoder.size());
  }

StreamingEncoder
----------------
pw_protobuf's StreamingEncoder class operates on pw::stream::Writer objects to
serialized proto data. This means you can directly encode a proto to something
like pw::sys_io without needing to build the complete message in memory first.

.. Code:: cpp

  #include "pw_protobuf/streaming_encoder.h"
  #include "pw_stream/sys_io_stream.h"
  #include "pw_bytes/span.h"

  pw::stream::SysIoWriter sys_io_writer;
  pw::protobuf::StreamingEncoder my_proto_encoder(sys_io_writer,
                                                  pw::ByteSpan());

  // Once this line returns, the field has been written to the Writer.
  my_proto_encoder.WriteInt64(kTimestampFieldNumber, system::GetUnixEpoch());

  // There's no intermediate buffering when writing a string directly to a
  // StreamingEncoder.
  my_proto_encoder.WriteString(kWelcomeMessageFieldNumber,
                               "Welcome to Pigweed!");
  if (!my_proto_encoder.status().ok()) {
    PW_LOG_INFO("Failed to encode proto; %s", my_proto_encoder.status().str());
  }

Nested submessages
------------------
Writing proto messages with nested submessages requires buffering due to
limitations of the proto format. Every proto submessage must know the size of
the submessage before its final serialization can begin. A streaming encoder can
be passed a scratch buffer to use when constructing nested messages. All
submessage data is buffered to this scratch buffer until the submessage is
finalized. Note that the contents of this scratch buffer is not necessarily
valid proto data, so don't try to use it directly.

MemoryEncoder objects use the final destination buffer rather than relying on a
scratch buffer. Note that this means your destination buffer might need
additional space for overhead incurred by nesting submessages. The
``MaxScratchBufferSize()`` helper function can be useful in estimating how much
space to allocate to account for nested submessage encoding overhead.

.. Code:: cpp

  #include "pw_protobuf/streaming_encoder.h"
  #include "pw_stream/sys_io_stream.h"
  #include "pw_bytes/span.h"

  pw::stream::SysIoWriter sys_io_writer;
  // The scratch buffer should be at least as big as the largest nested
  // submessage. It's a good idea to be a little generous.
  std::byte submessage_scratch_buffer[64];

  // Provide the scratch buffer to the proto encoder. The buffer's lifetime must
  // match the lifetime of the encoder.
  pw::protobuf::StreamingEncoder my_proto_encoder(sys_io_writer,
                                                  submessage_scratch_buffer);

  StreamingEncoder nested_encoder =
      my_proto_encoder.GetNestedEncoder(kPetsFieldNumber);

  // There's no intermediate buffering when writing a string directly to a
  // StreamingEncoder.
  nested_encoder.WriteString(kNameFieldNumber, "Spot");
  nested_encoder.WriteString(kPetTypeFieldNumber, "dog");
  // Since this message is only nested one deep, the submessage is serialized to
  // the writer as soon as we finalize the submessage.
  PW_CHECK_OK(nested_encoder.Finalize());

  {  // If a nested_encoder is destroyed it will silently Finalize().
    StreamingEncoder nested_encoder_2 =
        my_proto_encoder.GetNestedEncoder(kPetsFieldNumber);
    nested_encoder_2.WriteString(kNameFieldNumber, "Slippers");
    nested_encoder_2.WriteString(kPetTypeFieldNumber, "rabbit");
  }  // When this scope ends, the nested encoder is serialized to the Writer.

  // If an encode error occurs when encoding the nested messages, it will be
  // reflected at the root encoder.
  if (!my_proto_encoder.status().ok()) {
    PW_LOG_INFO("Failed to encode proto; %s", my_proto_encoder.status().str());
  }

.. warning::
  When a nested submessage is created, any writes to the parent encoder that
  created the nested encoder will trigger a crash. To resume writing to
  a parent encoder, Finalize() the submessage encoder first.

Error Handling
--------------
While individual write calls on a proto encoder return pw::Status objects, the
encoder tracks all status returns and "latches" onto the first error
encountered. This status can be accessed via ``StreamingEncoder::status()``.

Codegen
-------
pw_protobuf encoder codegen integration is supported in GN, Bazel, and CMake.
The codegen is just a light wrapper around the ``StreamEncoder`` and
``MemoryEncoder`` objects, providing named helper functions to write proto
fields rather than requiring that field numbers are directly passed to an
encoder. Namespaced proto enums are also generated, and used as the arguments
when writing enum fields of a proto message.

All generated messages provide a ``Fields`` enum that can be used directly for
out-of-band encoding, or with the ``pw::protobuf::Decoder``.

This module's codegen is available through the ``*.pwpb`` sub-target of a
``pw_proto_library`` in GN, CMake, and Bazel. See :ref:`pw_protobuf_compiler's
documentation <module-pw_protobuf_compiler>` for more information on build
system integration for pw_protobuf codegen.

Example ``BUILD.gn``:

.. Code:: none

  import("//build_overrides/pigweed.gni")

  import("$dir_pw_build/target_types.gni")
  import("$dir_pw_protobuf_compiler/proto.gni")

  # This target controls where the *.pwpb.h headers end up on the include path.
  # In this example, it's at "pet_daycare_protos/client.pwpb.h".
  pw_proto_library("pet_daycare_protos") {
    sources = [
      "pet_daycare_protos/client.proto",
    ]
  }

  pw_source_set("example_client") {
    sources = [ "example_client.cc" ]
    deps = [
      ":pet_daycare_protos.pwpb",
      dir_pw_bytes,
      dir_pw_stream,
    ]
  }

Example ``pet_daycare_protos/client.proto``:

.. Code:: none

  syntax = "proto3";
  // The proto package controls the namespacing of the codegen. If this package
  // were fuzzy.friends, the namespace for codegen would be fuzzy::friends::*.
  package fuzzy_friends;

  message Pet {
    string name = 1;
    string pet_type = 2;
  }

  message Client {
    repeated Pet pets = 1;
  }

Example ``example_client.cc``:

.. Code:: cpp

  #include "pet_daycare_protos/client.pwpb.h"
  #include "pw_protobuf/streaming_encoder.h"
  #include "pw_stream/sys_io_stream.h"
  #include "pw_bytes/span.h"

  pw::stream::SysIoWriter sys_io_writer;
  std::byte submessage_scratch_buffer[64];
  // The constructor is the same as a pw::protobuf::StreamingEncoder.
  fuzzy_friends::Client::StreamEncoder client(sys_io_writer,
                                              submessage_scratch_buffer);

  fuzzy_friends::Pet::StreamEncoder pet1 = client.GetPetsEncoder();

  pet1.WriteName("Spot");
  pet1.WritePetType("dog");
  PW_CHECK_OK(pet1.Finalize());

  {  // Since pet2 is scoped, it will automatically Finalize() on destruction.
    fuzzy_friends::Pet::StreamEncoder pet2 = client.GetPetsEncoder();
    pet2.WriteName("Slippers");
    pet2.WritePetType("rabbit");
  }

  if (!client.status().ok()) {
    PW_LOG_INFO("Failed to encode proto; %s", client.status().str());
  }