Convert documentation to markdown format (#587)
Includes general documentation cleanup and updates, and also added a feedback form.
Squash merge from dev_markdown_docs branch.
diff --git a/docs/Makefile b/docs/Makefile
index 18282b6..697aa32 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -1,5 +1,23 @@
-all: index.html concepts.html reference.html security.html migration.html whats_new.html
+INPUTS = index.md concepts.md reference.md security.md migration.md whats_new.md
-%.html: %.rst
- rst2html --field-name-limit=32 --stylesheet=lsr.css --link-stylesheet $< $@
- sed -i 's!</head>!<link href="favicon.ico" type="image/x-icon" rel="shortcut icon" />\n</head>!' $@
+all: $(INPUTS:.md=.html)
+
+tmp_menu.html: $(INPUTS)
+ echo '<div id="index">' > $@
+ (echo '<h2>Documentation index</h2>'; \
+ for file in $^; do echo -n '1. ['; sed -n '1 s!^# Nanopb: !! p' $$file; \
+ echo -n "]("; echo $$file | sed 's/.md/.html)/' ; done;) | \
+ pandoc -f markdown -t html5 >> $@
+ echo '</div>' >> $@
+
+%.html: %.md tmp_menu.html
+ sed '1 s!#!%!' $< | \
+ pandoc -s -f markdown -t html5 -c lsr.css --toc --toc-depth=4 \
+ --variable 'header-includes=<link href="favicon.ico" type="image/x-icon" rel="shortcut icon" />' \
+ --indented-code-classes=c \
+ -o $@
+ sed -i '/<nav/e cat feedback.html' $@
+ sed -i 's/doc_page_name_placeholder/$</' $@
+ sed -i 's!<nav[^>]*>!\0<b>Contents:</b>!' $@
+ sed -i '/<nav/e cat tmp_menu.html' $@
+
diff --git a/docs/concepts.md b/docs/concepts.md
new file mode 100644
index 0000000..74e1ca3
--- /dev/null
+++ b/docs/concepts.md
@@ -0,0 +1,579 @@
+# Nanopb: Basic concepts
+
+The things outlined here are the underlying concepts of the nanopb
+design.
+
+## Proto files
+
+All Protocol Buffers implementations use .proto files to describe the
+message format. The point of these files is to be a portable interface
+description language.
+
+### Compiling .proto files for nanopb
+
+Nanopb comes with a Python script to generate `.pb.c` and
+`.pb.h` files from the `.proto` definition:
+
+ user@host:~$ nanopb/generator/nanopb_generator.py message.proto
+ Writing to message.pb.h and message.pb.c
+
+Internally this script uses Google `protoc` to parse the
+input file. If you do not have it available, you may receive an error
+message. You can install either `grpcio-tools` Python
+package using `pip`, or the `protoc` compiler
+itself from `protobuf-compiler` distribution package.
+Generally the Python package is recommended, because nanopb requires
+protoc version 3.6 or newer to support all features, and some distributions come with an older
+version.
+
+### Modifying generator behaviour
+
+Using generator options, you can set maximum sizes for fields in order
+to allocate them statically. The preferred way to do this is to create
+an .options file with the same name as your .proto file:
+
+ # Foo.proto
+ message Foo {
+ required string name = 1;
+ }
+
+ # Foo.options
+ Foo.name max_size:16
+
+For more information on this, see the [Proto file
+options](reference.html#proto-file-options) section in the reference
+manual.
+
+## Streams
+
+Nanopb uses streams for accessing the data in encoded format. The stream
+abstraction is very lightweight, and consists of a structure
+(`pb_ostream_t` or `pb_istream_t`) which contains a pointer to a
+callback function.
+
+There are a few generic rules for callback functions:
+
+1) Return false on IO errors. The encoding or decoding process will
+ abort immediately.
+2) Use state to store your own data, such as a file descriptor.
+3) `bytes_written` and `bytes_left` are updated by pb_write and
+ pb_read.
+4) Your callback may be used with substreams. In this case
+ `bytes_left`, `bytes_written` and `max_size` have smaller values
+ than the original stream. Don't use these values to calculate
+ pointers.
+5) Always read or write the full requested length of data. For example,
+ POSIX `recv()` needs the `MSG_WAITALL` parameter to accomplish
+ this.
+
+### Output streams
+
+ struct _pb_ostream_t
+ {
+ bool (*callback)(pb_ostream_t *stream, const uint8_t *buf, size_t count);
+ void *state;
+ size_t max_size;
+ size_t bytes_written;
+ };
+
+The `callback` for output stream may be NULL, in which case the stream
+simply counts the number of bytes written. In this case, `max_size` is
+ignored.
+
+Otherwise, if `bytes_written` + bytes_to_be_written is larger than
+`max_size`, pb_write returns false before doing anything else. If you
+don\'t want to limit the size of the stream, pass SIZE_MAX.
+
+**Example 1:**
+
+This is the way to get the size of the message without storing it
+anywhere:
+
+ Person myperson = ...;
+ pb_ostream_t sizestream = {0};
+ pb_encode(&sizestream, Person_fields, &myperson);
+ printf("Encoded size is %d\n", sizestream.bytes_written);
+
+**Example 2:**
+
+Writing to stdout:
+
+ bool callback(pb_ostream_t `stream, const uint8_t `buf, size_t count)
+ {
+ FILE *file = (FILE*) stream->state;
+ return fwrite(buf, 1, count, file) == count;
+ }
+
+ pb_ostream_t stdoutstream = {&callback, stdout, SIZE_MAX, 0};
+
+### Input streams
+
+For input streams, there is one extra rule:
+
+6) You don't need to know the length of the message in advance. After
+ getting EOF error when reading, set `bytes_left` to 0 and return
+ `false`. `pb_decode()` will detect this and if the EOF was in a proper
+ position, it will return true.
+
+Here is the structure:
+
+ struct _pb_istream_t
+ {
+ bool (*callback)(pb_istream_t *stream, uint8_t *buf, size_t count);
+ void *state;
+ size_t bytes_left;
+ };
+
+The `callback` must always be a function pointer. `Bytes_left` is an
+upper limit on the number of bytes that will be read. You can use
+SIZE_MAX if your callback handles EOF as described above.
+
+**Example:**
+
+This function binds an input stream to stdin:
+
+ bool callback(pb_istream_t *stream, uint8_t *buf, size_t count)
+ {
+ FILE *file = (FILE*)stream->state;
+ bool status;
+
+ if (buf == NULL)
+ {
+ while (count-- && fgetc(file) != EOF);
+ return count == 0;
+ }
+
+ status = (fread(buf, 1, count, file) == count);
+
+ if (feof(file))
+ stream->bytes_left = 0;
+
+ return status;
+ }
+
+ pb_istream_t stdinstream = {&callback, stdin, SIZE_MAX};
+
+## Data types
+
+Most Protocol Buffers datatypes have directly corresponding C datatypes,
+such as `int32` is `int32_t`, `float` is `float` and `bool` is `bool`. However, the
+variable-length datatypes are more complex:
+
+1) Strings, bytes and repeated fields of any type map to callback
+ functions by default.
+2) If there is a special option `(nanopb).max_size` specified in the
+ .proto file, string maps to null-terminated char array and bytes map
+ to a structure containing a char array and a size field.
+3) If `(nanopb).fixed_length` is set to `true` and
+ `(nanopb).max_size` is also set, then bytes map to an inline byte
+ array of fixed size.
+4) If there is a special option `(nanopb).max_count` specified on a
+ repeated field, it maps to an array of whatever type is being
+ repeated. Another field will be created for the actual number of
+ entries stored.
+5) If `(nanopb).fixed_count` is set to `true` and
+ `(nanopb).max_count` is also set, the field for the actual number
+ of entries will not by created as the count is always assumed to be
+ max count.
+
+### Examples of .proto specifications vs. generated structure
+
+**Simple integer field:**\
+.proto: `int32 age = 1;`\
+.pb.h: `int32_t age;`
+
+**String with unknown length:**\
+.proto: `string name = 1;`\
+.pb.h: `pb_callback_t name;`
+
+**String with known maximum length:**\
+.proto: `string name = 1 [(nanopb).max_length = 40];`\
+.pb.h: `char name[41];`
+
+**Repeated string with unknown count:**\
+.proto: `repeated string names = 1;`\
+.pb.h: `pb_callback_t names;`
+
+**Repeated string with known maximum count and size:**\
+.proto: `repeated string names = 1 [(nanopb).max_length = 40, (nanopb).max_count = 5];`\
+.pb.h: `size_t names_count;` `char names[5][41];`
+
+**Bytes field with known maximum size:**\
+.proto: `bytes data = 1 [(nanopb).max_size = 16];`\
+.pb.h: `PB_BYTES_ARRAY_T(16) data;`, where the struct contains `{pb_size_t size; pb_byte_t bytes[n];}`
+
+**Bytes field with fixed length:**\
+.proto: `bytes data = 1 [(nanopb).max_size = 16, (nanopb).fixed_length = true];`\
+.pb.h: `pb_byte_t data[16];`
+
+**Repeated integer array with known maximum size:**\
+.proto: `repeated int32 numbers = 1 [(nanopb).max_count = 5];`\
+.pb.h: `pb_size_t numbers_count;` `int32_t numbers[5];`
+
+**Repeated integer array with fixed count:**\
+.proto: `repeated int32 numbers = 1 [(nanopb).max_count = 5, (nanopb).fixed_count = true];`\
+.pb.h: `int32_t numbers[5];`
+
+The maximum lengths are checked in runtime. If string/bytes/array
+exceeds the allocated length, `pb_decode()` will return false.
+
+> **Note:** For the `bytes` datatype, the field length checking may not be
+exact. The compiler may add some padding to the `pb_bytes_t`
+structure, and the nanopb runtime doesn't know how much of the
+structure size is padding. Therefore it uses the whole length of the
+structure for storing data, which is not very smart but shouldn't cause
+problems. In practise, this means that if you specify
+`(nanopb).max_size=5` on a `bytes` field, you may be able to store 6
+bytes there. For the `string` field type, the length limit is exact.
+
+> **Note:** The decoder only keeps track of one `fixed_count` repeated field at a time. Usually this it not an issue because all elements of a repeated field occur end-to-end. Interleaved array elements of several `fixed_count` repeated fields would be a valid protobuf message, but would get rejected by nanopb decoder with error `"wrong size for fixed count field"`.
+
+## Field callbacks
+
+When a field has dynamic length, nanopb cannot statically allocate
+storage for it. Instead, it allows you to handle the field in whatever
+way you want, using a callback function.
+
+The [pb_callback_t](reference.html#pb-callback-t) structure contains a
+function pointer and a `void` pointer called `arg` you can use for
+passing data to the callback. If the function pointer is NULL, the field
+will be skipped. A pointer to the `arg` is passed to the function, so
+that it can modify it and retrieve the value.
+
+The actual behavior of the callback function is different in encoding
+and decoding modes. In encoding mode, the callback is called once and
+should write out everything, including field tags. In decoding mode, the
+callback is called repeatedly for every data item.
+
+To write more complex field callbacks, it is recommended to read the
+[Google Protobuf Encoding Specification](https://developers.google.com/protocol-buffers/docs/encoding).
+
+### Encoding callbacks
+
+ bool (*encode)(pb_ostream_t *stream, const pb_field_iter_t *field, void * const *arg);
+
+| | |
+| ---------- | ------------------------------------------------------------------ |
+| `stream` | Output stream to write to |
+| `field` | Iterator for the field currently being encoded or decoded. |
+| `arg` | Pointer to the `arg` field in the `pb_callback_t` structure. |
+
+When encoding, the callback should write out complete fields, including
+the wire type and field number tag. It can write as many or as few
+fields as it likes. For example, if you want to write out an array as
+`repeated` field, you should do it all in a single call.
+
+Usually you can use [pb_encode_tag_for_field](reference.html#pb-encode-tag-for-field) to
+encode the wire type and tag number of the field. However, if you want
+to encode a repeated field as a packed array, you must call
+[pb_encode_tag](reference.html#pb-encode-tag) instead to specify a
+wire type of `PB_WT_STRING`.
+
+If the callback is used in a submessage, it will be called multiple
+times during a single call to [pb_encode](reference.html#pb-encode). In
+this case, it must produce the same amount of data every time. If the
+callback is directly in the main message, it is called only once.
+
+This callback writes out a dynamically sized string:
+
+ bool write_string(pb_ostream_t *stream, const pb_field_iter_t *field, void * const *arg)
+ {
+ char *str = get_string_from_somewhere();
+ if (!pb_encode_tag_for_field(stream, field))
+ return false;
+
+ return pb_encode_string(stream, (uint8_t*)str, strlen(str));
+ }
+
+### Decoding callbacks
+
+ bool (*decode)(pb_istream_t *stream, const pb_field_iter_t *field, void **arg);
+
+| | |
+| ---------- | ------------------------------------------------------------------ |
+| `stream` | Input stream to read from |
+| `field` | Iterator for the field currently being encoded or decoded. |
+| `arg` | Pointer to the `arg` field in the `pb_callback_t` structure. |
+
+When decoding, the callback receives a length-limited substring that
+reads the contents of a single field. The field tag has already been
+read. For `string` and `bytes`, the length value has already been
+parsed, and is available at `stream->bytes_left`.
+
+The callback will be called multiple times for repeated fields. For
+packed fields, you can either read multiple values until the stream
+ends, or leave it to [pb_decode](reference.html#pb-decode) to call your
+function over and over until all values have been read.
+
+This callback reads multiple integers and prints them:
+
+ bool read_ints(pb_istream_t *stream, const pb_field_iter_t *field, void **arg)
+ {
+ while (stream->bytes_left)
+ {
+ uint64_t value;
+ if (!pb_decode_varint(stream, &value))
+ return false;
+ printf("%lld\n", value);
+ }
+ return true;
+ }
+
+### Function name bound callbacks
+
+ bool MyMessage_callback(pb_istream_t *istream, pb_ostream_t *ostream, const pb_field_iter_t *field);
+
+| | |
+| ---------- | ------------------------------------------------------------------ |
+| `istream` | Input stream to read from, or NULL if called in encoding context. |
+| `ostream` | Output stream to write to, or NULL if called in decoding context. |
+| `field` | Iterator for the field currently being encoded or decoded. |
+
+Storing function pointer in `pb_callback_t` fields inside
+the message requires extra storage space and is often cumbersome. As an
+alternative, the generator options `callback_function` and
+`callback_datatype` can be used to bind a callback function
+based on its name.
+
+Typically this feature is used by setting
+`callback_datatype` to e.g. `void\*` or other
+data type used for callback state. Then the generator will automatically
+set `callback_function` to
+`MessageName_callback` and produce a prototype for it in
+generated `.pb.h`. By implementing this function in your own
+code, you will receive callbacks for fields without having to separately
+set function pointers.
+
+If you want to use function name bound callbacks for some fields and
+`pb_callback_t` for other fields, you can call
+`pb_default_field_callback` from the message-level
+callback. It will then read a function pointer from
+`pb_callback_t` and call it.
+
+## Message descriptor
+
+For using the `pb_encode()` and `pb_decode()` functions, you need a
+description of all the fields contained in a message. This description
+is usually autogenerated from .proto file.
+
+For example this submessage in the Person.proto file:
+
+~~~~ protobuf
+message Person {
+ message PhoneNumber {
+ required string number = 1 [(nanopb).max_size = 40];
+ optional PhoneType type = 2 [default = HOME];
+ }
+}
+~~~~
+
+This in turn generates a macro list in the `.pb.h` file:
+
+ #define Person_PhoneNumber_FIELDLIST(X, a) \
+ X(a, STATIC, REQUIRED, STRING, number, 1) \
+ X(a, STATIC, OPTIONAL, UENUM, type, 2)
+
+Inside the `.pb.c` file there is a macro call to
+`PB_BIND`:
+
+ PB_BIND(Person_PhoneNumber, Person_PhoneNumber, AUTO)
+
+These macros will in combination generate `pb_msgdesc_t`
+structure and associated lists:
+
+ const uint32_t Person_PhoneNumber_field_info[] = { ... };
+ const pb_msgdesc_t * const Person_PhoneNumber_submsg_info[] = { ... };
+ const pb_msgdesc_t Person_PhoneNumber_msg = {
+ 2,
+ Person_PhoneNumber_field_info,
+ Person_PhoneNumber_submsg_info,
+ Person_PhoneNumber_DEFAULT,
+ NULL,
+ };
+
+The encoding and decoding functions take a pointer to this structure and
+use it to process each field in the message.
+
+## Oneof
+
+Protocol Buffers supports
+[oneof](https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#oneof_and_oneof_field)
+sections, where only one of the fields contained within can be present. Here is an example of `oneof` usage:
+
+~~~~ protobuf
+message MsgType1 {
+ required int32 value = 1;
+}
+
+message MsgType2 {
+ required bool value = 1;
+}
+
+message MsgType3 {
+ required int32 value1 = 1;
+ required int32 value2 = 2;
+}
+
+message MyMessage {
+ required uint32 uid = 1;
+ required uint32 pid = 2;
+ required uint32 utime = 3;
+
+ oneof payload {
+ MsgType1 msg1 = 4;
+ MsgType2 msg2 = 5;
+ MsgType3 msg3 = 6;
+ }
+}
+~~~~
+
+Nanopb will generate `payload` as a C union and add an additional field
+`which_payload`:
+
+ typedef struct _MyMessage {
+ uint32_t uid;
+ uint32_t pid;
+ uint32_t utime;
+ pb_size_t which_payload;
+ union {
+ MsgType1 msg1;
+ MsgType2 msg2;
+ MsgType3 msg3;
+ } payload;
+ } MyMessage;
+
+`which_payload` indicates which of the `oneof` fields is actually set.
+The user is expected to set the field manually using the correct field
+tag:
+
+ MyMessage msg = MyMessage_init_zero;
+ msg.payload.msg2.value = true;
+ msg.which_payload = MyMessage_msg2_tag;
+
+Notice that neither `which_payload` field nor the unused fields in
+`payload` will consume any space in the resulting encoded message.
+
+When a field inside `oneof` contains `pb_callback_t`
+fields, the callback values cannot be set before decoding. This is
+because the different fields share the same storage space in C
+`union`. Instead either function name bound callbacks or a
+separate message level callback can be used. See
+[tests/oneof_callback](https://github.com/nanopb/nanopb/tree/master/tests/oneof_callback)
+for an example on this.
+
+## Extension fields
+
+Protocol Buffers supports a concept of [extension
+fields](https://developers.google.com/protocol-buffers/docs/proto#extensions),
+which are additional fields to a message, but defined outside the actual
+message. The definition can even be in a completely separate .proto
+file.
+
+The base message is declared as extensible by keyword `extensions` in
+the .proto file:
+
+~~~~ protobuf
+message MyMessage {
+ .. fields ..
+ extensions 100 to 199;
+}
+~~~~
+
+For each extensible message, `nanopb_generator.py` declares an
+additional callback field called `extensions`. The field and associated
+datatype `pb_extension_t` forms a linked list of handlers. When an
+unknown field is encountered, the decoder calls each handler in turn
+until either one of them handles the field, or the list is exhausted.
+
+The actual extensions are declared using the `extend` keyword in the
+.proto, and are in the global namespace:
+
+~~~~ protobuf
+extend MyMessage {
+ optional int32 myextension = 100;
+}
+~~~~
+
+For each extension, `nanopb_generator.py` creates a constant of type
+`pb_extension_type_t`. To link together the base message and the
+extension, you have to:
+
+1. Allocate storage for your field, matching the datatype in the
+ .proto. For example, for a `int32` field, you need a `int32_t`
+ variable to store the value.
+2. Create a `pb_extension_t` constant, with pointers to your variable
+ and to the generated `pb_extension_type_t`.
+3. Set the `message.extensions` pointer to point to the
+ `pb_extension_t`.
+
+An example of this is available in `tests/test_encode_extensions.c`
+and `tests/test_decode_extensions.c`.
+
+## Default values
+
+Protobuf has two syntax variants, proto2 and proto3. Of these proto2 has
+user definable default values that can be given in .proto file:
+
+~~~~ protobuf
+message MyMessage {
+ optional bytes foo = 1 [default = "ABC\x01\x02\x03"];
+ optional string bar = 2 [default = "åäö"];
+}
+~~~~
+
+Nanopb will generate both static and runtime initialization for the
+default values. In `myproto.pb.h` there will be a
+`#define MyMessage_init_default {...}` that can be used to initialize
+whole message into default values:
+
+ MyMessage msg = MyMessage_init_default;
+
+In addition to this, `pb_decode()` will initialize message
+fields to defaults at runtime. If this is not desired,
+`pb_decode_ex()` can be used instead.
+
+## Message framing
+
+Protocol Buffers does not specify a method of framing the messages for
+transmission. This is something that must be provided by the library
+user, as there is no one-size-fits-all solution. Typical needs for a
+framing format are to:
+
+1. Encode the message length.
+2. Encode the message type.
+3. Perform any synchronization and error checking that may be needed
+ depending on application.
+
+For example UDP packets already fullfill all the requirements, and TCP
+streams typically only need a way to identify the message length and
+type. Lower level interfaces such as serial ports may need a more robust
+frame format, such as HDLC (high-level data link control).
+
+Nanopb provides a few helpers to facilitate implementing framing
+formats:
+
+1. Functions `pb_encode_ex` and `pb_decode_ex` prefix the message
+ data with a varint-encoded length.
+2. Union messages and oneofs are supported in order to implement
+ top-level container messages.
+3. Message IDs can be specified using the `(nanopb_msgopt).msgid`
+ option and can then be accessed from the header.
+
+## Return values and error handling
+
+Most functions in nanopb return bool: `true` means success, `false`
+means failure. There is also support for error messages for
+debugging purposes: the error messages go in `stream->errmsg`.
+
+The error messages help in guessing what is the underlying cause of the
+error. The most common error conditions are:
+
+1) Invalid protocol buffers binary message.
+2) Mismatch between binary message and .proto message type.
+3) Unterminated message (incorrect message length).
+4) Exceeding the max_size or bytes_left of a stream.
+5) Exceeding the max_size/max_count of a string or array field
+6) IO errors in your own stream callbacks.
+7) Errors that happen in your callback functions.
+8) Running out of memory, i.e. stack overflow.
+9) Invalid field descriptors (would usually mean a bug in the generator).
diff --git a/docs/concepts.rst b/docs/concepts.rst
deleted file mode 100644
index a8d5e52..0000000
--- a/docs/concepts.rst
+++ /dev/null
@@ -1,468 +0,0 @@
-======================
-Nanopb: Basic concepts
-======================
-
-.. include :: menu.rst
-
-The things outlined here are the underlying concepts of the nanopb design.
-
-.. contents::
-
-Proto files
-===========
-All Protocol Buffers implementations use .proto files to describe the message
-format. The point of these files is to be a portable interface description
-language.
-
-Compiling .proto files for nanopb
----------------------------------
-Nanopb comes with a Python script to generate `.pb.c` and `.pb.h` files from
-the `.proto` definition::
-
- user@host:~$ python nanopb/generator/nanopb_generator.py message.proto
- Writing to message.pb.h and message.pb.c
-
-Internally this script uses Google `protoc` to parse the input file. If you
-do not have it available, you may receive an error message. You can install
-either `grpcio-tools` Python package using `pip`, or the `protoc` compiler
-itself from `protobuf-compiler` distribution package. Generally the Python
-package is recommended, because nanopb requires protoc version 3.6 or newer,
-and some distributions come with an older version.
-
-Modifying generator behaviour
------------------------------
-Using generator options, you can set maximum sizes for fields in order to
-allocate them statically. The preferred way to do this is to create an .options
-file with the same name as your .proto file::
-
- # Foo.proto
- message Foo {
- required string name = 1;
- }
-
-::
-
- # Foo.options
- Foo.name max_size:16
-
-For more information on this, see the `Proto file options`_ section in the
-reference manual.
-
-.. _`Proto file options`: reference.html#proto-file-options
-
-Streams
-=======
-
-Nanopb uses streams for accessing the data in encoded format.
-The stream abstraction is very lightweight, and consists of a structure (*pb_ostream_t* or *pb_istream_t*) which contains a pointer to a callback function.
-
-There are a few generic rules for callback functions:
-
-#) Return false on IO errors. The encoding or decoding process will abort immediately.
-#) Use state to store your own data, such as a file descriptor.
-#) *bytes_written* and *bytes_left* are updated by pb_write and pb_read.
-#) Your callback may be used with substreams. In this case *bytes_left*, *bytes_written* and *max_size* have smaller values than the original stream. Don't use these values to calculate pointers.
-#) Always read or write the full requested length of data. For example, POSIX *recv()* needs the *MSG_WAITALL* parameter to accomplish this.
-
-Output streams
---------------
-
-::
-
- struct _pb_ostream_t
- {
- bool (*callback)(pb_ostream_t *stream, const uint8_t *buf, size_t count);
- void *state;
- size_t max_size;
- size_t bytes_written;
- };
-
-The *callback* for output stream may be NULL, in which case the stream simply counts the number of bytes written. In this case, *max_size* is ignored.
-
-Otherwise, if *bytes_written* + bytes_to_be_written is larger than *max_size*, pb_write returns false before doing anything else. If you don't want to limit the size of the stream, pass SIZE_MAX.
-
-**Example 1:**
-
-This is the way to get the size of the message without storing it anywhere::
-
- Person myperson = ...;
- pb_ostream_t sizestream = {0};
- pb_encode(&sizestream, Person_fields, &myperson);
- printf("Encoded size is %d\n", sizestream.bytes_written);
-
-**Example 2:**
-
-Writing to stdout::
-
- bool callback(pb_ostream_t *stream, const uint8_t *buf, size_t count)
- {
- FILE *file = (FILE*) stream->state;
- return fwrite(buf, 1, count, file) == count;
- }
-
- pb_ostream_t stdoutstream = {&callback, stdout, SIZE_MAX, 0};
-
-Input streams
--------------
-For input streams, there is one extra rule:
-
-#) You don't need to know the length of the message in advance. After getting EOF error when reading, set bytes_left to 0 and return false. Pb_decode will detect this and if the EOF was in a proper position, it will return true.
-
-Here is the structure::
-
- struct _pb_istream_t
- {
- bool (*callback)(pb_istream_t *stream, uint8_t *buf, size_t count);
- void *state;
- size_t bytes_left;
- };
-
-The *callback* must always be a function pointer. *Bytes_left* is an upper limit on the number of bytes that will be read. You can use SIZE_MAX if your callback handles EOF as described above.
-
-**Example:**
-
-This function binds an input stream to stdin:
-
-::
-
- bool callback(pb_istream_t *stream, uint8_t *buf, size_t count)
- {
- FILE *file = (FILE*)stream->state;
- bool status;
-
- if (buf == NULL)
- {
- while (count-- && fgetc(file) != EOF);
- return count == 0;
- }
-
- status = (fread(buf, 1, count, file) == count);
-
- if (feof(file))
- stream->bytes_left = 0;
-
- return status;
- }
-
- pb_istream_t stdinstream = {&callback, stdin, SIZE_MAX};
-
-Data types
-==========
-
-Most Protocol Buffers datatypes have directly corresponding C datatypes, such as int32 is int32_t, float is float and bool is bool. However, the variable-length datatypes are more complex:
-
-1) Strings, bytes and repeated fields of any type map to callback functions by default.
-2) If there is a special option *(nanopb).max_size* specified in the .proto file, string maps to null-terminated char array and bytes map to a structure containing a char array and a size field.
-3) If *(nanopb).fixed_length* is set to *true* and *(nanopb).max_size* is also set, then bytes map to an inline byte array of fixed size.
-4) If there is a special option *(nanopb).max_count* specified on a repeated field, it maps to an array of whatever type is being repeated. Another field will be created for the actual number of entries stored.
-5) If *(nanopb).fixed_count* is set to *true* and *(nanopb).max_count* is also set, the field for the actual number of entries will not by created as the count is always assumed to be max count.
-
-=============================================================================== =======================
- field in .proto autogenerated in .h
-=============================================================================== =======================
-required string name = 1; pb_callback_t name;
-required string name = 1 [(nanopb).max_size = 40]; char name[40];
-repeated string name = 1 [(nanopb).max_size = 40]; pb_callback_t name;
-repeated string name = 1 [(nanopb).max_size = 40, (nanopb).max_count = 5]; | size_t name_count;
- | char name[5][40];
-required bytes data = 1 [(nanopb).max_size = 40]; | typedef struct {
- | size_t size;
- | pb_byte_t bytes[40];
- | } Person_data_t;
- | Person_data_t data;
-required bytes data = 1 [(nanopb).max_size = 40, (nanopb).fixed_length = true]; | pb_byte_t data[40];
-repeated int32 data = 1 [(nanopb).max_count = 5, (nanopb).fixed_count true]; | int32_t data[5];
-=============================================================================== =======================
-
-The maximum lengths are checked in runtime. If string/bytes/array exceeds the allocated length, *pb_decode* will return false.
-
-Note: For the *bytes* datatype, the field length checking may not be exact.
-The compiler may add some padding to the *pb_bytes_t* structure, and the nanopb runtime doesn't know how much of the structure size is padding. Therefore it uses the whole length of the structure for storing data, which is not very smart but shouldn't cause problems. In practise, this means that if you specify *(nanopb).max_size=5* on a *bytes* field, you may be able to store 6 bytes there. For the *string* field type, the length limit is exact.
-
-Note: When using the *fixed_count* option, the decoder assumes the repeated elements are
-received sequentially or that repeated elements for a non-packed field will not be interleaved with
-another *fixed_count* non-packed field.
-
-Field callbacks
-===============
-When a field has dynamic length, nanopb cannot statically allocate storage for it. Instead, it allows you to handle the field in whatever way you want, using a callback function.
-
-The `pb_callback_t`_ structure contains a function pointer and a *void* pointer called *arg* you can use for passing data to the callback. If the function pointer is NULL, the field will be skipped. A pointer to the *arg* is passed to the function, so that it can modify it and retrieve the value.
-
-The actual behavior of the callback function is different in encoding and decoding modes. In encoding mode, the callback is called once and should write out everything, including field tags. In decoding mode, the callback is called repeatedly for every data item.
-
-.. _`pb_callback_t`: reference.html#pb-callback-t
-
-Encoding callbacks
-------------------
-::
-
- bool (*encode)(pb_ostream_t *stream, const pb_field_iter_t *field, void * const *arg);
-
-When encoding, the callback should write out complete fields, including the wire type and field number tag. It can write as many or as few fields as it likes. For example, if you want to write out an array as *repeated* field, you should do it all in a single call.
-
-Usually you can use `pb_encode_tag_for_field`_ to encode the wire type and tag number of the field. However, if you want to encode a repeated field as a packed array, you must call `pb_encode_tag`_ instead to specify a wire type of *PB_WT_STRING*.
-
-If the callback is used in a submessage, it will be called multiple times during a single call to `pb_encode`_. In this case, it must produce the same amount of data every time. If the callback is directly in the main message, it is called only once.
-
-.. _`pb_encode`: reference.html#pb-encode
-.. _`pb_encode_tag_for_field`: reference.html#pb-encode-tag-for-field
-.. _`pb_encode_tag`: reference.html#pb-encode-tag
-
-This callback writes out a dynamically sized string::
-
- bool write_string(pb_ostream_t *stream, const pb_field_iter_t *field, void * const *arg)
- {
- char *str = get_string_from_somewhere();
- if (!pb_encode_tag_for_field(stream, field))
- return false;
-
- return pb_encode_string(stream, (uint8_t*)str, strlen(str));
- }
-
-Decoding callbacks
-------------------
-::
-
- bool (*decode)(pb_istream_t *stream, const pb_field_iter_t *field, void **arg);
-
-When decoding, the callback receives a length-limited substring that reads the contents of a single field. The field tag has already been read. For *string* and *bytes*, the length value has already been parsed, and is available at *stream->bytes_left*.
-
-The callback will be called multiple times for repeated fields. For packed fields, you can either read multiple values until the stream ends, or leave it to `pb_decode`_ to call your function over and over until all values have been read.
-
-.. _`pb_decode`: reference.html#pb-decode
-
-This callback reads multiple integers and prints them::
-
- bool read_ints(pb_istream_t *stream, const pb_field_iter_t *field, void **arg)
- {
- while (stream->bytes_left)
- {
- uint64_t value;
- if (!pb_decode_varint(stream, &value))
- return false;
- printf("%lld\n", value);
- }
- return true;
- }
-
-Function name bound callbacks
------------------------------
-::
-
- bool MyMessage_callback(pb_istream_t *istream, pb_ostream_t *ostream, const pb_field_iter_t *field);
-
-:istream: Input stream to read from, or NULL if called in encoding context.
-:ostream: Output stream to write to, or NULL if called in decoding context.
-:field: Iterator for the field currently being encoded or decoded.
-
-Storing function pointer in `pb_callback_t` fields inside the message requires extra storage space and is often cumbersome.
-As an alternative, the generator options `callback_function` and `callback_datatype` can be used to bind a callback function based on its name.
-
-Typically this feature is used by setting `callback_datatype` to e.g. `void*` or other data type used for callback state.
-Then the generator will automatically set `callback_function` to `MessageName_callback` and produce a prototype for it in generated `.pb.h`.
-By implementing this function in your own code, you will receive callbacks for fields without having to separately set function pointers.
-
-If you want to use function name bound callbacks for some fields and `pb_callback_t` for other fields,
-you can call `pb_default_field_callback` from the message-level callback.
-It will then read a function pointer from `pb_callback_t` and call it.
-
-Message descriptor
-==================
-
-For using the *pb_encode* and *pb_decode* functions, you need a description of
-all the fields contained in a message. This description is usually autogenerated from .proto file.
-
-For example this submessage in the Person.proto file::
-
- message Person {
- message PhoneNumber {
- required string number = 1 [(nanopb).max_size = 40];
- optional PhoneType type = 2 [default = HOME];
- }
- }
-
-This in turn generates a macro list in the `.pb.h` file::
-
- #define Person_PhoneNumber_FIELDLIST(X, a) \
- X(a, STATIC, REQUIRED, STRING, number, 1) \
- X(a, STATIC, OPTIONAL, UENUM, type, 2)
-
-Inside the `.pb.c` file there is a macro call to `PB_BIND`::
-
- PB_BIND(Person_PhoneNumber, Person_PhoneNumber, AUTO)
-
-These macros will in combination generate `pb_msgdesc_t` structure and associated lists::
-
- const uint32_t Person_PhoneNumber_field_info[] = { ... };
- const pb_msgdesc_t * const Person_PhoneNumber_submsg_info[] = { ... };
- const pb_msgdesc_t Person_PhoneNumber_msg = {
- 2,
- Person_PhoneNumber_field_info,
- Person_PhoneNumber_submsg_info,
- Person_PhoneNumber_DEFAULT,
- NULL,
- };
-
-The encoding and decoding functions take a pointer to this structure and use it
-to process each field in the message.
-
-Oneof
-=====
-Protocol Buffers supports `oneof`_ sections. Here is an example of ``oneof`` usage::
-
- message MsgType1 {
- required int32 value = 1;
- }
-
- message MsgType2 {
- required bool value = 1;
- }
-
- message MsgType3 {
- required int32 value1 = 1;
- required int32 value2 = 2;
- }
-
- message MyMessage {
- required uint32 uid = 1;
- required uint32 pid = 2;
- required uint32 utime = 3;
-
- oneof payload {
- MsgType1 msg1 = 4;
- MsgType2 msg2 = 5;
- MsgType3 msg3 = 6;
- }
- }
-
-Nanopb will generate ``payload`` as a C union and add an additional field ``which_payload``::
-
- typedef struct _MyMessage {
- uint32_t uid;
- uint32_t pid;
- uint32_t utime;
- pb_size_t which_payload;
- union {
- MsgType1 msg1;
- MsgType2 msg2;
- MsgType3 msg3;
- } payload;
- /* @@protoc_insertion_point(struct:MyMessage) */
- } MyMessage;
-
-``which_payload`` indicates which of the ``oneof`` fields is actually set.
-The user is expected to set the field manually using the correct field tag::
-
- MyMessage msg = MyMessage_init_zero;
- msg.payload.msg2.value = true;
- msg.which_payload = MyMessage_msg2_tag;
-
-Notice that neither ``which_payload`` field nor the unused fields in ``payload``
-will consume any space in the resulting encoded message.
-
-When a field inside ``oneof`` contains `pb_callback_t` fields, the callback
-values cannot be set before decoding. This is because the different fields
-share the same storage space in C `union`. Instead either function name bound
-callbacks or a separate message level callback can be used.
-See `tests/oneof_callback`_ for an example on this.
-
-.. _`oneof`: https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#oneof_and_oneof_field
-.. _`tests/oneof_callback`: https://github.com/nanopb/nanopb/tree/master/tests/oneof_callback
-
-Extension fields
-================
-Protocol Buffers supports a concept of `extension fields`_, which are
-additional fields to a message, but defined outside the actual message.
-The definition can even be in a completely separate .proto file.
-
-The base message is declared as extensible by keyword *extensions* in
-the .proto file::
-
- message MyMessage {
- .. fields ..
- extensions 100 to 199;
- }
-
-For each extensible message, *nanopb_generator.py* declares an additional
-callback field called *extensions*. The field and associated datatype
-*pb_extension_t* forms a linked list of handlers. When an unknown field is
-encountered, the decoder calls each handler in turn until either one of them
-handles the field, or the list is exhausted.
-
-The actual extensions are declared using the *extend* keyword in the .proto,
-and are in the global namespace::
-
- extend MyMessage {
- optional int32 myextension = 100;
- }
-
-For each extension, *nanopb_generator.py* creates a constant of type
-*pb_extension_type_t*. To link together the base message and the extension,
-you have to:
-
-1. Allocate storage for your field, matching the datatype in the .proto.
- For example, for a *int32* field, you need a *int32_t* variable to store
- the value.
-2. Create a *pb_extension_t* constant, with pointers to your variable and
- to the generated *pb_extension_type_t*.
-3. Set the *message.extensions* pointer to point to the *pb_extension_t*.
-
-An example of this is available in *tests/test_encode_extensions.c* and
-*tests/test_decode_extensions.c*.
-
-.. _`extension fields`: https://developers.google.com/protocol-buffers/docs/proto#extensions
-
-Default values
-==============
-Protobuf has two syntax variants, proto2 and proto3. Of these proto2 has user
-definable default values that can be given in .proto file::
-
- message MyMessage {
- optional bytes foo = 1 [default = "ABC\x01\x02\x03"];
- optional string bar = 2 [default = "åäö"];
- }
-
-Nanopb will generate both static and runtime initialization for the default
-values. In `myproto.pb.h` there will be a `#define MyMessage_init_default` that
-can be used to initialize whole message into default values::
-
- MyMessage msg = MyMessage_init_default;
-
-In addition to this, `pb_decode()` will initialize message fields to defaults
-at runtime. If this is not desired, `pb_decode_ex()` can be used instead.
-
-Message framing
-===============
-Protocol Buffers does not specify a method of framing the messages for transmission.
-This is something that must be provided by the library user, as there is no one-size-fits-all
-solution. Typical needs for a framing format are to:
-
-1. Encode the message length.
-2. Encode the message type.
-3. Perform any synchronization and error checking that may be needed depending on application.
-
-For example UDP packets already fullfill all the requirements, and TCP streams typically only
-need a way to identify the message length and type. Lower level interfaces such as serial ports
-may need a more robust frame format, such as HDLC (high-level data link control).
-
-Nanopb provides a few helpers to facilitate implementing framing formats:
-
-1. Functions *pb_encode_ex* and *pb_decode_ex* prefix the message data with a varint-encoded length.
-2. Union messages and oneofs are supported in order to implement top-level container messages.
-3. Message IDs can be specified using the *(nanopb_msgopt).msgid* option and can then be accessed from the header.
-
-Return values and error handling
-================================
-
-Most functions in nanopb return bool: *true* means success, *false* means failure. There is also some support for error messages for debugging purposes: the error messages go in *stream->errmsg*.
-
-The error messages help in guessing what is the underlying cause of the error. The most common error conditions are:
-
-1) Running out of memory, i.e. stack overflow.
-2) Invalid field descriptors (would usually mean a bug in the generator).
-3) IO errors in your own stream callbacks.
-4) Errors that happen in your callback functions.
-5) Exceeding the max_size or bytes_left of a stream.
-6) Exceeding the max_size/max_count of a string or array field
-7) Invalid protocol buffers binary message.
diff --git a/docs/feedback.html b/docs/feedback.html
new file mode 100644
index 0000000..15a45b2
--- /dev/null
+++ b/docs/feedback.html
@@ -0,0 +1,23 @@
+<div id="feedback_link">
+<a href="#feedback">Send feedback</a>
+</div>
+<div id="feedback">
+<b>Documentation feedback:</b>
+<form action="https://jpa.kapsi.fi/nanopb/documentation_feedback.cgi" method="post">
+<input type="hidden" name="page" value="doc_page_name_placeholder" />
+<label id="url_label" for="url">Type "human" in this box:</label>
+<input type="text" name="url" id="url" value=""></input><br />
+<textarea rows="8" name="message" cols="55"></textarea>
+<br />
+<input type="submit" name="btn_send" value="Send" />
+<a class="cancel" href="#">Cancel</a>
+</form>
+</div>
+<script type="text/javascript">
+window.onload = function() {
+ document.getElementById('url').value='javascript';
+ document.getElementById('url').style.display = 'none';
+ document.getElementById('url_label').style.display = 'none';
+}
+</script>
+
diff --git a/docs/index.md b/docs/index.md
new file mode 100644
index 0000000..d1ee234
--- /dev/null
+++ b/docs/index.md
@@ -0,0 +1,158 @@
+# Nanopb: Overview
+
+Nanopb is an ANSI-C library for encoding and decoding messages in
+Google's [Protocol Buffers](https://developers.google.com/protocol-buffers/docs/reference/overview)
+format with minimal requirements for RAM and code space. It is primarily
+suitable for 32-bit microcontrollers.
+
+Documentation version
+---------------------
+
+This documentation applies for nanopb 0.4.0 and later versions. For
+documentation of older releases,
+[see here](https://github.com/nanopb/nanopb/blob/maintenance_0.3/docs/index.rst).
+
+Overall structure
+-----------------
+
+For the runtime program, you always need `pb.h` for type declarations
+and `pb_common.h/c` for base functions. Depending on whether you want
+to encode, decode, or both, you also need `pb_encode.h/c` or
+`pb_decode.h/c`.
+
+The high-level encoding and decoding functions take a pointer to
+`pb_msgdesc_t` structure, which describes the fields of a message
+structure. Usually you want these autogenerated from a `.proto` file.
+The tool script `nanopb_generator.py` accomplishes this.
+
+
+
+So a typical project might include these files:
+
+1. Nanopb runtime library:
+ - pb.h
+ - pb_common.h and pb_common.c (always needed)
+ - pb_decode.h and pb_decode.c (needed for decoding messages)
+ - pb_encode.h and pb_encode.c (needed for encoding messages)
+
+2. Protocol description (you can have many):
+ - person.proto (just an example)
+ - person.pb.c (autogenerated, contains message descriptors)
+ - person.pb.h (autogenerated, contains type declarations and macros)
+
+Features and limitations
+------------------------
+
+**Features**
+
+1) Pure C runtime
+2) Small code size (5--10 kB depending on processor and compilation options, plus any message definitions)
+3) Small ram usage (typically \~300 bytes stack, plus any message structs)
+4) Allows specifying maximum size for strings and arrays, so that they can be allocated statically.
+5) No malloc needed: everything can be allocated statically or on the stack. Optional malloc support available.
+6) You can use either encoder or decoder alone to cut the code size in half.
+7) Support for most protobuf features, including: all data types,
+ nested submessages, default values, repeated and optional fields,
+ oneofs, packed arrays, extension fields.
+8) Callback mechanism for handling messages larger than can fit in available RAM.
+9) Extensive set of tests.
+
+**Limitations**
+
+1) Some speed has been sacrificed for code size.
+2) Encoding is focused on writing to streams. For memory buffers only it could be made more efficient.
+3) The deprecated Protocol Buffers feature called "groups" is not supported.
+4) Fields in the generated structs are ordered by the tag number, instead of the natural ordering in .proto file.
+5) Unknown fields are not preserved when decoding and re-encoding a message.
+6) Reflection (runtime introspection) is not supported. E.g. you can't request a field by giving its name in a string.
+7) Numeric arrays are always encoded as packed, even if not marked as packed in .proto.
+8) Cyclic references between messages are supported only in callback and malloc mode.
+9) Nanopb doesn't have a stable ABI (application binary interface)
+ between versions, so using it as a shared library (.so / .dll)
+ requires extra care.
+
+Getting started
+---------------
+
+For starters, consider this simple message:
+
+~~~~ protobuf
+message Example {
+ required int32 value = 1;
+}
+~~~~
+
+Save this in `message.proto` and compile it:
+
+ user@host:~$ python nanopb/generator/nanopb_generator.py message.proto
+
+You should now have in `message.pb.h`:
+
+ typedef struct {
+ int32_t value;
+ } Example;
+
+ extern const pb_msgdesc_t Example_msg;
+ #define Example_fields &Example_msg
+
+Then you have to include the nanopb headers and the generated header:
+
+ #include <pb_encode.h>
+ #include "message.pb.h"
+
+Now in your main program do this to encode a message:
+
+ Example mymessage = {42};
+ uint8_t buffer[10];
+ pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+ pb_encode(&stream, Example_fields, &mymessage);
+
+After that, buffer will contain the encoded message. The number of bytes
+in the message is stored in `stream.bytes_written`. You can feed the
+message to `protoc --decode=Example message.proto` to verify its
+validity.
+
+For a complete example of the simple case, see [examples/simple/simple.c](https://github.com/nanopb/nanopb/blob/master/examples/simple/simple.c).
+For a more complex example with network interface, see the [examples/network_server](https://github.com/nanopb/nanopb/tree/master/examples/network_server) subdirectory.
+
+Compiler requirements
+---------------------
+
+Nanopb should compile with most ansi-C compatible compilers. It however
+requires a few header files to be available:
+
+1) `string.h`, with these functions: `strlen`, `memcpy`, `memset`
+2) `stdint.h`, for definitions of `int32_t` etc.
+3) `stddef.h`, for definition of `size_t`
+4) `stdbool.h`, for definition of `bool`
+5) `limits.h`, for definition of `CHAR_BIT`
+
+If these header files do not come with your compiler, you can use the
+file `extra/pb_syshdr.h` instead. It contains an example of how to
+provide the dependencies. You may have to edit it a bit to suit your
+custom platform.
+
+To use the pb_syshdr.h, define `PB_SYSTEM_HEADER` as
+`"pb_syshdr.h"` (including the quotes). Similarly, you can provide a
+custom include file, which should provide all the dependencies listed
+above.
+
+Running the test cases
+----------------------
+
+Extensive unittests and test cases are included under the `tests`
+folder.
+
+To build the tests, you will need the [scons](http://www.scons.org/)
+build system. The tests should be runnable on most platforms. Windows
+and Linux builds are regularly tested. The tests also support embedded
+targets: STM32 (ARM Cortex-M) and AVR builds are regularly tested.
+
+In addition to the build system, you will also need a working Google
+Protocol Buffers `protoc` compiler, and the Python bindings for Protocol
+Buffers.
+
+Easiest way to install dependencies is to use the Python package manager
+[pip](https://pypi.org/project/pip/), which works on all platforms supported by Python:
+
+ pip3 install scons protobuf grpcio-tools
diff --git a/docs/index.rst b/docs/index.rst
deleted file mode 100644
index f404a96..0000000
--- a/docs/index.rst
+++ /dev/null
@@ -1,148 +0,0 @@
-=============================================
-Nanopb: Protocol Buffers with small code size
-=============================================
-
-.. include :: menu.rst
-
-Nanopb is an ANSI-C library for encoding and decoding messages in Google's `Protocol Buffers`__ format with minimal requirements for RAM and code space.
-It is primarily suitable for 32-bit microcontrollers.
-
-__ https://developers.google.com/protocol-buffers/docs/reference/overview
-
-Documentation version
-=====================
-
-This documentation applies for nanopb 0.4.0 and later versions. For documentation
-of older releases, `see here`__.
-
-__ https://github.com/nanopb/nanopb/blob/maintenance_0.3/docs/index.rst
-
-Overall structure
-=================
-
-For the runtime program, you always need *pb.h* for type declarations and *pb_common.h/c* for base functions.
-Depending on whether you want to encode, decode, or both, you also need *pb_encode.h/c* or *pb_decode.h/c*.
-
-The high-level encoding and decoding functions take a pointer to *pb_msgdesc_t* structure, which describes the fields of a message structure.
-Usually you want these autogenerated from a *.proto* file. The tool script *nanopb_generator.py* accomplishes this.
-
-.. image:: generator_flow.svg
-
-So a typical project might include these files:
-
-1) Nanopb runtime library:
- - pb.h
- - pb_common.h and pb_common.c (always needed)
- - pb_decode.h and pb_decode.c (needed for decoding messages)
- - pb_encode.h and pb_encode.c (needed for encoding messages)
-2) Protocol description (you can have many):
- - person.proto (just an example)
- - person.pb.c (autogenerated, contains initializers for const arrays)
- - person.pb.h (autogenerated, contains type declarations)
-
-Features and limitations
-========================
-
-**Features**
-
-#) Pure C runtime
-#) Small code size (5–10 kB depending on processor and compilation options, plus any message definitions)
-#) Small ram usage (typically ~300 bytes stack, plus any message structs)
-#) Allows specifying maximum size for strings and arrays, so that they can be allocated statically.
-#) No malloc needed: everything can be allocated statically or on the stack. Optional malloc support available.
-#) You can use either encoder or decoder alone to cut the code size in half.
-#) Support for most protobuf features, including: all data types, nested submessages, default values, repeated and optional fields, oneofs, packed arrays, extension fields.
-#) Callback mechanism for handling messages larger than can fit in available RAM.
-#) Extensive set of tests.
-
-**Limitations**
-
-#) Some speed has been sacrificed for code size.
-#) Encoding is focused on writing to streams. For memory buffers only it could be made more efficient.
-#) The deprecated Protocol Buffers feature called "groups" is not supported.
-#) Fields in the generated structs are ordered by the tag number, instead of the natural ordering in .proto file.
-#) Unknown fields are not preserved when decoding and re-encoding a message.
-#) Reflection (runtime introspection) is not supported. E.g. you can't request a field by giving its name in a string.
-#) Numeric arrays are always encoded as packed, even if not marked as packed in .proto.
-#) Cyclic references between messages are supported only in callback and malloc mode.
-#) Nanopb doesn't have a stable ABI (application binary interface) between versions, so using it as a shared library (.so / .dll) requires extra care.
-
-Getting started
-===============
-
-For starters, consider this simple message::
-
- message Example {
- required int32 value = 1;
- }
-
-Save this in *message.proto* and compile it::
-
- user@host:~$ python nanopb/generator/nanopb_generator.py message.proto
-
-You should now have in *message.pb.h*::
-
- typedef struct {
- int32_t value;
- } Example;
-
- extern const pb_msgdesc_t Example_msg;
- #define Example_fields &Example_msg
-
-Then you have to include the nanopb headers and the generated header::
-
- #include <pb_encode.h>
- #include "message.pb.h"
-
-Now in your main program do this to encode a message::
-
- Example mymessage = {42};
- uint8_t buffer[10];
- pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
- pb_encode(&stream, Example_fields, &mymessage);
-
-After that, buffer will contain the encoded message.
-The number of bytes in the message is stored in *stream.bytes_written*.
-You can feed the message to *protoc --decode=Example message.proto* to verify its validity.
-
-For a complete example of the simple case, see *examples/simple/simple.c*.
-For a more complex example with network interface, see the *examples/network_server* subdirectory.
-
-Compiler requirements
-=====================
-Nanopb should compile with most ansi-C compatible compilers. It however
-requires a few header files to be available:
-
-#) *string.h*, with these functions: *strlen*, *memcpy*, *memset*
-#) *stdint.h*, for definitions of *int32_t* etc.
-#) *stddef.h*, for definition of *size_t*
-#) *stdbool.h*, for definition of *bool*
-#) *limits.h*, for definition of *CHAR_BIT*
-
-If these header files do not come with your compiler, you can use the
-file *extra/pb_syshdr.h* instead. It contains an example of how to provide
-the dependencies. You may have to edit it a bit to suit your custom platform.
-
-To use the pb_syshdr.h, define *PB_SYSTEM_HEADER* as *"pb_syshdr.h"* (including the quotes).
-Similarly, you can provide a custom include file, which should provide all the dependencies
-listed above.
-
-Running the test cases
-======================
-Extensive unittests and test cases are included under the *tests* folder.
-
-To build the tests, you will need the `scons`__ build system. The tests should
-be runnable on most platforms. Windows and Linux builds are regularly tested.
-The tests also support embedded targets: STM32 (ARM Cortex-M) and AVR builds
-are regularly tested.
-
-__ http://www.scons.org/
-
-In addition to the build system, you will also need a working Google Protocol
-Buffers *protoc* compiler, and the Python bindings for Protocol Buffers.
-
-Easiest way to install dependencies is to use the Python package manager `pip`,
-which works on all platforms supported by Python::
-
- pip install scons protobuf grpcio-tools
-
diff --git a/docs/lsr.css b/docs/lsr.css
index 9a15717..2762f1c 100644
--- a/docs/lsr.css
+++ b/docs/lsr.css
@@ -1,8 +1,13 @@
/*
-Author: Peter Parente
+Original author: Peter Parente
Date: 2008/01/22
Version: 1.0 (modified)
Copyright: This stylesheet has been placed in the public domain - free to edit and use for all uses.
+
+--
+
+Heavily modified for use in nanopb documentation.
+2011-2020 Petteri Aimonen
*/
body {
@@ -47,6 +52,16 @@
border: 0;
}
+blockquote {
+ border-left: 3px solid black;
+ background-color: #f6f6f6;
+ padding: 0.5em 1em 0.2em 1em;
+}
+
+li {
+ margin-bottom: 0.5em;
+}
+
p {
margin: 0.5em 0 1em 0;
line-height: 1.5em;
@@ -88,45 +103,53 @@
background-color: #f6f6f6;
}
-h1.title {
- color: #003a6b;
- font-size: 180%;
- margin-bottom: 0em;
+code {
+ border: 1px solid rgba(128, 128, 128, 0.1);
+ padding: 1px 5px;
+ border-radius: 5px;
+ background-color: rgba(128, 128, 128, 0.05);
+ white-space: nowrap;
}
-h2.subtitle {
- color: #003a6b;
- border-bottom: 0px;
+pre code {
+ border: none;
+ background-color: transparent;
+ padding: 0px;
}
h1, h2, h3, h4, h5, h6 {
- color: #555;
+ color: #000;
background-color: transparent;
margin: 0em;
padding-top: 0.5em;
}
h1 {
- font-size: 150%;
+ color: #003a6b;
+ font-size: 180%;
margin-bottom: 0.5em;
border-bottom: 2px solid #aaa;
}
h2 {
+ color: #003a6b;
font-size: 130%;
margin-bottom: 0.5em;
border-bottom: 1px solid #aaa;
+ margin-top: 1.5em;
}
h3 {
font-size: 120%;
margin-bottom: 0.5em;
+ margin-top: 1.0em;
}
h4 {
font-size: 110%;
font-weight: bold;
margin-bottom: 0.5em;
+ margin-top: 0.5em;
}
h5 {
@@ -149,93 +172,62 @@
margin-bottom: 1.5em;
}
-div.admonition, div.note, div.tip, div.caution, div.important {
- margin: 2em 2em;
- padding: 0em 1em;
- border-top: 1px solid #aaa;
- border-left: 1px solid #aaa;
- border-bottom: 2px solid #555;
- border-right: 2px solid #555;
-}
-div.important {
- background: transparent url('../images/important.png') 10px 2px no-repeat;
-}
-div.caution {
- background: transparent url('../images/caution.png') 10px 2px no-repeat;
-}
-
-div.note {
- background: transparent url('../images/note.png') 10px 2px no-repeat;
-}
-
-div.tip {
- background: transparent url('../images/tip.png') 10px 2px no-repeat;
-}
-
-div.admonition-example {
- background: transparent url('../images/tip.png') 10px 2px no-repeat;
-}
-
-div.admonition-critical-example {
- background: transparent url('../images/important.png') 10px 2px no-repeat;
-}
-
-p.admonition-title {
- font-weight: bold;
- border-bottom: 1px solid #aaa;
- padding-left: 30px;
-}
-
-table.docutils {
+table {
text-align: left;
- border: 1px solid gray;
border-collapse: collapse;
margin: 1.5em 0em;
}
-table.docutils caption {
- font-style: italic;
+table td, table th {
+ padding: 0.25em 1em;
+ border-top: 1px solid gray;
+ border-bottom: 1px solid gray;
}
-table.docutils td, table.docutils th {
- padding: 0.25em 0.5em;
-}
-
-th.field-name {
- text-align: right;
- width: 15em;
-}
-
-table.docutils th {
- font-family: monospace;
- background-color: #f6f6f6;
- vertical-align: middle;
-}
-
-table.field-list {
- border: none;
-}
-
-div.sidebar {
+#index {
margin: 2em 2em 2em 0em;
padding: 0em 1em;
border-top: 1px solid #aaa;
border-left: 1px solid #aaa;
border-bottom: 2px solid #555;
border-right: 2px solid #555;
+ max-width: 20em;
}
-p.sidebar-title {
+#index h2 {
margin-bottom: 0em;
+ margin-top: 0em;
color: #003a6b;
+ font-size: 100%;
border-bottom: 1px solid #aaa;
font-weight: bold;
}
-p.sidebar-subtitle {
- margin-top: 0em;
- font-style: italic;
- color: #003a6b;
+#feedback_link {
+ float: right;
}
+
+#feedback {
+ visibility: hidden;
+ padding: 1em;
+ border-radius: 5px;
+ position: fixed;
+ top: 10em;
+ right: 10em;
+ background: white;
+ border-top: 1px solid #aaa;
+ border-left: 1px solid #aaa;
+ border-bottom: 2px solid #555;
+ border-right: 2px solid #555;
+}
+
+#feedback:target {
+ visibility: visible;
+}
+
+#feedback .cancel {
+ color: #666;
+ padding-left: 2em;
+}
\ No newline at end of file
diff --git a/docs/menu.rst b/docs/menu.rst
deleted file mode 100644
index afd9772..0000000
--- a/docs/menu.rst
+++ /dev/null
@@ -1,15 +0,0 @@
-.. sidebar :: Documentation index
-
- 1) `Overview`_
- 2) `Concepts`_
- 3) `API reference`_
- 4) `Security model`_
- 5) `Migration from older versions`_
- 6) `New features`_
-
-.. _`Overview`: index.html
-.. _`Concepts`: concepts.html
-.. _`API reference`: reference.html
-.. _`Security model`: security.html
-.. _`Migration from older versions`: migration.html
-.. _`New features`: whats_new.html
diff --git a/docs/migration.md b/docs/migration.md
new file mode 100644
index 0000000..a4a14a4
--- /dev/null
+++ b/docs/migration.md
@@ -0,0 +1,590 @@
+# Nanopb: Migration from older versions
+
+This document details all the breaking changes that have been made to
+nanopb since its initial release. For each change, the rationale and
+required modifications of user applications are explained. Also any
+error indications are included, in order to make it easier to find this
+document.
+
+Nanopb-0.4.3 (2020-09-21)
+-------------------------
+
+### pb_msgdesc_t struct has new fields
+
+**Changes:** New fields `required_field_count` and
+`largest_tag` were added to `pb_msgdesc_t`
+and existing fields were reordered.
+
+**Required actions:** All `.pb.c` files must be recompiled.
+Regeneration is not needed.
+
+**Error indications:** Messages may fail to encode or decode, or the
+code can crash inside `load_descriptor_values()` in
+`pb_common.c`.
+
+Nanopb-0.4.2 (2020-06-23)
+-------------------------
+
+### Generator now uses Python 3 by default
+
+**Rationale:** Previously `nanopb-generator.py` had hashbang
+of `#!/usr/bin/env python`, which would execute with Python
+2 on most systems. Python 2 is now deprecated and many libraries are
+dropping support for it, which makes installing dependencies difficult.
+While `nanopb_generator.py` has worked with Python 3 for
+years now, and overriding the python version was possible with
+virtualenv, that was an extra complication.
+
+**Changes:** Hashbang now uses `#!/usr/bin/env python3`.
+New file `nanopb_generator.py2` can be used to run with
+Python 2, if necessary.
+
+**Required actions:** If possible, just verify Python 3 is installed and
+necessary dependencies are installed for it. For example `pip3 install protobuf grpcio-tools`
+should take care of it. If this is not possible, call `nanopb_generator.py2` from your build
+scripts instead.
+
+**Error indications:** `python3: command not found` if
+Python 3 is not installed.
+`Could not import the Google protobuf Python libraries` if dependencies are only installed for Python 2.
+
+Nanopb-0.4.0 (2019-12-20)
+-------------------------
+
+### New field descriptor format
+
+**Rationale:** Previously information about struct fields was stored as
+an array of `pb_field_t` structures. This was a
+straightforward method, but required allocating space for e.g.
+submessage type and array size for all fields, even though most fields
+are not submessages nor arrays.
+
+**Changes:** Now field information is encoded more efficiently in
+`uint32_t` array in a variable-length format. Old
+`pb_field_t` structure has been removed and it is now a
+typedef for `pb_field_iter_t`. This retains compatibility
+with most old callback definitions. The field definitions in
+`.pb.h` files are now of type `pb_msgdesc_t`.
+
+**Required actions:** If your own code accesses the low-level field
+information in `pb_field_t`, it must be modified to do so
+only through the functions declared in `pb_common.h`.
+
+**Error indications:** `incompatible pointer type` errors
+relating to `pb_field_t`
+
+### Changes to generator default options
+
+**Rationale:** Previously nanopb_generator added a timestamp header to
+generated files and used only basename of files in
+`#include` directives. This is different than what the
+`protoc` C++ backend does.
+
+**Changes:** Now default options are `--no-timestamp` and
+`--no-strip-path`.
+
+**Required actions:** If old behaviour is desired, add
+`--timestamp` and `--strip-path` options to
+`nanopb_generator.py` or on `protoc` command
+line as `--nanopb_out=--timestamp,--strip-path:outdir`.
+
+**Error indications:** Compiler error: cannot find include file
+`mymessage.pb.h` when compiling
+`mymessage.pb.c`.
+
+### Removal of bundled plugin.proto
+
+**Rationale:** Google's Python protobuf library, which is used in
+nanopb generator, has included `plugin_pb2` with it since
+version 3.1.0. It is not necessary to bundle it with nanopb anymore.
+
+**Required actions:** Update `python-protobuf` to version
+3.1.0 or newer.
+
+**Error indications:** `ImportError: No module named compiler.plugin_pb2`
+
+### .options file is now always case-sensitive
+
+**Rationale:** Previously field names in `.options` file
+were case-sensitive on Linux and case-insensitive on Windows. This was
+by accident. Because `.proto` files are case-sensitive,
+`.options` files should be too.
+
+**Changes:** Now field names in `.options` are always
+case-sensitive, and matched by `fnmatchcase()` instead of
+`fnmatch()`.
+
+**Required actions:** If field names in `.options` are not
+capitalized the same as in `.proto`, they must be updated.
+
+### `CHAR_BIT` define is now needed
+
+**Rationale:** To check whether the platform has 8-bit or larger chars,
+the C standard `CHAR_BIT` macro is needed.
+
+**Changes:** `pb.h` now includes `limits.h` for this macro.
+
+**Required actions:** If your platform doesn't have `limits.h`
+available, you can define the macro in `pb_syshdr.h`. There is an
+example in `extra` directory.
+
+**Error indications:** `"Cannot find include file <limits.h>."` or
+`"Undefined identifier: CHAR_BIT."`
+
+### Strings must now always be null-terminated
+
+**Rationale:** Previously `pb_encode()` would accept non-terminated
+strings and assume that they are the full length of the defined array.
+However, `pb_decode()` would reject such messages because null
+terminator wouldn't fit in the array.
+
+**Changes:** `pb_encode()` will now return an error if null terminator
+is missing. Maximum encoded message size calculation is changed
+accordingly so that at most `max_size-1` strings are assumed. New field
+option `max_length` can be used to define the maximum string length,
+instead of the array size.
+
+**Required actions:** If your strings were previously filling the whole
+allocated array, increase the size of the field by 1.
+
+**Error indications:** `pb_encode()` returns error `unterminated string`.
+
+### Removal of per-field default value constants
+
+**Rationale:** Previously nanopb declared a
+`fieldname_default` constant variable for each field with a
+default value, and used these internally to initialize messages. This
+however used unnecessarily large amount of storage for the values. The
+variables were mostly for internal usage, but were available in the
+header file.
+
+**Changes:** Default values are now stored as an encoded protobuf
+message.
+
+**Required actions:** If your code previously used default constants, it
+will have to be adapted to take the default value in some other way,
+such as by defining
+`static const MyMessage msg_default = MyMessage_init_default;` and accessing
+`msg_default.fieldname`.
+
+**Error indications:** Compiler error about `fieldname_default` being undeclared.
+
+### Zero tag in message now raises error by default
+
+**Rationale:** Previously nanopb has allowed messages to be terminated
+by a null byte, which is read as zero tag value. Most other protobuf
+implementations don't support this, so it is not very useful feature.
+It has also been noted that this can complicate debugging issues with
+corrupted messages.
+
+**Changes:** `pb_decode()` now gives error when it
+encounters zero tag value. A new function `pb_decode_ex()`
+supports flag `PB_DECODE_NULLTERMINATED` that supports
+decoding null terminated messages.
+
+**Required actions:** If application uses null termination for messages,
+switch it to use `pb_decode_ex()` and
+`pb_encode_ex()`. If compatibility with 0.3.9.x is needed,
+there are also `pb_decode_nullterminated()` and
+`pb_encode_nullterminated()` macros, which work both in
+0.4.0 and 0.3.9.
+
+**Error indications:** Error message from `pb_decode()`: `zero_tag`.
+
+### Submessages now have has_field in proto3 mode
+
+**Rationale:** Previously nanopb considered proto3 submessages as
+present only when their contents was non-zero. Most other protobuf
+libraries allow explicit null state for submessages.
+
+**Changes:** Submessages now have separate `has_field` in
+proto3 mode also.
+
+**Required actions:** When using submessages in proto3 mode, user code
+must now set `mymsg.has_submsg = true` for each submessage
+that is present. Alternatively, the field option
+`proto3_singular_msgs` can be used to restore the old
+behavior.
+
+**Error indications:** Submessages do not get encoded.
+
+### PB_OLD_CALLBACK_STYLE option has been removed
+
+**Rationale:** Back in 2013, function signature for callbacks was
+changed. The `PB_OLD_CALLBACK_STYLE` option allowed
+compatibility with old code, but complicated code and testing because of
+the different options.
+
+**Changes:** `PB_OLD_CALLBACK_STYLE` option no-longer has
+any effect.
+
+**Required actions:** If `PB_OLD_CALLBACK_STYLE` option
+was in use previously, function signatures must be updated to use double
+pointers (`void**` and `void * const *`).
+
+**Error indications:** Assignment from incompatible pointer type.
+
+### protoc insertion points are no longer included by default
+
+**Rationale:** Protoc allows including comments in form
+`@@protoc_insertion_point` to identify locations for
+other plugins to insert their own extra content. Previously these were
+included by default, but they clutter the generated files and are rarely
+used.
+
+**Changes:** Insertion points are now included only when
+`--protoc-insertion-points` option is passed to the
+generator.
+
+Nanopb-0.3.9.4, 0.4.0 (2019-10-13)
+----------------------------------
+
+### Fix generation of min/max defines for enum types
+
+**Rationale:** Nanopb generator makes \#defines for enum minimum and
+maximum value. Previously these defines incorrectly had the first and
+last enum value, instead of the actual minimum and maximum. (issue
+#405)
+
+**Changes:** Minimum define now always has the smallest value, and
+maximum define always has the largest value.
+
+**Required actions:** If these defines are used and enum values in
+.proto file are not defined in ascending order, user code behaviour may
+change. Check that user code doesn\'t expect the old, incorrect
+first/last behaviour.
+
+### Fix undefined behavior related to bool fields
+
+**Rationale:** In C99, `bool` variables are not allowed to
+have other values than `true` and `false`.
+Compilers use this fact in optimization, and constructs like
+`int foo = msg.has_field ? 100 : 0;` will give unexpected results
+otherwise. Previously nanopb didn\'t enforce that decoded bool fields
+had valid values.
+
+**Changes:** Bool fields are now handled separately as
+`PB_LTYPE_BOOL`. The `LTYPE` descriptor
+numbers for other field types were renumbered.
+
+**Required actions:** Source code files must be recompiled, but
+regenerating `.pb.h`/`.pb.c` files from
+`.proto` is not required. If user code directly uses the
+nanopb internal field representation (search for
+`PB_LTYPE_VARINT` in source), it may need updating.
+
+Nanopb-0.3.9.1, 0.4.0 (2018-04-14)
+----------------------------------
+
+### Fix handling of string and bytes default values
+
+**Rationale:** Previously nanopb didn't properly decode special
+character escapes like `\200` emitted by protoc. This caused these
+escapes to end up verbatim in the default values in .pb.c file.
+
+**Changes:** Escapes are now decoded, and e.g. `\200` or `\x80`
+results in {0x80} for bytes field and `"\x80"` for string field.
+
+**Required actions:** If code has previously relied on `\` in default
+value being passed through verbatim, it must now be changed to `\\`.
+
+Nanopb-0.3.8 (2017-03-05)
+-------------------------
+
+### Fully drain substreams before closing
+
+**Rationale:** If the substream functions were called directly and the
+caller did not completely empty the substring before closing it, the
+parent stream would be put into an incorrect state.
+
+**Changes:** `pb_close_string_substream` can now error and returns a
+boolean.
+
+**Required actions:** Add error checking onto any call to
+`pb_close_string_substream`.
+
+### Change oneof format in .pb.c files
+
+**Rationale:** Previously two oneofs in a single message would be
+erroneously handled as part of the same union.
+
+**Changes:** Oneofs fields now use special `PB_DATAOFFSET_UNION`
+offset type in generated .pb.c files to distinguish whether they are the
+first or following field inside an union.
+
+**Required actions:** Regenerate `.pb.c/.pb.h` files with new nanopb
+version if oneofs are used.
+
+Nanopb-0.3.5 (2016-02-13)
+-------------------------
+
+### Add support for platforms without uint8_t
+
+**Rationale:** Some platforms cannot access 8-bit sized values directly,
+and do not define `uint8_t`. Nanopb previously didn\'t support these
+platforms.
+
+**Changes:** References to `uint8_t` were replaced with several
+alternatives, one of them being a new `pb_byte_t` typedef. This in
+turn uses `uint_least8_t` which means the smallest available type.
+
+**Required actions:** If your platform does not have a
+standards-compliant `stdint.h`, it may lack the definition for
+`[u]int_least8_t`. This must be added manually, example can be found
+in `extra/pb_syshdr.h`.
+
+**Error indications:** Compiler error: `"unknown type name 'uint_least8_t'"`.
+
+Nanopb-0.3.2 (2015-01-24)
+-------------------------
+
+### Add support for OneOfs
+
+**Rationale:** Previously nanopb did not support the `oneof` construct
+in `.proto` files. Those fields were generated as regular `optional`
+fields.
+
+**Changes:** OneOfs are now generated as C unions. Callback fields are
+not supported inside oneof and generator gives an error.
+
+**Required actions:** The generator option `no_unions` can be used to
+restore old behaviour and to allow callbacks to be used. To use unions,
+one change is needed: use `which_xxxx` field to detect which field is
+present, instead of `has_xxxx`. Compare the value against
+`MyStruct_myfield_tag`.
+
+**Error indications:** Generator error: `"Callback fields inside of
+oneof are not supported"`. Compiler error: `"Message"` has no member
+named `"has_xxxx"`.
+
+Nanopb-0.3.0 (2014-08-26)
+-------------------------
+
+### Separate field iterator logic to pb_common.c
+
+**Rationale:** Originally, the field iteration logic was simple enough
+to be duplicated in `pb_decode.c` and `pb_encode.c`. New field types
+have made the logic more complex, which required the creation of a new
+file to contain the common functionality.
+
+**Changes:** There is a new file, `pb_common.c`, which must be included
+in builds.
+
+**Required actions:** Add `pb_common.c` to build rules. This file is
+always required. Either `pb_decode.c` or `pb_encode.c` can still be
+left out if some functionality is not needed.
+
+**Error indications:** Linker error: undefined reference to
+`pb_field_iter_begin`, `pb_field_iter_next` or similar.
+
+### Change data type of field counts to pb_size_t
+
+**Rationale:** Often nanopb is used with small arrays, such as 255 items
+or less. Using a full `size_t` field to store the array count wastes
+memory if there are many arrays. There already exists parameters
+`PB_FIELD_16BIT` and `PB_FIELD_32BIT` which tell nanopb what is the
+maximum size of arrays in use.
+
+**Changes:** Generator will now use `pb_size_t` for the array
+`_count` fields. The size of the type will be controlled by the
+`PB_FIELD_16BIT` and `PB_FIELD_32BIT` compilation time options.
+
+**Required actions:** Regenerate all `.pb.h` files. In some cases casts
+to the `pb_size_t` type may need to be added in the user code when
+accessing the `_count` fields.
+
+**Error indications:** Incorrect data at runtime, crashes. But note that
+other changes in the same version already require regenerating the files
+and have better indications of errors, so this is only an issue for
+development versions.
+
+### Renamed some macros and identifiers
+
+**Rationale:** Some names in nanopb core were badly chosen and
+conflicted with ISO C99 reserved names or lacked a prefix. While they
+haven\'t caused trouble so far, it is reasonable to switch to
+non-conflicting names as these are rarely used from user code.
+
+**Changes:** The following identifier names have changed:
+
+ - Macros:
+ - STATIC_ASSERT(x) -> PB_STATIC_ASSERT(x)
+ - UNUSED(x) -> PB_UNUSED(x)
+ - Include guards:
+ - PB_filename -> PB_filename_INCLUDED
+ - Structure forward declaration tags:
+ - _pb_field_t -> pb_field_s
+ - _pb_bytes_array_t -> pb_bytes_array_s
+ - _pb_callback_t -> pb_callback_s
+ - _pb_extension_type_t -> pb_extension_type_s
+ - _pb_extension_t -> pb_extension_s
+ - _pb_istream_t -> pb_istream_s
+ - _pb_ostream_t -> pb_ostream_s
+
+**Required actions:** Regenerate all `.pb.c` files. If you use any of
+the above identifiers in your application code, perform search-replace
+to the new name.
+
+**Error indications:** Compiler errors on lines with the macro/type
+names.
+
+Nanopb-0.2.9 (2014-08-09)
+-------------------------
+
+### Change semantics of generator -e option
+
+**Rationale:** Some compilers do not accept filenames with two dots
+(like in default extension .pb.c). The `-e` option to the generator
+allowed changing the extension, but not skipping the extra dot.
+
+**Changes:** The `-e` option in generator will no longer add the
+prepending dot. The default value has been adjusted accordingly to
+`.pb.c` to keep the default behaviour the same as before.
+
+**Required actions:** Only if using the generator -e option. Add dot
+before the parameter value on the command line.
+
+**Error indications:** File not found when trying to compile generated
+files.
+
+Nanopb-0.2.7 (2014-04-07)
+-------------------------
+
+### Changed pointer-type bytes field datatype
+
+**Rationale:** In the initial pointer encoding support since
+nanopb-0.2.5, the bytes type used a separate `pb_bytes_ptr_t` type to
+represent `bytes` fields. This made it easy to encode data from a
+separate, user-allocated buffer. However, it made the internal logic
+more complex and was inconsistent with the other types.
+
+**Changes:** Dynamically allocated bytes fields now have the
+`pb_bytes_array_t` type, just like statically allocated ones.
+
+**Required actions:** Only if using pointer-type fields with the bytes
+datatype. Change any access to `msg->field.size` to
+`msg->field->size`. Change any allocation to reserve space of amount
+`PB_BYTES_ARRAY_T_ALLOCSIZE(n)`. If the data pointer was begin
+assigned from external source, implement the field using a callback
+function instead.
+
+**Error indications:** Compiler error: unknown type name
+`pb_bytes_ptr_t`.
+
+Nanopb-0.2.4 (2013-11-07)
+-------------------------
+
+### Remove the NANOPB_INTERNALS compilation option
+
+**Rationale:** Having the option in the headers required the functions
+to be non-static, even if the option is not used. This caused errors on
+some static analysis tools.
+
+**Changes:** The `\#ifdef` and associated functions were removed from
+the header.
+
+**Required actions:** Only if the `NANOPB_INTERNALS` option was
+previously used. Actions are as listed under nanopb-0.1.3 and
+nanopb-0.1.6.
+
+**Error indications:** Compiler warning: implicit declaration of
+function `pb_dec_string`, `pb_enc_string`, or similar.
+
+Nanopb-0.2.1 (2013-04-14)
+-------------------------
+
+### Callback function signature
+
+**Rationale:** Previously the auxilary data to field callbacks was
+passed as `void*`. This allowed passing of any data, but made it
+unnecessarily complex to return a pointer from callback.
+
+**Changes:** The callback function parameter was changed to `void**`.
+
+**Required actions:** You can continue using the old callback style by
+defining `PB_OLD_CALLBACK_STYLE`. Recommended action is to:
+
+- Change the callback signatures to contain `void**` for decoders and `void * const *` for encoders.
+- Change the callback function body to use **arg` instead of `arg`.
+
+**Error indications:** Compiler warning: assignment from incompatible
+pointer type, when initializing `funcs.encode` or `funcs.decode`.
+
+Nanopb-0.2.0 (2013-03-02)
+-------------------------
+
+### Reformatted generated .pb.c file using macros
+
+**Rationale:** Previously the generator made a list of C `pb_field_t`
+initializers in the .pb.c file. This led to a need to regenerate all
+.pb.c files after even small changes to the `pb_field_t` definition.
+
+**Changes:** Macros were added to pb.h which allow for cleaner
+definition of the .pb.c contents. By changing the macro definitions,
+changes to the field structure are possible without breaking
+compatibility with old .pb.c files.
+
+**Required actions:** Regenerate all .pb.c files from the .proto
+sources.
+
+**Error indications:** Compiler warning: implicit declaration of
+function `pb_delta_end`.
+
+### Changed pb_type_t definitions
+
+**Rationale:** The `pb_type_t` was previously an enumeration type.
+This caused warnings on some compilers when using bitwise operations to
+set flags inside the values.
+
+**Changes:** The `pb_type_t` was changed to *typedef uint8_t*. The
+values were changed to `#define`. Some value names were changed for
+consistency.
+
+**Required actions:** Only if you directly access the
+`pb_field_t` contents in your own code, something which is
+not usually done. Needed changes:
+
+- Change `PB_HTYPE_ARRAY` to `PB_HTYPE_REPEATED`.
+- Change `PB_HTYPE_CALLBACK` to `PB_ATYPE()` and `PB_ATYPE_CALLBACK`.
+
+**Error indications:** Compiler error: `PB_HTYPE_ARRAY` or
+`PB_HTYPE_CALLBACK` undeclared.
+
+Nanopb-0.1.6 (2012-09-02)
+-------------------------
+
+### Refactored field decoder interface
+
+**Rationale:** Similarly to field encoders in nanopb-0.1.3.
+
+**Changes:** New functions with names `pb_decode_*` were added.
+
+**Required actions:** By defining NANOPB_INTERNALS, you can still keep
+using the old functions. Recommended action is to replace any calls with
+the newer `pb_decode_*` equivalents.
+
+**Error indications:** Compiler warning: implicit declaration of
+function `pb_dec_string`, `pb_dec_varint`, `pb_dec_submessage` or
+similar.
+
+Nanopb-0.1.3 (2012-06-12)
+-------------------------
+
+### Refactored field encoder interface
+
+**Rationale:** The old `pb_enc_*` functions were designed mostly for
+the internal use by the core. Because they are internally accessed
+through function pointers, their signatures had to be common. This led
+to a confusing interface for external users.
+
+**Changes:** New functions with names `pb_encode_*` were added. These
+have easier to use interfaces. The old functions are now only thin
+wrappers for the new interface.
+
+**Required actions:** By defining NANOPB_INTERNALS, you can still keep
+using the old functions. Recommended action is to replace any calls with
+the newer `pb_encode_*` equivalents.
+
+**Error indications:** Compiler warning: implicit declaration of
+function `pb_enc_string`, *pb_enc_varint,`pb_enc_submessage\` or
+similar.
diff --git a/docs/migration.rst b/docs/migration.rst
deleted file mode 100644
index b8b8a1b..0000000
--- a/docs/migration.rst
+++ /dev/null
@@ -1,557 +0,0 @@
-=====================================
-Nanopb: Migration from older versions
-=====================================
-
-.. include :: menu.rst
-
-This document details all the breaking changes that have been made to nanopb
-since its initial release. For each change, the rationale and required
-modifications of user applications are explained. Also any error indications
-are included, in order to make it easier to find this document.
-
-.. contents ::
-
-Nanopb-0.4.3 (2020-09-21)
-=========================
-
-pb_msgdesc_t struct has new fields
-----------------------------------
-
-**Changes:** New fields `required_field_count` and `largest_tag` were added
-to `pb_msgdesc_t` and existing fields were reordered.
-
-**Required actions:** All `.pb.c` files must be recompiled. Regeneration is not
-needed.
-
-**Error indications:** Messages may fail to encode or decode, or the code can
-crash inside `load_descriptor_values()` in `pb_common.c`.
-
-Nanopb-0.4.2 (2020-06-23)
-=========================
-
-Generator now uses Python 3 by default
---------------------------------------
-
-**Rationale:** Previously `nanopb-generator.py` had hashbang of
-`#!/usr/bin/env python`, which would execute with Python 2 on most systems.
-Python 2 is now deprecated and many libraries are dropping support for it, which
-makes installing dependencies difficult. While `nanopb_generator.py` has worked
-with Python 3 for years now, and overriding the python version was possible
-with virtualenv, that was an extra complication.
-
-**Changes:** Hashbang now uses `#!/usr/bin/env python3`. New file
-`nanopb_generator.py2` can be used to run with Python 2, if necessary.
-
-**Required actions:** If possible, just verify Python 3 is installed and necessary
-dependencies are installed for it. For example `pip3 install protobuf grpcio-tools`
-should take care of it. If this is not possible, call `nanopb_generator.py2` from
-your build scripts instead.
-
-**Error indications:** `python3: command not found` if Python 3 is not installed.
-`Could not import the Google protobuf Python libraries` if dependencies are only
-installed for Python 2.
-
-Nanopb-0.4.0 (2019-12-20)
-=========================
-
-New field descriptor format
----------------------------
-
-**Rationale:** Previously information about struct fields was stored as an array
-of `pb_field_t` structures. This was a straightforward method, but required
-allocating space for e.g. submessage type and array size for all fields, even
-though most fields are not submessages nor arrays.
-
-**Changes:** Now field information is encoded more efficiently in `uint32_t`
-array in a variable-length format. Old `pb_field_t` structure has been removed
-and it is now a typedef for `pb_field_iter_t`. This retains compatibility with
-most old callback definitions. The field definitions in `.pb.h` files are now
-of type `pb_msgdesc_t`.
-
-**Required actions:** If your own code accesses the low-level field information
-in `pb_field_t`, it must be modified to do so only through the functions declared
-in `pb_common.h`.
-
-**Error indications:** `incompatible pointer type` errors relating to `pb_field_t`
-
-
-Changes to generator default options
-------------------------------------
-
-**Rationale:** Previously nanopb_generator added a timestamp header to generated
-files and used only basename of files in `#include` directives. This is different
-than what the `protoc` C++ backend does.
-
-**Changes:** Now default options are `--no-timestamp` and `--no-strip-path`.
-
-**Required actions:** If old behaviour is desired, add `--timestamp` and
-`--strip-path` options to `nanopb_generator.py` or on `protoc` command line
-as `--nanopb_out=--timestamp,--strip-path:outdir`.
-
-**Error indications:** Compiler error: cannot find include file `mymessage.pb.h`
-when compiling `mymessage.pb.c`.
-
-Removal of bundled plugin.proto
--------------------------------
-
-**Rationale:** Google's Python protobuf library, which is used in nanopb
-generator, has included `plugin_pb2` with it since version 3.1.0. It is
-not necessary to bundle it with nanopb anymore.
-
-**Required actions:** Update `python-protobuf` to version 3.1.0 or newer.
-
-**Error indications:** `ImportError: No module named compiler.plugin_pb2`
-
-.options file is now always case-sensitive
-------------------------------------------
-
-**Rationale:** Previously field names in `.options` file were case-sensitive
-on Linux and case-insensitive on Windows. This was by accident. Because
-`.proto` files are case-sensitive, `.options` files should be too.
-
-**Changes:** Now field names in `.options` are always case-sensitive, and
-matched by `fnmatchcase()` instead of `fnmatch()`.
-
-**Required actions:** If field names in `.options` are not capitalized the
-same as in `.proto`, they must be updated.
-
-*CHAR_BIT* define is now needed
--------------------------------
-**Rationale:** To check whether the platform has 8-bit or larger chars, the
-C standard *CHAR_BIT* macro is needed.
-
-**Changes:** *pb.h* now includes *limits.h* for this macro.
-
-**Required actions:** If your platform doesn't have *limits.h* available, you
-can define the macro in *pb_syshdr.h*. There is an example in *extra* directory.
-
-**Error indications:** "Cannot find include file <limits.h>." or "Undefined
-identifier: CHAR_BIT."
-
-Strings must now always be null-terminated
-------------------------------------------
-**Rationale:** Previously *pb_encode()* would accept non-terminated strings and
-assume that they are the full length of the defined array. However, *pb_decode()*
-would reject such messages because null terminator wouldn't fit in the array.
-
-**Changes:** *pb_encode()* will now return an error if null terminator is missing.
-Maximum encoded message size calculation is changed accordingly so that at most
-*max_size-1* strings are assumed. New field option *max_length* can be used to
-define the maximum string length, instead of the array size.
-
-**Required actions:** If your strings were previously filling the whole allocated
-array, increase the size of the field by 1.
-
-**Error indications:** *pb_encode()* returns error *unterminated string*.
-
-Removal of per-field default value constants
---------------------------------------------
-**Rationale:** Previously nanopb declared a `fieldname_default` constant variable
-for each field with a default value, and used these internally to initialize messages.
-This however used unnecessarily large amount of storage for the values. The variables
-were mostly for internal usage, but were available in the header file.
-
-**Changes:** Default values are now stored as an encoded protobuf message.
-
-**Required actions:** If your code previously used default constants, it will have to
-be adapted to take the default value in some other way, such as by accessing
-`static const MyMessage msg_default = MyMessage_init_default;` and `msg_default.fieldname`.
-
-**Error indications:** Compiler error about `fieldname_default` being undeclared.
-
-Zero tag in message now raises error by default
------------------------------------------------
-**Rationale:** Previously nanopb has allowed messages to be terminated by a null byte,
-which is read as zero tag value. Most other protobuf implementations don't support this,
-so it is not very useful feature. It has also been noted that this can complicate
-debugging issues with corrupted messages.
-
-**Changes:** `pb_decode()` now gives error when it encounters zero tag value. A new
-function `pb_decode_ex()` supports flag `PB_DECODE_NULLTERMINATED` that supports
-decoding null terminated messages.
-
-**Required actions:** If application uses null termination for messages, switch it to
-use `pb_decode_ex()` and `pb_encode_ex()`. If compatibility with 0.3.9.x is needed,
-there are also `pb_decode_nullterminated()` and `pb_encode_nullterminated()` macros,
-which work both in 0.4.0 and 0.3.9.
-
-**Error indications:** Error message from `pb_decode()`: 'zero_tag'.
-
-Submessages now have has_field in proto3 mode
----------------------------------------------
-**Rationale:** Previously nanopb considered proto3 submessages as 'present' only
-when their contents was non-zero. Most other protobuf libraries allow explicit
-'null' state for submessages.
-
-**Changes:** Submessages now have separate `has_field` in proto3 mode also.
-
-**Required actions:** When using submessages in proto3 mode, user code must now
-set `mymsg.has_submsg = true` for each submessage that is present. Alternatively,
-the field option `proto3_singular_msgs` can be used to restore the old behavior.
-
-**Error indications:** Submessages do not get encoded.
-
-PB_OLD_CALLBACK_STYLE option has been removed
----------------------------------------------
-**Rationale:** Back in 2013, function signature for callbacks was changed. The
-`PB_OLD_CALLBACK_STYLE` option allowed compatibility with old code, but
-complicated code and testing because of the different options.
-
-**Changes:** `PB_OLD_CALLBACK_STYLE` option no-longer has any effect.
-
-**Required actions:** If `PB_OLD_CALLBACK_STYLE` option was in use previously,
-function signatures must be updated to use double pointers (`void**` and
-`void * const *`).
-
-**Error indications:** Assignment from incompatible pointer type.
-
-protoc insertion points are no longer included by default
----------------------------------------------------------
-**Rationale:** Protoc allows including comments in form `@@protoc_insertion_point`
-to identify locations for other plugins to insert their own extra content.
-Previously these were included by default, but they clutter the generated files
-and are rarely used.
-
-**Changes:** Insertion points are now included only when `--protoc-insertion-points`
-option is passed to the generator.
-
-Nanopb-0.3.9.4, 0.4.0 (2019-xx-xx)
-==================================
-
-Fix generation of min/max defines for enum types
-------------------------------------------------
-
-**Rationale:** Nanopb generator makes #defines for enum minimum and maximum
-value. Previously these defines incorrectly had the first and last enum value,
-instead of the actual minimum and maximum. (issue #405)
-
-**Changes:** Minimum define now always has the smallest value, and maximum
-define always has the largest value.
-
-**Required actions:** If these defines are used and enum values in .proto file
-are not defined in ascending order, user code behaviour may change. Check that
-user code doesn't expect the old, incorrect first/last behaviour.
-
-Fix undefined behavior related to bool fields
----------------------------------------------
-
-**Rationale:** In C99, `bool` variables are not allowed to have other values
-than `true` and `false`. Compilers use this fact in optimization, and constructs
-like `int foo = msg.has_field ? 100 : 0` will give unexpected results otherwise.
-Previously nanopb didn't enforce that decoded bool fields had valid values.
-
-**Changes:** Bool fields are now handled separately as `PB_LTYPE_BOOL`. The
-`LTYPE` descriptor numbers for other field types were renumbered.
-
-**Required actions:** Source code files must be recompiled, but regenerating
-`.pb.h`/`.pb.c` files from `.proto` is not required. If user code directly uses
-the nanopb internal field representation (search for `PB_LTYPE_VARINT` in source),
-it may need updating.
-
-Nanopb-0.3.9.1, 0.4.0 (2018-04-14)
-==================================
-
-Fix handling of string and bytes default values
------------------------------------------------
-
-**Rationale:** Previously nanopb didn't properly decode special character
-escapes like \\200 emitted by protoc. This caused these escapes to end up
-verbatim in the default values in .pb.c file.
-
-**Changes:** Escapes are now decoded, and e.g. "\\200" or "\\x80" results in
-{0x80} for bytes field and "\\x80" for string field.
-
-**Required actions:** If code has previously relied on '\\' in default value
-being passed through verbatim, it must now be changed to '\\\\'.
-
-Nanopb-0.3.8 (2017-03-05)
-=========================
-
-Fully drain substreams before closing
--------------------------------------
-
-**Rationale:** If the substream functions were called directly and the caller
-did not completely empty the substring before closing it, the parent stream
-would be put into an incorrect state.
-
-**Changes:** *pb_close_string_substream* can now error and returns a boolean.
-
-**Required actions:** Add error checking onto any call to
-*pb_close_string_substream*.
-
-Change oneof format in .pb.c files
-----------------------------------
-
-**Rationale:** Previously two oneofs in a single message would be erroneously
-handled as part of the same union.
-
-**Changes:** Oneofs fields now use special *PB_DATAOFFSET_UNION* offset type
-in generated .pb.c files to distinguish whether they are the first or following
-field inside an union.
-
-**Required actions:** Regenerate *.pb.c/.pb.h* files with new nanopb version if
-oneofs are used.
-
-Nanopb-0.3.5 (2016-02-13)
-=========================
-
-Add support for platforms without uint8_t
------------------------------------------
-**Rationale:** Some platforms cannot access 8-bit sized values directly, and
-do not define *uint8_t*. Nanopb previously didn't support these platforms.
-
-**Changes:** References to *uint8_t* were replaced with several alternatives,
-one of them being a new *pb_byte_t* typedef. This in turn uses *uint_least8_t*
-which means the smallest available type.
-
-**Required actions:** If your platform does not have a standards-compliant
-*stdint.h*, it may lack the definition for *[u]int_least8_t*. This must be
-added manually, example can be found in *extra/pb_syshdr.h*.
-
-**Error indications:** Compiler error: "unknown type name 'uint_least8_t'".
-
-Nanopb-0.3.2 (2015-01-24)
-=========================
-
-Add support for OneOfs
-----------------------
-**Rationale:** Previously nanopb did not support the *oneof* construct in
-*.proto* files. Those fields were generated as regular *optional* fields.
-
-**Changes:** OneOfs are now generated as C unions. Callback fields are not
-supported inside oneof and generator gives an error.
-
-**Required actions:** The generator option *no_unions* can be used to restore old
-behaviour and to allow callbacks to be used. To use unions, one change is
-needed: use *which_xxxx* field to detect which field is present, instead
-of *has_xxxx*. Compare the value against *MyStruct_myfield_tag*.
-
-**Error indications:** Generator error: "Callback fields inside of oneof are
-not supported". Compiler error: "Message" has no member named "has_xxxx".
-
-Nanopb-0.3.0 (2014-08-26)
-=========================
-
-Separate field iterator logic to pb_common.c
---------------------------------------------
-**Rationale:** Originally, the field iteration logic was simple enough to be
-duplicated in *pb_decode.c* and *pb_encode.c*. New field types have made the
-logic more complex, which required the creation of a new file to contain the
-common functionality.
-
-**Changes:** There is a new file, *pb_common.c*, which must be included in
-builds.
-
-**Required actions:** Add *pb_common.c* to build rules. This file is always
-required. Either *pb_decode.c* or *pb_encode.c* can still be left out if some
-functionality is not needed.
-
-**Error indications:** Linker error: undefined reference to
-*pb_field_iter_begin*, *pb_field_iter_next* or similar.
-
-Change data type of field counts to pb_size_t
----------------------------------------------
-**Rationale:** Often nanopb is used with small arrays, such as 255 items or
-less. Using a full *size_t* field to store the array count wastes memory if
-there are many arrays. There already exists parameters *PB_FIELD_16BIT* and
-*PB_FIELD_32BIT* which tell nanopb what is the maximum size of arrays in use.
-
-**Changes:** Generator will now use *pb_size_t* for the array *_count* fields.
-The size of the type will be controlled by the *PB_FIELD_16BIT* and
-*PB_FIELD_32BIT* compilation time options.
-
-**Required actions:** Regenerate all *.pb.h* files. In some cases casts to the
-*pb_size_t* type may need to be added in the user code when accessing the
-*_count* fields.
-
-**Error indications:** Incorrect data at runtime, crashes. But note that other
-changes in the same version already require regenerating the files and have
-better indications of errors, so this is only an issue for development
-versions.
-
-Renamed some macros and identifiers
------------------------------------
-**Rationale:** Some names in nanopb core were badly chosen and conflicted with
-ISO C99 reserved names or lacked a prefix. While they haven't caused trouble
-so far, it is reasonable to switch to non-conflicting names as these are rarely
-used from user code.
-
-**Changes:** The following identifier names have changed:
-
- * Macros:
-
- * STATIC_ASSERT(x) -> PB_STATIC_ASSERT(x)
- * UNUSED(x) -> PB_UNUSED(x)
-
- * Include guards:
-
- * _PB_filename_ -> PB_filename_INCLUDED
-
- * Structure forward declaration tags:
-
- * _pb_field_t -> pb_field_s
- * _pb_bytes_array_t -> pb_bytes_array_s
- * _pb_callback_t -> pb_callback_s
- * _pb_extension_type_t -> pb_extension_type_s
- * _pb_extension_t -> pb_extension_s
- * _pb_istream_t -> pb_istream_s
- * _pb_ostream_t -> pb_ostream_s
-
-**Required actions:** Regenerate all *.pb.c* files. If you use any of the above
-identifiers in your application code, perform search-replace to the new name.
-
-**Error indications:** Compiler errors on lines with the macro/type names.
-
-Nanopb-0.2.9 (2014-08-09)
-=========================
-
-Change semantics of generator -e option
----------------------------------------
-**Rationale:** Some compilers do not accept filenames with two dots (like
-in default extension .pb.c). The *-e* option to the generator allowed changing
-the extension, but not skipping the extra dot.
-
-**Changes:** The *-e* option in generator will no longer add the prepending
-dot. The default value has been adjusted accordingly to *.pb.c* to keep the
-default behaviour the same as before.
-
-**Required actions:** Only if using the generator -e option. Add dot before
-the parameter value on the command line.
-
-**Error indications:** File not found when trying to compile generated files.
-
-Nanopb-0.2.7 (2014-04-07)
-=========================
-
-Changed pointer-type bytes field datatype
------------------------------------------
-**Rationale:** In the initial pointer encoding support since nanopb-0.2.5,
-the bytes type used a separate *pb_bytes_ptr_t* type to represent *bytes*
-fields. This made it easy to encode data from a separate, user-allocated
-buffer. However, it made the internal logic more complex and was inconsistent
-with the other types.
-
-**Changes:** Dynamically allocated bytes fields now have the *pb_bytes_array_t*
-type, just like statically allocated ones.
-
-**Required actions:** Only if using pointer-type fields with the bytes datatype.
-Change any access to *msg->field.size* to *msg->field->size*. Change any
-allocation to reserve space of amount *PB_BYTES_ARRAY_T_ALLOCSIZE(n)*. If the
-data pointer was begin assigned from external source, implement the field using
-a callback function instead.
-
-**Error indications:** Compiler error: unknown type name *pb_bytes_ptr_t*.
-
-Nanopb-0.2.4 (2013-11-07)
-=========================
-
-Remove the NANOPB_INTERNALS compilation option
-----------------------------------------------
-**Rationale:** Having the option in the headers required the functions to
-be non-static, even if the option is not used. This caused errors on some
-static analysis tools.
-
-**Changes:** The *#ifdef* and associated functions were removed from the
-header.
-
-**Required actions:** Only if the *NANOPB_INTERNALS* option was previously
-used. Actions are as listed under nanopb-0.1.3 and nanopb-0.1.6.
-
-**Error indications:** Compiler warning: implicit declaration of function
-*pb_dec_string*, *pb_enc_string*, or similar.
-
-Nanopb-0.2.1 (2013-04-14)
-=========================
-
-Callback function signature
----------------------------
-**Rationale:** Previously the auxilary data to field callbacks was passed
-as *void\**. This allowed passing of any data, but made it unnecessarily
-complex to return a pointer from callback.
-
-**Changes:** The callback function parameter was changed to *void\*\**.
-
-**Required actions:** You can continue using the old callback style by
-defining *PB_OLD_CALLBACK_STYLE*. Recommended action is to:
-
- * Change the callback signatures to contain *void\*\** for decoders and
- *void \* const \** for encoders.
- * Change the callback function body to use *\*arg* instead of *arg*.
-
-**Error indications:** Compiler warning: assignment from incompatible
-pointer type, when initializing *funcs.encode* or *funcs.decode*.
-
-Nanopb-0.2.0 (2013-03-02)
-=========================
-
-Reformatted generated .pb.c file using macros
----------------------------------------------
-**Rationale:** Previously the generator made a list of C *pb_field_t*
-initializers in the .pb.c file. This led to a need to regenerate all .pb.c
-files after even small changes to the *pb_field_t* definition.
-
-**Changes:** Macros were added to pb.h which allow for cleaner definition
-of the .pb.c contents. By changing the macro definitions, changes to the
-field structure are possible without breaking compatibility with old .pb.c
-files.
-
-**Required actions:** Regenerate all .pb.c files from the .proto sources.
-
-**Error indications:** Compiler warning: implicit declaration of function
-*pb_delta_end*.
-
-Changed pb_type_t definitions
------------------------------
-**Rationale:** The *pb_type_t* was previously an enumeration type. This
-caused warnings on some compilers when using bitwise operations to set flags
-inside the values.
-
-**Changes:** The *pb_type_t* was changed to *typedef uint8_t*. The values
-were changed to *#define*. Some value names were changed for consistency.
-
-**Required actions:** Only if you directly access the `pb_field_t` contents
-in your own code, something which is not usually done. Needed changes:
-
- * Change *PB_HTYPE_ARRAY* to *PB_HTYPE_REPEATED*.
- * Change *PB_HTYPE_CALLBACK* to *PB_ATYPE()* and *PB_ATYPE_CALLBACK*.
-
-**Error indications:** Compiler error: *PB_HTYPE_ARRAY* or *PB_HTYPE_CALLBACK*
-undeclared.
-
-Nanopb-0.1.6 (2012-09-02)
-=========================
-
-Refactored field decoder interface
-----------------------------------
-**Rationale:** Similarly to field encoders in nanopb-0.1.3.
-
-**Changes:** New functions with names *pb_decode_\** were added.
-
-**Required actions:** By defining NANOPB_INTERNALS, you can still keep using
-the old functions. Recommended action is to replace any calls with the newer
-*pb_decode_\** equivalents.
-
-**Error indications:** Compiler warning: implicit declaration of function
-*pb_dec_string*, *pb_dec_varint*, *pb_dec_submessage* or similar.
-
-Nanopb-0.1.3 (2012-06-12)
-=========================
-
-Refactored field encoder interface
-----------------------------------
-**Rationale:** The old *pb_enc_\** functions were designed mostly for the
-internal use by the core. Because they are internally accessed through
-function pointers, their signatures had to be common. This led to a confusing
-interface for external users.
-
-**Changes:** New functions with names *pb_encode_\** were added. These have
-easier to use interfaces. The old functions are now only thin wrappers for
-the new interface.
-
-**Required actions:** By defining NANOPB_INTERNALS, you can still keep using
-the old functions. Recommended action is to replace any calls with the newer
-*pb_encode_\** equivalents.
-
-**Error indications:** Compiler warning: implicit declaration of function
-*pb_enc_string*, *pb_enc_varint, *pb_enc_submessage* or similar.
-
diff --git a/docs/reference.md b/docs/reference.md
new file mode 100644
index 0000000..84e9024
--- /dev/null
+++ b/docs/reference.md
@@ -0,0 +1,1034 @@
+# Nanopb: API reference
+
+## Compilation options
+
+The following options can be specified in one of two ways:
+
+1. Using the -D switch on the C compiler command line.
+2. Using a `#define` at the top of pb.h.
+
+> **NOTE:** You must have the same settings for the nanopb library and all code that
+includes nanopb headers.
+
+* `PB_ENABLE_MALLOC`: Enable dynamic allocation support in the decoder.
+* `PB_MAX_REQUIRED_FIELDS`: Maximum number of proto2 `required` fields to check for presence. Default value is 64. Compiler warning will tell if you need this.
+* `PB_FIELD_32BIT`: Add support for field tag numbers over 65535, fields larger than 64 kiB and arrays larger than 65535 entries. Compiler warning will tell if you need this.
+* `PB_NO_ERRMSG`: Disable error message support to save code size. Only error information is the `true`/`false` return value.
+* `PB_BUFFER_ONLY`: Disable support for custom streams. Only supports encoding and decoding with memory buffers. Speeds up execution and slightly decreases code size.
+* `PB_SYSTEM_HEADER`: Replace the standards header files with a single system-specific header file. Value must include quotes, for example `#define PB_SYSTEM_HEADER "foo.h"`. See [extra/pb_syshdr.h](https://github.com/nanopb/nanopb/blob/master/extra/pb_syshdr.h) for an example.
+* `PB_WITHOUT_64BIT`: Disable support of 64-bit integer fields, for old compilers or for a slight speedup on 8-bit platforms.
+* `PB_ENCODE_ARRAYS_UNPACKED`: Encode scalar arrays in the unpacked format, which takes up more space. Only to be used when the decoder on the receiving side cannot process packed arrays, such as [protobuf.js versions before 2020](https://github.com/protocolbuffers/protobuf/issues/1701).
+* `PB_CONVERT_DOBULE_FLOAT`: Convert doubles to floats for platforms that do not support 64-bit `double` datatype. Mainly `AVR` processors.
+* `PB_VALIDATE_UTF8`: Check whether incoming strings are valid UTF-8 sequences. Adds a small performance and code size penalty.
+
+The `PB_MAX_REQUIRED_FIELDS` and `PB_FIELD_32BIT` settings allow
+raising some datatype limits to suit larger messages. Their need is
+recognized automatically by C-preprocessor `#if`-directives in the
+generated `.pb.c` files. The default setting is to use the smallest
+datatypes (least resources used).
+
+## Proto file options
+
+The generator behaviour can be adjusted using several options, defined
+in the [nanopb.proto](https://github.com/nanopb/nanopb/blob/master/generator/proto/nanopb.proto) file in the generator folder. Here is a list of the most common options, but see the file for a full list:
+
+* `max_size`: Allocated maximum size for `bytes` and `string` fields. For strings, this includes the terminating zero.
+* `max_length`: Maximum length for `string` fields. Setting this is equivalent to setting `max_size` to a value of length + 1.
+* `max_count`: Allocated maximum number of entries in arrays (`repeated` fields).
+* `type`: Select how memory is allocated for the generated field. Default value is `FT_DEFAULT`, which defaults to `FT_STATIC` when possible and `FT_CALLBACK` if not possible. You can use `FT_CALLBACK`, `FT_POINTER`, `FT_STATIC` or `FT_IGNORE` to select a callback field, a dynamically allocate dfield, a statically allocated field or to completely ignore the field.
+* `long_names`: Prefix the enum name to the enum value in definitions, i.e. `EnumName_EnumValue`. Enabled by default.
+* `packed_struct`: Make the generated structures packed, which saves some RAM space but slows down execution. This can only be used if the CPU supports unaligned access to variables.
+* `skip_message`: Skip a whole message from generation. Can be used to remove message types that are not needed in an application.
+* `no_unions`: Generate `oneof` fields as multiple optional fields instead of a C `union {}`.
+* `anonymous_oneof`: Generate `oneof` fields as an anonymous union.
+* `msgid`: Specifies a unique id for this message type. Can be used by user code as an identifier.
+* `fixed_length`: Generate `bytes` fields with a constant length defined by `max_size`. A separate `.size` field will then not be generated.
+* `fixed_count`: Generate arrays with constant length defined by `max_count`.
+* `package`: Package name that applies only for nanopb generator. Defaults to name defined by `package` keyword in .proto file, which applies for all languages.
+* `int_size`: Override the integer type of a field. For example, specify `int_size = IS_8` to convert `int32` from protocol definition into `int8_t` in the structure.
+
+These options can be defined for the .proto files before they are
+converted using the nanopb-generatory.py. There are three ways to define
+the options:
+
+1. Using a separate .options file. This allows using wildcards for
+ applying same options to multiple fields.
+2. Defining the options on the command line of nanopb_generator.py.
+ This only makes sense for settings that apply to a whole file.
+3. Defining the options in the .proto file using the nanopb extensions.
+ This keeps the options close to the fields they apply to, but can be
+ problematic if the same .proto file is shared with many projects.
+
+The effect of the options is the same no matter how they are given. The
+most common purpose is to define maximum size for string fields in order
+to statically allocate them.
+
+### Defining the options in a .options file
+
+The preferred way to define options is to have a separate file
+'myproto.options' in the same directory as the 'myproto.proto'. :
+
+ # myproto.proto
+ message MyMessage {
+ required string name = 1;
+ repeated int32 ids = 4;
+ }
+
+ # myproto.options
+ MyMessage.name max_size:40
+ MyMessage.ids max_count:5
+
+The generator will automatically search for this file and read the
+options from it. The file format is as follows:
+
+- Lines starting with `#` or `//` are regarded as comments.
+- Blank lines are ignored.
+- All other lines should start with a field name pattern, followed by
+ one or more options. For example: `MyMessage.myfield max_size:5 max_count:10`.
+- The field name pattern is matched against a string of form
+ `Message.field`. For nested messages, the string is
+ `Message.SubMessage.field`. A whole file can be matched by its
+ filename `dir/file.proto`.
+- The field name pattern may use the notation recognized by Python
+ fnmatch():
+ - `*` matches any part of string, like `Message.*` for all
+ fields
+ - `?` matches any single character
+ - `[seq]` matches any of characters `s`, `e` and `q`
+ - `[!seq]` matches any other character
+- The options are written as `option_name:option_value` and
+ several options can be defined on same line, separated by
+ whitespace.
+- Options defined later in the file override the ones specified
+ earlier, so it makes sense to define wildcard options first in the
+ file and more specific ones later.
+
+To debug problems in applying the options, you can use the `-v` option
+for the nanopb generator. With protoc, plugin options are specified with
+`--nanopb_opt`:
+
+ nanopb_generator -v message.proto # When invoked directly
+ protoc ... --nanopb_opt=-v --nanopb_out=. message.proto # When invoked through protoc
+
+Protoc doesn't currently pass include path into plugins. Therefore if
+your `.proto` is in a subdirectory, nanopb may have trouble finding the
+associated `.options` file. A workaround is to specify include path
+separately to the nanopb plugin, like:
+
+ protoc -Isubdir --nanopb_opt=-Isubdir --nanopb_out=. message.proto
+
+If preferred, the name of the options file can be set using generator
+argument `-f`.
+
+### Defining the options on command line
+
+The nanopb_generator.py has a simple command line option `-s OPTION:VALUE`.
+The setting applies to the whole file that is being processed.
+
+### Defining the options in the .proto file
+
+The .proto file format allows defining custom options for the fields.
+The nanopb library comes with *nanopb.proto* which does exactly that,
+allowing you do define the options directly in the .proto file:
+
+~~~~ protobuf
+import "nanopb.proto";
+
+message MyMessage {
+ required string name = 1 [(nanopb).max_size = 40];
+ repeated int32 ids = 4 [(nanopb).max_count = 5];
+}
+~~~~
+
+A small complication is that you have to set the include path of protoc
+so that nanopb.proto can be found. Therefore, to compile a .proto file
+which uses options, use a protoc command similar to:
+
+ protoc -Inanopb/generator/proto -I. --nanopb_out=. message.proto
+
+The options can be defined in file, message and field scopes:
+
+~~~~ protobuf
+option (nanopb_fileopt).max_size = 20; // File scope
+message Message
+{
+ option (nanopb_msgopt).max_size = 30; // Message scope
+ required string fieldsize = 1 [(nanopb).max_size = 40]; // Field scope
+}
+~~~~
+
+## pb.h
+
+### pb_byte_t
+
+Type used for storing byte-sized data, such as raw binary input and
+bytes-type fields.
+
+ typedef uint_least8_t pb_byte_t;
+
+For most platforms this is equivalent to `uint8_t`. Some platforms
+however do not support 8-bit variables, and on those platforms 16 or 32
+bits need to be used for each byte.
+
+### pb_size_t
+
+Type used for storing tag numbers and sizes of message fields. By
+default the type is 16-bit:
+
+ typedef uint_least16_t pb_size_t;
+
+If tag numbers or fields larger than 65535 are needed, `PB_FIELD_32BIT`
+option can be used to change the type to 32-bit value.
+
+### pb_type_t
+
+Type used to store the type of each field, to control the
+encoder/decoder behaviour.
+
+ typedef uint_least8_t pb_type_t;
+
+The low-order nibble of the enumeration values defines the function that
+can be used for encoding and decoding the field data:
+
+| LTYPE identifier |Value |Storage format
+| ---------------------------------|-------|------------------------------------------------
+| `PB_LTYPE_BOOL` |0x00 |Boolean.
+| `PB_LTYPE_VARINT` |0x01 |Integer.
+| `PB_LTYPE_UVARINT` |0x02 |Unsigned integer.
+| `PB_LTYPE_SVARINT` |0x03 |Integer, zigzag encoded.
+| `PB_LTYPE_FIXED32` |0x04 |32-bit integer or floating point.
+| `PB_LTYPE_FIXED64` |0x05 |64-bit integer or floating point.
+| `PB_LTYPE_BYTES` |0x06 |Structure with `size_t` field and byte array.
+| `PB_LTYPE_STRING` |0x07 |Null-terminated string.
+| `PB_LTYPE_SUBMESSAGE` |0x08 |Submessage structure.
+| `PB_LTYPE_SUBMSG_W_CB` |0x09 |Submessage with pre-decoding callback.
+| `PB_LTYPE_EXTENSION` |0x0A |Pointer to `pb_extension_t`.
+| `PB_LTYPE_FIXED_LENGTH_BYTES` |0x0B |Inline `pb_byte_t` array of fixed size.
+
+The bits 4-5 define whether the field is required, optional or repeated.
+There are separate definitions for semantically different modes, even
+though some of them share values and are distinguished based on values
+of other fields:
+
+ |HTYPE identifier |Value |Field handling
+ |---------------------|-------|--------------------------------------------------------------------------------------------
+ |`PB_HTYPE_REQUIRED` |0x00 |Verify that field exists in decoded message.
+ |`PB_HTYPE_OPTIONAL` |0x10 |Use separate `has_<field>` boolean to specify whether the field is present.
+ |`PB_HTYPE_SINGULAR` |0x10 |Proto3 field, which is present when its value is non-zero.
+ |`PB_HTYPE_REPEATED` |0x20 |A repeated field with preallocated array. Separate `<field>_count` for number of items.
+ |`PB_HTYPE_FIXARRAY` |0x20 |A repeated field that has constant length.
+ |`PB_HTYPE_ONEOF` |0x30 |Oneof-field, only one of each group can be present.
+
+The bits 6-7 define the how the storage for the field is allocated:
+
+|ATYPE identifier |Value |Allocation method
+|---------------------|-------|--------------------------------------------------------------------------------------------
+|`PB_ATYPE_STATIC` |0x00 |Statically allocated storage in the structure.
+|`PB_ATYPE_POINTER` |0x80 |Dynamically allocated storage. Struct field contains a pointer to the storage.
+|`PB_ATYPE_CALLBACK` |0x40 |A field with dynamic storage size. Struct field contains a pointer to a callback function.
+
+### pb_msgdesc_t
+
+Autogenerated structure that contains information about a message and
+pointers to the field descriptors. Use functions defined in
+`pb_common.h` to process the field information.
+
+ typedef struct pb_msgdesc_s pb_msgdesc_t;
+ struct pb_msgdesc_s {
+ pb_size_t field_count;
+ const uint32_t *field_info;
+ const pb_msgdesc_t * const * submsg_info;
+ const pb_byte_t *default_value;
+
+ bool (*field_callback)(pb_istream_t *istream, pb_ostream_t *ostream, const pb_field_iter_t *field);
+ };
+
+| | |
+|-----------------|--------------------------------------------------------|
+|`field_count` | Total number of fields in the message.
+|`field_info` | Pointer to compact representation of the field information.
+|`submsg_info` | Pointer to array of pointers to descriptors for submessages.
+|`default_value` | Default values for this message as an encoded protobuf message.
+|`field_callback` | Function used to handle all callback fields in this message. By default `pb_default_field_callback()` which loads per-field callbacks from a `pb_callback_t` structure.
+
+### pb_field_iter_t
+
+Describes a single structure field with memory position in relation to
+others. The field information is stored in a compact format and loaded
+into `pb_field_iter_t` by the functions defined in `pb_common.h`.
+
+ typedef struct pb_field_iter_s pb_field_iter_t;
+ struct pb_field_iter_s {
+ const pb_msgdesc_t *descriptor;
+ void *message;
+
+ pb_size_t index;
+ pb_size_t field_info_index;
+ pb_size_t required_field_index;
+ pb_size_t submessage_index;
+
+ pb_size_t tag;
+ pb_size_t data_size;
+ pb_size_t array_size;
+ pb_type_t type;
+
+ void *pField;
+ void *pData;
+ void *pSize;
+
+ const pb_msgdesc_t *submsg_desc;
+ };
+
+| | |
+|----------------------|--------------------------------------------------------|
+| descriptor | Pointer to `pb_msgdesc_t` for the message that contains this field.
+| message | Pointer to the start of the message structure.
+| index | Index of the field inside the message
+| field_info_index | Index to the internal `field_info` array
+| required_field_index | Index that counts only the required fields
+| submessage_index | Index that counts only submessages
+| tag | Tag number defined in `.proto` file for this field.
+| data_size | `sizeof()` of the field in the structure. For repeated fields this is for a single array entry.
+| array_size | Maximum number of items in a statically allocated array.
+| type | Type ([pb_type_t](#pb_type_t)) of the field.
+| pField | Pointer to the field storage in the structure.
+| pData | Pointer to data contents. For arrays and pointers this can be different than `pField`.
+| pSize | Pointer to count or has field, or NULL if this field doesn't have such.
+| submsg_desc | For submessage fields, points to the descriptor for the submessage.
+
+By default [pb_size_t](#pb_size_t) is 16-bit, limiting the sizes and
+tags to 65535. The limit can be raised by defining `PB_FIELD_32BIT`.
+
+### pb_bytes_array_t
+
+An byte array with a field for storing the length:
+
+ typedef struct {
+ pb_size_t size;
+ pb_byte_t bytes[1];
+ } pb_bytes_array_t;
+
+In an actual array, the length of `bytes` may be different. The macros
+`PB_BYTES_ARRAY_T()` and `PB_BYTES_ARRAY_T_ALLOCSIZE()`
+are used to allocate variable length storage for bytes fields.
+
+### pb_callback_t
+
+Part of a message structure, for fields with type PB_HTYPE_CALLBACK:
+
+ typedef struct _pb_callback_t pb_callback_t;
+ struct _pb_callback_t {
+ union {
+ bool (*decode)(pb_istream_t *stream, const pb_field_iter_t *field, void **arg);
+ bool (*encode)(pb_ostream_t *stream, const pb_field_iter_t *field, void * const *arg);
+ } funcs;
+
+ void *arg;
+ };
+
+A pointer to the *arg* is passed to the callback when calling. It can be
+used to store any information that the callback might need. Note that
+this is a double pointer. If you set `field.arg` to point to
+`&data` in your main code, in the callback you can access it like this:
+
+ myfunction(*arg); /* Gives pointer to data as argument */
+ myfunction(*(data_t*)*arg); /* Gives value of data as argument */
+ *arg = newdata; /* Alters value of field.arg in structure */
+
+When calling [pb_encode](#pb_encode), `funcs.encode` is used, and
+similarly when calling [pb_decode](#pb_decode), `funcs.decode` is used.
+The function pointers are stored in the same memory location but are of
+incompatible types. You can set the function pointer to NULL to skip the
+field.
+
+### pb_wire_type_t
+
+Protocol Buffers wire types. These are used with
+[pb_encode_tag](#pb_encode_tag). :
+
+ typedef enum {
+ PB_WT_VARINT = 0,
+ PB_WT_64BIT = 1,
+ PB_WT_STRING = 2,
+ PB_WT_32BIT = 5
+ } pb_wire_type_t;
+
+### pb_extension_type_t
+
+Defines the handler functions and auxiliary data for a field that
+extends another message. Usually autogenerated by
+`nanopb_generator.py`.
+
+ typedef struct {
+ bool (*decode)(pb_istream_t *stream, pb_extension_t *extension,
+ uint32_t tag, pb_wire_type_t wire_type);
+ bool (*encode)(pb_ostream_t *stream, const pb_extension_t *extension);
+ const void *arg;
+ } pb_extension_type_t;
+
+In the normal case, the function pointers are `NULL` and the decoder and
+encoder use their internal implementations. The internal implementations
+assume that `arg` points to a [pb_field_iter_t](#pb_field_iter_t)
+that describes the field in question.
+
+To implement custom processing of unknown fields, you can provide
+pointers to your own functions. Their functionality is mostly the same
+as for normal callback fields, except that they get called for any
+unknown field when decoding.
+
+### pb_extension_t
+
+Ties together the extension field type and the storage for the field
+value. For message structs that have extensions, the generator will
+add a `pb_extension_t*` field. It should point to a linked list of
+extensions.
+
+ typedef struct {
+ const pb_extension_type_t *type;
+ void *dest;
+ pb_extension_t *next;
+ bool found;
+ } pb_extension_t;
+
+| | |
+|----------------------|--------------------------------------------------------|
+| type | Pointer to the structure that defines the callback functions.
+| dest | Pointer to the variable that stores the field value (as used by the default extension callback functions.)
+| next | Pointer to the next extension handler, or `NULL` for last handler.
+| found | Decoder sets this to true if the extension was found.
+
+### PB_GET_ERROR
+
+Get the current error message from a stream, or a placeholder string if
+there is no error message:
+
+ #define PB_GET_ERROR(stream) (string expression)
+
+This should be used for printing errors, for example:
+
+ if (!pb_decode(...))
+ {
+ printf("Decode failed: %s\n", PB_GET_ERROR(stream));
+ }
+
+The macro only returns pointers to constant strings (in code memory), so
+that there is no need to release the returned pointer.
+
+### PB_RETURN_ERROR
+
+Set the error message and return false:
+
+ #define PB_RETURN_ERROR(stream,msg) (sets error and returns false)
+
+This should be used to handle error conditions inside nanopb functions
+and user callback functions:
+
+ if (error_condition)
+ {
+ PB_RETURN_ERROR(stream, "something went wrong");
+ }
+
+The *msg* parameter must be a constant string.
+
+### PB_BIND
+
+This macro generates the [pb_msgdesc_t](#pb_msgdesc_t) and associated
+arrays, based on a list of fields in [X-macro](https://en.wikipedia.org/wiki/X_Macro) format. :
+
+ #define PB_BIND(msgname, structname, width) ...
+
+| | |
+|----------------------|--------------------------------------------------------|
+| msgname | Name of the message type. Expects `msgname_FIELDLIST` macro to exist.
+| structname | Name of the C structure to bind to.
+| width | Number of words per field descriptor, or `AUTO` to use minimum size possible.
+
+This macro is automatically invoked inside the autogenerated `.pb.c`
+files. User code can also call it to bind message types with custom
+structures or class types.
+
+## pb_encode.h
+
+### pb_ostream_from_buffer
+
+Constructs an output stream for writing into a memory buffer. It uses an internal callback that
+stores the pointer in stream `state` field. :
+
+ pb_ostream_t pb_ostream_from_buffer(pb_byte_t *buf, size_t bufsize);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| buf | Memory buffer to write into.
+| bufsize | Maximum number of bytes to write.
+| returns | An output stream.
+
+After writing, you can check `stream.bytes_written` to find out how
+much valid data there is in the buffer. This should be passed as the
+message length on decoding side.
+
+### pb_write
+
+Writes data to an output stream. Always use this function, instead of
+trying to call stream callback manually. :
+
+ bool pb_write(pb_ostream_t *stream, const pb_byte_t *buf, size_t count);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Output stream to write to.
+| buf | Pointer to buffer with the data to be written.
+| count | Number of bytes to write.
+| returns | True on success, false if maximum length is exceeded or an IO error happens.
+
+> **NOTE:** If an error happens, *bytes_written* is not incremented. Depending on
+the callback used, calling pb_write again after it has failed once may
+cause undefined behavior. Nanopb itself never does this, instead it
+returns the error to user application. The builtin
+`pb_ostream_from_buffer` is safe to call again after failed write.
+
+### pb_encode
+
+Encodes the contents of a structure as a protocol buffers message and
+writes it to output stream. :
+
+ bool pb_encode(pb_ostream_t *stream, const pb_msgdesc_t *fields, const void *src_struct);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Output stream to write to.
+| fields | Message descriptor, usually autogenerated.
+| src_struct | Pointer to the message structure. Must match `fields` descriptor.
+| returns | True on success, false on any error condition. Error message is set to `stream->errmsg`.
+
+Normally pb_encode simply walks through the fields description array
+and serializes each field in turn. However, submessages must be
+serialized twice: first to calculate their size and then to actually
+write them to output. This causes some constraints for callback fields,
+which must return the same data on every call.
+
+### pb_encode_ex
+
+Encodes the message, with extended behavior set by flags:
+
+ bool pb_encode_ex(pb_ostream_t *stream, const pb_msgdesc_t *fields, const void *src_struct, unsigned int flags);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Output stream to write to.
+| fields | Message descriptor, usually autogenerated.
+| src_struct | Pointer to the message structure. Must match `fields` descriptor.
+| flags | Extended options, see below.
+| returns | True on success, false on any error condition. Error message is set to `stream->errmsg`.
+
+The options that can be defined are:
+
+* `PB_ENCODE_DELIMITED`: Indicate the length of the message by prefixing with a varint-encoded length. Compatible with `parseDelimitedFrom` in Google's protobuf library.
+* `PB_ENCODE_NULLTERMINATED`: Indicate the length of the message by appending a zero tag value after it. Supported by nanopb decoder, but not by most other protobuf libraries.
+
+### pb_get_encoded_size
+
+Calculates the length of the encoded message.
+
+ bool pb_get_encoded_size(size_t *size, const pb_msgdesc_t *fields, const void *src_struct);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| size | Calculated size of the encoded message.
+| fields | Message descriptor, usually autogenerated.
+| src_struct | Pointer to the data that will be serialized.
+| returns | True on success, false on detectable errors in field description or if a field encoder returns false.
+
+### Callback field encoders
+The functions with names `pb_encode_<datatype>` are used when dealing with
+callback fields. The typical reason for using callbacks is to have an
+array of unlimited size. In that case, [pb_encode](#pb_encode) will
+call your callback function, which in turn will call `pb_encode_<datatype>`
+functions repeatedly to write out values.
+
+The tag of a field must be encoded first with
+[pb_encode_tag_for_field](#pb_encode_tag_for_field). After that, you
+can call exactly one of the content-writing functions to encode the
+payload of the field. For repeated fields, you can repeat this process
+multiple times.
+
+Writing packed arrays is a little bit more involved: you need to use
+`pb_encode_tag` and specify `PB_WT_STRING` as the wire
+type. Then you need to know exactly how much data you are going to
+write, and use [pb_encode_varint](#pb_encode_varint) to write out the
+number of bytes before writing the actual data. Substreams can be used
+to determine the number of bytes beforehand; see
+[pb_encode_submessage](#pb_encode_submessage) source code for an
+example.
+
+See [Google Protobuf Encoding Format Documentation](https://developers.google.com/protocol-buffers/docs/encoding)
+for background information on the Protobuf wire format.
+
+#### pb_encode_tag
+
+Starts a field in the Protocol Buffers binary format: encodes the field
+number and the wire type of the data.
+
+ bool pb_encode_tag(pb_ostream_t *stream, pb_wire_type_t wiretype, uint32_t field_number);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Output stream to write to. 1-5 bytes will be written.
+| wiretype | `PB_WT_VARINT`, `PB_WT_64BIT`, `PB_WT_STRING` or `PB_WT_32BIT`
+| field_number | Identifier for the field, defined in the .proto file. You can get it from `field->tag`.
+| returns | True on success, false on IO error.
+
+#### pb_encode_tag_for_field
+
+Same as [pb_encode_tag](#pb_encode_tag), except takes the parameters
+from a `pb_field_iter_t` structure.
+
+ bool pb_encode_tag_for_field(pb_ostream_t *stream, const pb_field_iter_t *field);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Output stream to write to. 1-5 bytes will be written.
+| field | Field iterator for this field.
+| returns | True on success, false on IO error or unknown field type.
+
+This function only considers the `PB_LTYPE` of the field. You can use it from
+your field callbacks, because the source generator writes correct `LTYPE`
+also for callback type fields.
+
+Wire type mapping is as follows:
+
+| LTYPEs | Wire type
+|--------------------------------------------------|-----------------
+| BOOL, VARINT, UVARINT, SVARINT | PB_WT_VARINT
+| FIXED64 | PB_WT_64BIT
+| STRING, BYTES, SUBMESSAGE, FIXED_LENGTH_BYTES | PB_WT_STRING
+| FIXED32 | PB_WT_32BIT
+
+#### pb_encode_varint
+
+Encodes a signed or unsigned integer in the
+[varint](http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints)
+format. Works for fields of type `bool`, `enum`, `int32`, `int64`, `uint32` and `uint64`:
+
+ bool pb_encode_varint(pb_ostream_t *stream, uint64_t value);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Output stream to write to. 1-10 bytes will be written.
+| value | Value to encode, cast to `uint64_t`.
+| returns | True on success, false on IO error.
+
+> **NOTE:** Value will be converted to `uint64_t` in the argument.
+> To encode signed values, the argument should be cast to `int64_t` first for correct sign extension.
+
+#### pb_encode_svarint
+
+Encodes a signed integer in the [zig-zagged](https://developers.google.com/protocol-buffers/docs/encoding#signed_integers) format.
+Works for fields of type `sint32` and `sint64`:
+
+ bool pb_encode_svarint(pb_ostream_t *stream, int64_t value);
+
+(parameters are the same as for [pb_encode_varint](#pb_encode_varint)
+
+#### pb_encode_string
+
+Writes the length of a string as varint and then contents of the string.
+Works for fields of type `bytes` and `string`:
+
+ bool pb_encode_string(pb_ostream_t *stream, const pb_byte_t *buffer, size_t size);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Output stream to write to.
+| buffer | Pointer to string data.
+| size | Number of bytes in the string. Pass `strlen(s)` for strings.
+| returns | True on success, false on IO error.
+
+#### pb_encode_fixed32
+
+Writes 4 bytes to stream and swaps bytes on big-endian architectures.
+Works for fields of type `fixed32`, `sfixed32` and `float`:
+
+ bool pb_encode_fixed32(pb_ostream_t *stream, const void *value);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Output stream to write to. 4 bytes will be written.
+| value | Pointer to a 4-bytes large C variable, for example `uint32_t foo;`.
+| returns | True on success, false on IO error.
+
+#### pb_encode_fixed64
+
+Writes 8 bytes to stream and swaps bytes on big-endian architecture.
+Works for fields of type `fixed64`, `sfixed64` and `double`:
+
+ bool pb_encode_fixed64(pb_ostream_t *stream, const void *value);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Output stream to write to. 8 bytes will be written.
+| value | Pointer to a 8-bytes large C variable, for example `uint64_t foo;`.
+| returns | True on success, false on IO error.
+
+#### pb_encode_float_as_double
+
+Encodes a 32-bit `float` value so that it appears like a 64-bit `double` in the encoded message.
+This is sometimes needed when platforms like AVR that do not support 64-bit `double` need to communicate using a
+message type that contains `double` fields.
+
+ bool pb_encode_float_as_double(pb_ostream_t *stream, float value);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Output stream to write to. 8 bytes will be written.
+| value | Float value to encode.
+| returns | True on success, false on IO error.
+
+#### pb_encode_submessage
+
+Encodes a submessage field, including the size header for it. Works for
+fields of any message type.
+
+ bool pb_encode_submessage(pb_ostream_t *stream, const pb_msgdesc_t *fields, const void *src_struct);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Output stream to write to.
+| fields | Pointer to the autogenerated message descriptor for the submessage type, e.g. `MyMessage_fields`.
+| src | Pointer to the structure where submessage data is.
+| returns | True on success, false on IO errors, pb_encode errors or if submessage size changes between calls.
+
+In Protocol Buffers format, the submessage size must be written before
+the submessage contents. Therefore, this function has to encode the
+submessage twice in order to know the size beforehand.
+
+If the submessage contains callback fields, the callback function might
+misbehave and write out a different amount of data on the second call.
+This situation is recognized and `false` is returned, but garbage will
+be written to the output before the problem is detected.
+
+## pb_decode.h
+
+### pb_istream_from_buffer
+
+Helper function for creating an input stream that reads data from a
+memory buffer.
+
+ pb_istream_t pb_istream_from_buffer(const pb_byte_t *buf, size_t bufsize);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| buf | Pointer to byte array to read from.
+| bufsize | Size of the byte array.
+| returns | An input stream ready to use.
+
+### pb_read
+
+Read data from input stream. Always use this function, don't try to
+call the stream callback directly.
+
+ bool pb_read(pb_istream_t *stream, pb_byte_t *buf, size_t count);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Input stream to read from.
+| buf | Buffer to store the data to, or `NULL` to just read data without storing it anywhere.
+| count | Number of bytes to read.
+| returns | True on success, false if `stream->bytes_left` is less than `count` or if an IO error occurs.
+
+End of file is signalled by `stream->bytes_left` being zero after pb_read returns false.
+
+### pb_decode
+
+Read and decode all fields of a structure. Reads until EOF on input
+stream.
+
+ bool pb_decode(pb_istream_t *stream, const pb_msgdesc_t *fields, void *dest_struct);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Input stream to read from.
+| fields | Message descriptor, usually autogenerated.
+| dest_struct | Pointer to message structure where data will be stored.
+| returns | True on success, false on any error condition. Error message will be in `stream->errmsg`.
+
+In Protocol Buffers binary format, end-of-file is only allowed between fields.
+If it happens anywhere else, pb_decode will return `false`. If
+pb_decode returns `false`, you cannot trust any of the data in the
+structure.
+
+For optional fields, this function applies the default value and sets
+`has_<field>` to false if the field is not present.
+
+If `PB_ENABLE_MALLOC` is defined, this function may allocate storage
+for any pointer type fields. In this case, you have to call
+[pb_release](#pb_release) to release the memory after you are done with
+the message. On error return `pb_decode` will release the memory itself.
+
+### pb_decode_ex
+
+Same as [pb_decode](#pb_decode), but allows extended options.
+
+ bool pb_decode_ex(pb_istream_t *stream, const pb_msgdesc_t *fields, void *dest_struct, unsigned int flags);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Input stream to read from.
+| fields | Message descriptor, usually autogenerated.
+| dest_struct | Pointer to message structure where data will be stored.
+| flags | Extended options, see below
+| returns | True on success, false on any error condition. Error message will be in `stream->errmsg`.
+
+The following options can be defined and combined with bitwise `|` operator:
+
+* `PB_DECODE_NOINIT`: Do not initialize structure before decoding. This can be used to combine multiple messages, or if you have already initialized the message structure yourself.
+
+* `PB_DECODE_DELIMITED`: Expect a length prefix in varint format before message. The counterpart of `PB_ENCODE_DELIMITED`.
+
+* `PB_DECODE_NULLTERMINATED`: Expect the message to be terminated with zero tag. The counterpart of `PB_ENCODE_NULLTERMINATED`.
+
+If `PB_ENABLE_MALLOC` is defined, this function may allocate storage
+for any pointer type fields. In this case, you have to call
+[pb_release](#pb_release) to release the memory after you are done with
+the message. On error return `pb_decode_ex` will release the memory
+itself.
+
+### pb_release
+
+Releases any dynamically allocated fields:
+
+ void pb_release(const pb_msgdesc_t *fields, void *dest_struct);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| fields | Message descriptor, usually autogenerated.
+| dest_struct | Pointer to structure where data is stored. If `NULL`, function does nothing.
+
+This function is only available if `PB_ENABLE_MALLOC` is defined. It
+will release any pointer type fields in the structure and set the
+pointers to `NULL`.
+
+This function is safe to call multiple times, calling it again does nothing.
+
+### pb_decode_tag
+
+Decode the tag that comes before field in the protobuf encoding:
+
+ bool pb_decode_tag(pb_istream_t *stream, pb_wire_type_t *wire_type, uint32_t *tag, bool *eof);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Input stream to read from.
+| wire_type | Pointer to variable where to store the wire type of the field.
+| tag | Pointer to variable where to store the tag of the field.
+| eof | Pointer to variable where to store end-of-file status.
+| returns | True on success, false on error or EOF.
+
+When the message (stream) ends, this function will return `false` and set
+`eof` to true. On other errors, `eof` will be set to false.
+
+### pb_skip_field
+
+Remove the data for a field from the stream, without actually decoding it:
+
+ bool pb_skip_field(pb_istream_t *stream, pb_wire_type_t wire_type);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Input stream to read from.
+| wire_type | Type of field to skip.
+| returns | True on success, false on IO error.
+
+This function determines the amount of bytes to read based on the wire type.
+For `PB_WT_STRING`, it will read the length prefix of a string or submessage
+to determine its length.
+
+### Callback field decoders
+The functions with names `pb_decode_<datatype>` are used when dealing with callback fields.
+The typical reason for using callbacks is to have an array of unlimited size.
+In that case, [pb_decode](#pb_decode) will call your callback function repeatedly,
+which can then store the values into e.g. filesystem in the order received in.
+
+For decoding numeric (including enumerated and boolean) values, use
+[pb_decode_varint](#pb_decode_varint), [pb_decode_svarint](#pb_decode_svarint),
+[pb_decode_fixed32](#pb_decode_fixed32) and [pb_decode_fixed64](#pb_decode_fixed64).
+They take a pointer to a 32- or 64-bit C variable, which you may then cast to smaller datatype for storage.
+
+For decoding strings and bytes fields, the length has already been decoded and the callback function is given a length-limited substream.
+You can therefore check the total length in `stream->bytes_left` and read the data using [pb_read](#pb_read).
+
+Finally, for decoding submessages in a callback, use [pb_decode](#pb_decode) and pass it the `SubMessage_fields` descriptor array.
+
+#### pb_decode_varint
+
+Read and decode a [varint](http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints)
+encoded integer.
+
+ bool pb_decode_varint(pb_istream_t *stream, uint64_t *dest);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Input stream to read from. 1-10 bytes will be read.
+| dest | Storage for the decoded integer. Value is undefined on error.
+| returns | True on success, false if value exceeds uint64_t range or an IO error happens.
+
+#### pb_decode_varint32
+
+Same as `pb_decode_varint`, but limits the value to 32 bits:
+
+ bool pb_decode_varint32(pb_istream_t *stream, uint32_t *dest);
+
+Parameters are the same as `pb_decode_varint`. This function can be used
+for decoding lengths and other commonly occurring elements that you know
+shouldn't be larger than 32 bit. It will return an error if the value
+exceeds the `uint32_t` datatype.
+
+#### pb_decode_svarint
+
+Similar to [pb_decode_varint](#pb_decode_varint), except that it
+performs zigzag-decoding on the value. This corresponds to the Protocol
+Buffers `sint32` and `sint64` datatypes. :
+
+ bool pb_decode_svarint(pb_istream_t *stream, int64_t *dest);
+
+(parameters are the same as [pb_decode_varint](#pb_decode_varint))
+
+#### pb_decode_fixed32
+
+Decode a `fixed32`, `sfixed32` or `float` value.
+
+ bool pb_decode_fixed32(pb_istream_t *stream, void *dest);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Input stream to read from. 4 bytes will be read.
+| dest | Pointer to destination `int32_t`, `uint32_t` or `float`.
+| returns | True on success, false on IO errors.
+
+This function reads 4 bytes from the input stream. On big endian
+architectures, it then reverses the order of the bytes. Finally, it
+writes the bytes to `dest`.
+
+#### pb_decode_fixed64
+
+Decode a `fixed64`, `sfixed64` or `double` value. :
+
+ bool pb_decode_fixed64(pb_istream_t *stream, void *dest);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Input stream to read from. 8 bytes will be read.
+| dest | Pointer to destination `int64_t`, `uint64_t` or `double`.
+| returns | True on success, false on IO errors.
+
+Same as [pb_decode_fixed32](#pb_decode_fixed32), except this reads 8
+bytes.
+
+#### pb_decode_double_as_float
+
+Decodes a 64-bit `double` value into a 32-bit `float`
+variable. Counterpart of [pb_encode_float_as_double](#pb_encode_float_as_double). :
+
+ bool pb_decode_double_as_float(pb_istream_t *stream, float *dest);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Input stream to read from. 8 bytes will be read.
+| dest | Pointer to destination *float*.
+| returns | True on success, false on IO errors.
+
+#### pb_make_string_substream
+
+Decode the length for a field with wire type `PB_WT_STRING` and create
+a substream for reading the data.
+
+ bool pb_make_string_substream(pb_istream_t *stream, pb_istream_t *substream);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Original input stream to read the length and data from.
+| substream | Storage for a new substream that has limited length. Filled in by the function.
+| returns | True on success, false if reading the length fails.
+
+This function uses `pb_decode_varint` to read an integer from the stream.
+This is interpreted as a number of bytes, and the substream is set up so that its `bytes_left` is initially the same as the
+length, and its callback function and state the same as the parent stream.
+
+#### pb_close_string_substream
+
+Close the substream created with
+[pb_make_string_substream](#pb_make_string_substream).
+
+ void pb_close_string_substream(pb_istream_t *stream, pb_istream_t *substream);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| stream | Original input stream to read data from.
+| substream | Substream to close
+
+This function copies back the state from the substream to the parent stream,
+and throws away any unread data from the substream.
+It must be called after done with the substream.
+
+## pb_common.h
+
+### pb_field_iter_begin
+
+Begins iterating over the fields in a message type:
+
+ bool pb_field_iter_begin(pb_field_iter_t *iter, const pb_msgdesc_t *desc, void *message);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| iter | Pointer to destination [pb_field_iter_t](#pb_field_iter_t) variable.
+| desc | Autogenerated message descriptor.
+| message | Pointer to message structure.
+| returns | True on success, false if the message type has no fields.
+
+### pb_field_iter_next
+
+Advance to the next field in the message:
+
+ bool pb_field_iter_next(pb_field_iter_t *iter);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| iter | Pointer to `pb_field_iter_t` previously initialized by [pb_field_iter_begin](#pb_field_iter_begin).
+| returns | True on success, false after last field in the message.
+
+When the last field in the message has been processed, this function
+will return false and initialize `iter` back to the first field in the
+message.
+
+### pb_field_iter_find
+
+Find a field specified by tag number in the message:
+
+ bool pb_field_iter_find(pb_field_iter_t *iter, uint32_t tag);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| iter | Pointer to `pb_field_iter_t` previously initialized by [pb_field_iter_begin](#pb_field_iter_begin).
+| tag | Tag number to search for.
+| returns | True if field was found, false otherwise.
+
+This function is functionally identical to calling `pb_field_iter_next()` until `iter.tag` equals the searched value.
+Internally this function avoids fully processing the descriptor for intermediate fields.
+
+### pb_validate_utf8
+
+Validates an UTF8 encoded string:
+
+ bool pb_validate_utf8(const char *s);
+
+| | |
+|----------------------|--------------------------------------------------------|
+| s | Pointer to beginning of a string.
+| returns | True, if string is valid UTF-8, false otherwise.
+
+The protobuf standard requires that `string` fields only contain valid
+UTF-8 encoded text, while `bytes` fields can contain arbitrary data.
+When the compilation option `PB_VALIDATE_UTF8` is defined, nanopb will
+automatically validate strings on both encoding and decoding.
+
+User code can call this function to validate strings in e.g. custom
+callbacks.
diff --git a/docs/reference.rst b/docs/reference.rst
deleted file mode 100644
index 6e47f78..0000000
--- a/docs/reference.rst
+++ /dev/null
@@ -1,954 +0,0 @@
-=====================
-Nanopb: API reference
-=====================
-
-.. include :: menu.rst
-
-.. contents ::
-
-
-Compilation options
-===================
-The following options can be specified in one of two ways:
-
-1. Using the -D switch on the C compiler command line.
-2. By #defining them at the top of pb.h.
-
-You must have the same settings for the nanopb library and all code that
-includes pb.h.
-
-============================ ================================================
-PB_NO_PACKED_STRUCTS Disable packed structs. Increases RAM usage but
- is necessary on some platforms that do not
- support unaligned memory access.
-PB_ENABLE_MALLOC Set this to enable dynamic allocation support
- in the decoder.
-PB_MAX_REQUIRED_FIELDS Maximum number of required fields to check for
- presence. Default value is 64. Increases stack
- usage 1 byte per every 8 fields. Compiler
- warning will tell if you need this.
-PB_FIELD_32BIT Add support for tag numbers > 65535 and fields
- larger than 65535 bytes or 65535 array entries.
- Compiler error will tell if you need this.
-PB_NO_ERRMSG Disables the support for error messages; only
- error information is the true/false return
- value. Decreases the code size by a few hundred
- bytes.
-PB_BUFFER_ONLY Disables the support for custom streams. Only
- supports encoding and decoding with memory
- buffers. Speeds up execution and decreases code
- size slightly.
-PB_SYSTEM_HEADER Replace the standard header files with a single
- header file. It should define all the required
- functions and typedefs listed on the
- `overview page`_. Value must include quotes,
- for example *#define PB_SYSTEM_HEADER "foo.h"*.
-PB_WITHOUT_64BIT Disable 64-bit support, for old compilers or
- for a slight speedup on 8-bit platforms.
-PB_ENCODE_ARRAYS_UNPACKED Don't encode scalar arrays as packed.
- This is only to be used when the decoder on the
- receiving side cannot process packed scalar
- arrays. Such example is older protobuf.js.
-PB_CONVERT_DOUBLE_FLOAT Convert doubles to floats for platforms that do
- not support 64-bit doubles. Mainly AVR.
-PB_VALIDATE_UTF8 Check whether incoming strings are valid UTF-8
- sequences. Slows down the string processing
- slightly and slightly increases code size.
-============================ ================================================
-
-The PB_MAX_REQUIRED_FIELDS and PB_FIELD_32BIT settings allow
-raising some datatype limits to suit larger messages. Their need is recognized
-automatically by C-preprocessor #if-directives in the generated `.pb.c` files.
-The default setting is to use the smallest datatypes (least resources used).
-
-.. _`overview page`: index.html#compiler-requirements
-
-
-Proto file options
-==================
-The generator behaviour can be adjusted using several options, defined in the
-`nanopb.proto`_ file in the generator folder. Here is a list of the most common
-options, but see the file for a full list:
-
-.. _`nanopb.proto`: https://github.com/nanopb/nanopb/blob/master/generator/proto/nanopb.proto
-
-============================ ================================================
-max_size Allocated size for *bytes* and *string* fields.
-max_count Allocated number of entries in arrays
- (*repeated* fields).
-int_size Override the integer type of a field.
- (To use e.g. uint8_t to save RAM.)
-type Type of the generated field. Default value
- is *FT_DEFAULT*, which selects automatically.
- You can use *FT_CALLBACK*, *FT_POINTER*,
- *FT_STATIC* or *FT_IGNORE* to
- force a callback field, a dynamically
- allocated field, a static field or to
- completely ignore the field.
-long_names Prefix the enum name to the enum value in
- definitions, i.e. *EnumName_EnumValue*. Enabled
- by default.
-packed_struct Make the generated structures packed.
- NOTE: This cannot be used on CPUs that break
- on unaligned accesses to variables.
-skip_message Skip the whole message from generation.
-no_unions Generate 'oneof' fields as optional fields
- instead of C unions.
-msgid Specifies a unique id for this message type.
- Can be used by user code as an identifier.
-anonymous_oneof Generate 'oneof' fields as anonymous unions.
-fixed_length Generate 'bytes' fields with constant length
- (max_size must also be defined).
-fixed_count Generate arrays with constant length
- (max_count must also be defined).
-package Package name that applies only for nanopb
- generator, as opposed to the .proto file
- *package* keyword that applies for all languages.
-============================ ================================================
-
-These options can be defined for the .proto files before they are converted
-using the nanopb-generatory.py. There are three ways to define the options:
-
-1. Using a separate .options file. This allows using wildcards for applying
- same options to multiple fields.
-2. Defining the options on the command line of nanopb_generator.py.
- This only makes sense for settings that apply to a whole file.
-3. Defining the options in the .proto file using the nanopb extensions.
- This keeps the options close to the fields they apply to, but can be
- problematic if the same .proto file is shared with many projects.
-
-The effect of the options is the same no matter how they are given. The most
-common purpose is to define maximum size for string fields in order to
-statically allocate them.
-
-Defining the options in a .options file
----------------------------------------
-The preferred way to define options is to have a separate file
-'myproto.options' in the same directory as the 'myproto.proto'. ::
-
- # myproto.proto
- message MyMessage {
- required string name = 1;
- repeated int32 ids = 4;
- }
-
-::
-
- # myproto.options
- MyMessage.name max_size:40
- MyMessage.ids max_count:5
-
-The generator will automatically search for this file and read the
-options from it. The file format is as follows:
-
-* Lines starting with '#' or '//' are regarded as comments.
-* Blank lines are ignored.
-* All other lines should start with a field name pattern, followed by one or
- more options. For example: *"MyMessage.myfield max_size:5 max_count:10"*.
-* The field name pattern is matched against a string of form *'Message.field'*.
- For nested messages, the string is *'Message.SubMessage.field'*.
- A whole file can be matched by its filename *'dir/file.proto'*.
-* The field name pattern may use the notation recognized by Python fnmatch():
-
- - *\** matches any part of string, like 'Message.\*' for all fields
- - *\?* matches any single character
- - *[seq]* matches any of characters 's', 'e' and 'q'
- - *[!seq]* matches any other character
-
-* The options are written as *'option_name:option_value'* and several options
- can be defined on same line, separated by whitespace.
-* Options defined later in the file override the ones specified earlier, so
- it makes sense to define wildcard options first in the file and more specific
- ones later.
-
-To debug problems in applying the options, you can use the *-v* option for the
-nanopb generator. With protoc, plugin options are specified with *--nanopb_opt*::
-
- nanopb_generator -v message.proto # When invoked directly
- protoc ... --nanopb_opt=-v --nanopb_out=. message.proto # When invoked through protoc
-
-Protoc doesn't currently pass include path into plugins. Therefore if your
-*.proto* is in a subdirectory, nanopb may have trouble finding the associated
-*.options* file. A workaround is to specify include path separately to the
-nanopb plugin, like::
-
- protoc -Isubdir --nanopb_opt=-Isubdir --nanopb_out=. message.proto
-
-If preferred, the name of the options file can be set using generator argument
-*-f*.
-
-Defining the options on command line
-------------------------------------
-The nanopb_generator.py has a simple command line option *-s OPTION:VALUE*.
-The setting applies to the whole file that is being processed.
-
-Defining the options in the .proto file
----------------------------------------
-The .proto file format allows defining custom options for the fields.
-The nanopb library comes with *nanopb.proto* which does exactly that, allowing
-you do define the options directly in the .proto file::
-
- import "nanopb.proto";
-
- message MyMessage {
- required string name = 1 [(nanopb).max_size = 40];
- repeated int32 ids = 4 [(nanopb).max_count = 5];
- }
-
-A small complication is that you have to set the include path of protoc so that
-nanopb.proto can be found. Therefore, to compile a .proto file which uses options, use a
-protoc command similar to::
-
- protoc -Inanopb/generator/proto -I. --nanopb_out=. message.proto
-
-The options can be defined in file, message and field scopes::
-
- option (nanopb_fileopt).max_size = 20; // File scope
- message Message
- {
- option (nanopb_msgopt).max_size = 30; // Message scope
- required string fieldsize = 1 [(nanopb).max_size = 40]; // Field scope
- }
-
-
-pb.h
-====
-
-pb_byte_t
----------
-Type used for storing byte-sized data, such as raw binary input and bytes-type fields. ::
-
- typedef uint_least8_t pb_byte_t;
-
-For most platforms this is equivalent to `uint8_t`. Some platforms however do not support
-8-bit variables, and on those platforms 16 or 32 bits need to be used for each byte.
-
-pb_size_t
----------
-Type used for storing tag numbers and sizes of message fields. By default the type is 16-bit::
-
- typedef uint_least16_t pb_size_t;
-
-If tag numbers or fields larger than 65535 are needed, `PB_FIELD_32BIT` option
-can be used to change the type to 32-bit value.
-
-pb_type_t
----------
-Type used to store the type of each field, to control the encoder/decoder behaviour. ::
-
- typedef uint_least8_t pb_type_t;
-
-The low-order nibble of the enumeration values defines the function that can be used for encoding and decoding the field data:
-
-=========================== ===== ================================================
-LTYPE identifier Value Storage format
-=========================== ===== ================================================
-PB_LTYPE_BOOL 0x00 Boolean.
-PB_LTYPE_VARINT 0x01 Integer.
-PB_LTYPE_UVARINT 0x02 Unsigned integer.
-PB_LTYPE_SVARINT 0x03 Integer, zigzag encoded.
-PB_LTYPE_FIXED32 0x04 32-bit integer or floating point.
-PB_LTYPE_FIXED64 0x05 64-bit integer or floating point.
-PB_LTYPE_BYTES 0x06 Structure with *size_t* field and byte array.
-PB_LTYPE_STRING 0x07 Null-terminated string.
-PB_LTYPE_SUBMESSAGE 0x08 Submessage structure.
-PB_LTYPE_SUBMSG_W_CB 0x09 Submessage with pre-decoding callback.
-PB_LTYPE_EXTENSION 0x0A Point to *pb_extension_t*.
-PB_LTYPE_FIXED_LENGTH_BYTES 0x0B Inline *pb_byte_t* array of fixed size.
-=========================== ===== ================================================
-
-The bits 4-5 define whether the field is required, optional or repeated.
-There are separate definitions for semantically different modes, even though
-some of them share values and are distinguished based on values of other fields:
-
-==================== ===== ================================================
-HTYPE identifier Value Field handling
-==================== ===== ================================================
-PB_HTYPE_REQUIRED 0x00 Verify that field exists in decoded message.
-PB_HTYPE_OPTIONAL 0x10 Use separate *has_<field>* boolean to specify
- whether the field is present.
-PB_HTYPE_SINGULAR 0x10 Proto3 field, which is present when its value is
- non-zero.
-PB_HTYPE_REPEATED 0x20 A repeated field with preallocated array.
- Separate *<field>_count* for number of items.
-PB_HTYPE_FIXARRAY 0x20 A repeated field that has constant length.
-PB_HTYPE_ONEOF 0x30 Oneof-field, only one of each group can be present.
-==================== ===== ================================================
-
-The bits 6-7 define the how the storage for the field is allocated:
-
-==================== ===== ================================================
-ATYPE identifier Value Allocation method
-==================== ===== ================================================
-PB_ATYPE_STATIC 0x00 Statically allocated storage in the structure.
-PB_ATYPE_POINTER 0x80 Dynamically allocated storage. Struct field contains
- a pointer to the storage.
-PB_ATYPE_CALLBACK 0x40 A field with dynamic storage size. Struct field
- contains a pointer to a callback function.
-==================== ===== ================================================
-
-
-pb_msgdesc_t
-------------
-Autogenerated structure that contains information about a message and pointers
-to the field descriptors. Use functions defined in `pb_common.h` to process
-the field information::
-
- typedef struct pb_msgdesc_s pb_msgdesc_t;
- struct pb_msgdesc_s {
- pb_size_t field_count;
- const uint32_t *field_info;
- const pb_msgdesc_t * const * submsg_info;
- const pb_byte_t *default_value;
-
- bool (*field_callback)(pb_istream_t *istream, pb_ostream_t *ostream, const pb_field_iter_t *field);
- };
-
-:field_count: Total number of fields in the message.
-:field_info: Pointer to compact representation of the field information.
-:submsg_info: Pointer to array of pointers to descriptors for submessages.
-:default_value: Default values for this message as an encoded protobuf message.
-:field_callback: Function used to handle all callback fields in this message.
- By default `pb_default_field_callback()` loads per-field
- callbacks from a `pb_callback_t` structure.
-
-
-pb_field_iter_t
----------------
-Describes a single structure field with memory position in relation to others.
-The field information is stored in a compact format and loaded into `pb_field_iter_t`
-by the functions defined in `pb_common.h`. ::
-
- typedef struct pb_field_iter_s pb_field_iter_t;
- struct pb_field_iter_s {
- const pb_msgdesc_t *descriptor;
- void *message;
-
- pb_size_t index;
- pb_size_t field_info_index;
- pb_size_t required_field_index;
- pb_size_t submessage_index;
-
- pb_size_t tag;
- pb_size_t data_size;
- pb_size_t array_size;
- pb_type_t type;
-
- void *pField;
- void *pData;
- void *pSize;
-
- const pb_msgdesc_t *submsg_desc;
- };
-
-:descriptor: Pointer to `pb_msgdesc_t` for the message that contains this field.
-:message: Pointer to the start of the message structure.
-:index: Index of the field inside the message
-:field_info_index: Index to the internal `field_info` array
-:required_field_index: Index that counts only the required fields
-:submessage_index: Index that counts only submessages
-:tag: Tag number defined in `.proto` file for this field.
-:data_size: `sizeof()` of the field in the structure. For repeated fields this is for a single array entry.
-:array_size: Maximum number of items in a statically allocated array.
-:type: Type (`pb_type_t`_) of the field.
-:pField: Pointer to the field storage in the structure.
-:pData: Pointer to data contents. For arrays and pointers this can be different than `pField`.
-:pSize: Pointer to count or has field, or NULL if this field doesn't have such.
-:submsg_desc: For submessage fields, points to the descriptor for the submessage.
-
-By default `pb_size_t`_ is 16-bit, limiting the sizes and tags to 65535. The limit
-can be raised by defining `PB_FIELD_32BIT`.
-
-pb_bytes_array_t
-----------------
-An byte array with a field for storing the length::
-
- typedef struct {
- pb_size_t size;
- pb_byte_t bytes[1];
- } pb_bytes_array_t;
-
-In an actual array, the length of *bytes* may be different. The macros
-`PB_BYTES_ARRAY_T()` and `PB_BYTES_ARRAY_T_ALLOCSIZE()` are used to allocate
-variable length storage for bytes fields.
-
-pb_callback_t
--------------
-Part of a message structure, for fields with type PB_HTYPE_CALLBACK::
-
- typedef struct _pb_callback_t pb_callback_t;
- struct _pb_callback_t {
- union {
- bool (*decode)(pb_istream_t *stream, const pb_field_iter_t *field, void **arg);
- bool (*encode)(pb_ostream_t *stream, const pb_field_iter_t *field, void * const *arg);
- } funcs;
-
- void *arg;
- };
-
-A pointer to the *arg* is passed to the callback when calling.
-It can be used to store any information that the callback might need.
-Note that this is a double pointer. If you set `field.arg` to point to `&data` in your
-main code, in the callback you can access it like this::
-
- myfunction(*arg); /* Gives pointer to data as argument */
- myfunction(*(data_t*)*arg); /* Gives value of data as argument */
- *arg = newdata; /* Alters value of field.arg in structure */
-
-When calling `pb_encode`_, *funcs.encode* is used, and similarly when calling
-`pb_decode`_, *funcs.decode* is used. The function pointers are stored in the
-same memory location but are of incompatible types.
-You can set the function pointer to NULL to skip the field.
-
-pb_wire_type_t
---------------
-Protocol Buffers wire types. These are used with `pb_encode_tag`_. ::
-
- typedef enum {
- PB_WT_VARINT = 0,
- PB_WT_64BIT = 1,
- PB_WT_STRING = 2,
- PB_WT_32BIT = 5
- } pb_wire_type_t;
-
-pb_extension_type_t
--------------------
-Defines the handler functions and auxiliary data for a field that extends
-another message. Usually autogenerated by *nanopb_generator.py*::
-
- typedef struct {
- bool (*decode)(pb_istream_t *stream, pb_extension_t *extension,
- uint32_t tag, pb_wire_type_t wire_type);
- bool (*encode)(pb_ostream_t *stream, const pb_extension_t *extension);
- const void *arg;
- } pb_extension_type_t;
-
-In the normal case, the function pointers are *NULL* and the decoder and
-encoder use their internal implementations. The internal implementations
-assume that *arg* points to a `pb_field_iter_t`_ that describes the field in question.
-
-To implement custom processing of unknown fields, you can provide pointers
-to your own functions. Their functionality is mostly the same as for normal
-callback fields, except that they get called for any unknown field when decoding.
-
-pb_extension_t
---------------
-Ties together the extension field type and the storage for the field value::
-
- typedef struct {
- const pb_extension_type_t *type;
- void *dest;
- pb_extension_t *next;
- bool found;
- } pb_extension_t;
-
-:type: Pointer to the structure that defines the callback functions.
-:dest: Pointer to the variable that stores the field value
- (as used by the default extension callback functions.)
-:next: Pointer to the next extension handler, or *NULL*.
-:found: Decoder sets this to true if the extension was found.
-
-PB_GET_ERROR
-------------
-Get the current error message from a stream, or a placeholder string if
-there is no error message::
-
- #define PB_GET_ERROR(stream) (string expression)
-
-This should be used for printing errors, for example::
-
- if (!pb_decode(...))
- {
- printf("Decode failed: %s\n", PB_GET_ERROR(stream));
- }
-
-The macro only returns pointers to constant strings (in code memory),
-so that there is no need to release the returned pointer.
-
-PB_RETURN_ERROR
----------------
-Set the error message and return false::
-
- #define PB_RETURN_ERROR(stream,msg) (sets error and returns false)
-
-This should be used to handle error conditions inside nanopb functions
-and user callback functions::
-
- if (error_condition)
- {
- PB_RETURN_ERROR(stream, "something went wrong");
- }
-
-The *msg* parameter must be a constant string.
-
-PB_BIND
--------
-This macro generates the `pb_msgdesc_t`_ and associated arrays, based on a list
-of fields in `X-macro`_ format. ::
-
- #define PB_BIND(msgname, structname, width) ...
-
-:msgname: Name of the message type. Expects `msgname_FIELDLIST` macro to exist.
-:structname: Name of the C structure to bind to.
-:width: Number of words per field descriptor, or `AUTO` to use minimum size possible.
-
-This macro is automatically invoked inside the autogenerated `.pb.c` files.
-User code can also call it to bind message types with custom structures or class types.
-
-.. _`X-macro`: https://en.wikipedia.org/wiki/X_Macro
-
-
-pb_encode.h
-===========
-
-pb_ostream_from_buffer
-----------------------
-Constructs an output stream for writing into a memory buffer. This is just a helper function, it doesn't do anything you couldn't do yourself in a callback function. It uses an internal callback that stores the pointer in stream *state* field. ::
-
- pb_ostream_t pb_ostream_from_buffer(pb_byte_t *buf, size_t bufsize);
-
-:buf: Memory buffer to write into.
-:bufsize: Maximum number of bytes to write.
-:returns: An output stream.
-
-After writing, you can check *stream.bytes_written* to find out how much valid data there is in the buffer.
-
-pb_write
---------
-Writes data to an output stream. Always use this function, instead of trying to call stream callback manually. ::
-
- bool pb_write(pb_ostream_t *stream, const pb_byte_t *buf, size_t count);
-
-:stream: Output stream to write to.
-:buf: Pointer to buffer with the data to be written.
-:count: Number of bytes to write.
-:returns: True on success, false if maximum length is exceeded or an IO error happens.
-
-If an error happens, *bytes_written* is not incremented. Depending on the callback used, calling pb_write again after it has failed once may be dangerous. Nanopb itself never does this, instead it returns the error to user application. The builtin pb_ostream_from_buffer is safe to call again after failed write.
-
-pb_encode
----------
-Encodes the contents of a structure as a protocol buffers message and writes it to output stream. ::
-
- bool pb_encode(pb_ostream_t *stream, const pb_msgdesc_t *fields, const void *src_struct);
-
-:stream: Output stream to write to.
-:fields: Message descriptor, usually autogenerated.
-:src_struct: Pointer to the data that will be serialized.
-:returns: True on success, false on IO error, on detectable errors in field description, or if a field encoder returns false.
-
-Normally pb_encode simply walks through the fields description array and serializes each field in turn. However, submessages must be serialized twice: first to calculate their size and then to actually write them to output. This causes some constraints for callback fields, which must return the same data on every call.
-
-pb_encode_ex
--------------------
-Encodes the message, with several extended options::
-
- bool pb_encode_ex(pb_ostream_t *stream, const pb_msgdesc_t *fields, const void *src_struct, unsigned int flags);
-
-:stream: Output stream to write to.
-:fields: Message descriptor, usually autogenerated.
-:src_struct: Pointer to the data that will be serialized.
-:flags: Extended options, see below.
-:returns: True on success, false on IO error, on detectable errors in field description, or if a field encoder returns false.
-
-The options that can be defined are:
-
-:PB_ENCODE_DELIMITED: Indicate the length of the message by prefixing with a varint-encoded length. Compatible with *parseDelimitedFrom* in Google's protobuf library.
-:PB_ENCODE_NULLTERMINATED: Indicate the length of the message by appending a zero tag value after it. Supported by nanopb decoder, but not by most other protobuf libraries.
-
-pb_get_encoded_size
--------------------
-Calculates the length of the encoded message. ::
-
- bool pb_get_encoded_size(size_t *size, const pb_msgdesc_t *fields, const void *src_struct);
-
-:size: Calculated size of the encoded message.
-:fields: Message descriptor, usually autogenerated.
-:src_struct: Pointer to the data that will be serialized.
-:returns: True on success, false on detectable errors in field description or if a field encoder returns false.
-
-.. sidebar:: Encoding fields manually
-
- The functions with names *pb_encode_\** are used when dealing with callback fields. The typical reason for using callbacks is to have an array of unlimited size. In that case, `pb_encode`_ will call your callback function, which in turn will call *pb_encode_\** functions repeatedly to write out values.
-
- The tag of a field must be encoded separately with `pb_encode_tag_for_field`_. After that, you can call exactly one of the content-writing functions to encode the payload of the field. For repeated fields, you can repeat this process multiple times.
-
- Writing packed arrays is a little bit more involved: you need to use `pb_encode_tag` and specify `PB_WT_STRING` as the wire type. Then you need to know exactly how much data you are going to write, and use `pb_encode_varint`_ to write out the number of bytes before writing the actual data. Substreams can be used to determine the number of bytes beforehand; see `pb_encode_submessage`_ source code for an example.
-
-pb_encode_tag
--------------
-Starts a field in the Protocol Buffers binary format: encodes the field number and the wire type of the data. ::
-
- bool pb_encode_tag(pb_ostream_t *stream, pb_wire_type_t wiretype, uint32_t field_number);
-
-:stream: Output stream to write to. 1-5 bytes will be written.
-:wiretype: PB_WT_VARINT, PB_WT_64BIT, PB_WT_STRING or PB_WT_32BIT
-:field_number: Identifier for the field, defined in the .proto file. You can get it from field->tag.
-:returns: True on success, false on IO error.
-
-pb_encode_tag_for_field
------------------------
-Same as `pb_encode_tag`_, except takes the parameters from a *pb_field_iter_t* structure. ::
-
- bool pb_encode_tag_for_field(pb_ostream_t *stream, const pb_field_iter_t *field);
-
-:stream: Output stream to write to. 1-5 bytes will be written.
-:field: Field iterator for this field.
-:returns: True on success, false on IO error or unknown field type.
-
-This function only considers the LTYPE of the field. You can use it from your field callbacks, because the source generator writes correct LTYPE also for callback type fields.
-
-Wire type mapping is as follows:
-
-============================================= ============
-LTYPEs Wire type
-============================================= ============
-VARINT, UVARINT, SVARINT PB_WT_VARINT
-FIXED64 PB_WT_64BIT
-STRING, BYTES, SUBMESSAGE, FIXED_LENGTH_BYTES PB_WT_STRING
-FIXED32 PB_WT_32BIT
-============================================= ============
-
-pb_encode_varint
-----------------
-Encodes a signed or unsigned integer in the varint_ format. Works for fields of type `bool`, `enum`, `int32`, `int64`, `uint32` and `uint64`::
-
- bool pb_encode_varint(pb_ostream_t *stream, uint64_t value);
-
-:stream: Output stream to write to. 1-10 bytes will be written.
-:value: Value to encode. Just cast e.g. int32_t directly to uint64_t.
-:returns: True on success, false on IO error.
-
-.. _varint: http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints
-
-pb_encode_svarint
------------------
-Encodes a signed integer in the 'zig-zagged' format. Works for fields of type `sint32` and `sint64`::
-
- bool pb_encode_svarint(pb_ostream_t *stream, int64_t value);
-
-(parameters are the same as for `pb_encode_varint`_
-
-pb_encode_string
-----------------
-Writes the length of a string as varint and then contents of the string. Works for fields of type `bytes` and `string`::
-
- bool pb_encode_string(pb_ostream_t *stream, const pb_byte_t *buffer, size_t size);
-
-:stream: Output stream to write to.
-:buffer: Pointer to string data.
-:size: Number of bytes in the string. Pass `strlen(s)` for strings.
-:returns: True on success, false on IO error.
-
-pb_encode_fixed32
------------------
-Writes 4 bytes to stream and swaps bytes on big-endian architectures. Works for fields of type `fixed32`, `sfixed32` and `float`::
-
- bool pb_encode_fixed32(pb_ostream_t *stream, const void *value);
-
-:stream: Output stream to write to.
-:value: Pointer to a 4-bytes large C variable, for example `uint32_t foo;`.
-:returns: True on success, false on IO error.
-
-pb_encode_fixed64
------------------
-Writes 8 bytes to stream and swaps bytes on big-endian architecture. Works for fields of type `fixed64`, `sfixed64` and `double`::
-
- bool pb_encode_fixed64(pb_ostream_t *stream, const void *value);
-
-:stream: Output stream to write to.
-:value: Pointer to a 8-bytes large C variable, for example `uint64_t foo;`.
-:returns: True on success, false on IO error.
-
-pb_encode_float_as_double
--------------------------
-Encodes a 32-bit `float` value so that it appears like a 64-bit `double` in the
-encoded message. This is sometimes needed when platforms like AVR that do not
-support need to communicate using a message type that contains `double` fields. ::
-
- bool pb_encode_float_as_double(pb_ostream_t *stream, float value);
-
-:stream: Output stream to write to.
-:value: Float value to encode.
-:returns: True on success, false on IO error.
-
-pb_encode_submessage
---------------------
-Encodes a submessage field, including the size header for it. Works for fields of any message type::
-
- bool pb_encode_submessage(pb_ostream_t *stream, const pb_msgdesc_t *fields, const void *src_struct);
-
-:stream: Output stream to write to.
-:fields: Pointer to the autogenerated message descriptor for the submessage type, e.g. `MyMessage_fields`.
-:src: Pointer to the structure where submessage data is.
-:returns: True on success, false on IO errors, pb_encode errors or if submessage size changes between calls.
-
-In Protocol Buffers format, the submessage size must be written before the submessage contents. Therefore, this function has to encode the submessage twice in order to know the size beforehand.
-
-If the submessage contains callback fields, the callback function might misbehave and write out a different amount of data on the second call. This situation is recognized and *false* is returned, but garbage will be written to the output before the problem is detected.
-
-
-
-
-
-pb_decode.h
-===========
-
-pb_istream_from_buffer
-----------------------
-Helper function for creating an input stream that reads data from a memory buffer. ::
-
- pb_istream_t pb_istream_from_buffer(const pb_byte_t *buf, size_t bufsize);
-
-:buf: Pointer to byte array to read from.
-:bufsize: Size of the byte array.
-:returns: An input stream ready to use.
-
-pb_read
--------
-Read data from input stream. Always use this function, don't try to call the stream callback directly. ::
-
- bool pb_read(pb_istream_t *stream, pb_byte_t *buf, size_t count);
-
-:stream: Input stream to read from.
-:buf: Buffer to store the data to, or NULL to just read data without storing it anywhere.
-:count: Number of bytes to read.
-:returns: True on success, false if *stream->bytes_left* is less than *count* or if an IO error occurs.
-
-End of file is signalled by *stream->bytes_left* being zero after pb_read returns false.
-
-pb_decode
----------
-Read and decode all fields of a structure. Reads until EOF on input stream. ::
-
- bool pb_decode(pb_istream_t *stream, const pb_msgdesc_t *fields, void *dest_struct);
-
-:stream: Input stream to read from.
-:fields: Message descriptor, usually autogenerated.
-:dest_struct: Pointer to structure where data will be stored.
-:returns: True on success, false on IO error, on detectable errors in field description, if a field encoder returns false or if a required field is missing.
-
-In Protocol Buffers binary format, EOF is only allowed between fields. If it happens anywhere else, pb_decode will return *false*. If pb_decode returns false, you cannot trust any of the data in the structure.
-
-For optional fields, this function applies the default value and sets *has_<field>* to false if the field is not present.
-
-If *PB_ENABLE_MALLOC* is defined, this function may allocate storage for any pointer type fields.
-In this case, you have to call `pb_release`_ to release the memory after you are done with the message.
-On error return `pb_decode` will release the memory itself.
-
-pb_decode_ex
-------------
-Same as `pb_decode`_, but allows extended options. ::
-
- bool pb_decode_ex(pb_istream_t *stream, const pb_msgdesc_t *fields, void *dest_struct, unsigned int flags);
-
-:stream: Input stream to read from.
-:fields: Message descriptor, usually autogenerated.
-:dest_struct: Pointer to structure where data will be stored.
-:flags: Extended options, see below
-:returns: True on success, false on IO error, on detectable errors in field description, if a field encoder returns false or if a required field is missing.
-
-The following options can be defined and combined with bitwise `|` operator:
-
-:PB_DECODE_NOINIT: Do not initialize structure before decoding. This can be used to combine multiple messages, or if you have already initialized the message yourself.
-:PB_DECODE_DELIMITED: Expect a length prefix in varint format before message. The counterpart of `PB_ENCODE_DELIMITED`.
-:PB_DECODE_NULLTERMINATED: Expect the message to be terminated with zero tag. The counterpart of `PB_ENCODE_NULLTERMINATED`.
-
-If *PB_ENABLE_MALLOC* is defined, this function may allocate storage for any pointer type fields.
-In this case, you have to call `pb_release`_ to release the memory after you are done with the message.
-On error return `pb_decode_ex` will release the memory itself.
-
-pb_release
-----------
-Releases any dynamically allocated fields::
-
- void pb_release(const pb_msgdesc_t *fields, void *dest_struct);
-
-:fields: Message descriptor, usually autogenerated.
-:dest_struct: Pointer to structure where data is stored. If NULL, function does nothing.
-
-This function is only available if *PB_ENABLE_MALLOC* is defined. It will release any
-pointer type fields in the structure and set the pointers to NULL.
-
-pb_decode_tag
--------------
-Decode the tag that comes before field in the protobuf encoding::
-
- bool pb_decode_tag(pb_istream_t *stream, pb_wire_type_t *wire_type, uint32_t *tag, bool *eof);
-
-:stream: Input stream to read from.
-:wire_type: Pointer to variable where to store the wire type of the field.
-:tag: Pointer to variable where to store the tag of the field.
-:eof: Pointer to variable where to store end-of-file status.
-:returns: True on success, false on error or EOF.
-
-When the message (stream) ends, this function will return false and set *eof* to true. On other
-errors, *eof* will be set to false.
-
-pb_skip_field
--------------
-Remove the data for a field from the stream, without actually decoding it::
-
- bool pb_skip_field(pb_istream_t *stream, pb_wire_type_t wire_type);
-
-:stream: Input stream to read from.
-:wire_type: Type of field to skip.
-:returns: True on success, false on IO error.
-
-.. sidebar:: Decoding fields manually
-
- The functions with names beginning with *pb_decode_* are used when dealing with callback fields. The typical reason for using callbacks is to have an array of unlimited size. In that case, `pb_decode`_ will call your callback function repeatedly, which can then store the values into e.g. filesystem in the order received in.
-
- For decoding numeric (including enumerated and boolean) values, use `pb_decode_varint`_, `pb_decode_svarint`_, `pb_decode_fixed32`_ and `pb_decode_fixed64`_. They take a pointer to a 32- or 64-bit C variable, which you may then cast to smaller datatype for storage.
-
- For decoding strings and bytes fields, the length has already been decoded. You can therefore check the total length in *stream->bytes_left* and read the data using `pb_read`_.
-
- Finally, for decoding submessages in a callback, simply use `pb_decode`_ and pass it the *SubMessage_fields* descriptor array.
-
-pb_decode_varint
-----------------
-Read and decode a varint_ encoded integer. ::
-
- bool pb_decode_varint(pb_istream_t *stream, uint64_t *dest);
-
-:stream: Input stream to read from. 1-10 bytes will be read.
-:dest: Storage for the decoded integer. Value is undefined on error.
-:returns: True on success, false if value exceeds uint64_t range or an IO error happens.
-
-pb_decode_varint32
-------------------
-Same as `pb_decode_varint`, but limits the value to 32 bits::
-
- bool pb_decode_varint32(pb_istream_t *stream, uint32_t *dest);
-
-Parameters are the same as `pb_decode_varint`. This function can be used for
-decoding lengths and other commonly occurring elements that you know shouldn't
-be larger than 32 bit. It will return an error if the value exceeds the `uint32_t`
-datatype.
-
-pb_decode_svarint
------------------
-Similar to `pb_decode_varint`_, except that it performs zigzag-decoding on the value. This corresponds to the Protocol Buffers *sint32* and *sint64* datatypes. ::
-
- bool pb_decode_svarint(pb_istream_t *stream, int64_t *dest);
-
-(parameters are the same as `pb_decode_varint`_)
-
-pb_decode_fixed32
------------------
-Decode a *fixed32*, *sfixed32* or *float* value. ::
-
- bool pb_decode_fixed32(pb_istream_t *stream, void *dest);
-
-:stream: Input stream to read from. 4 bytes will be read.
-:dest: Pointer to destination *int32_t*, *uint32_t* or *float*.
-:returns: True on success, false on IO errors.
-
-This function reads 4 bytes from the input stream.
-On big endian architectures, it then reverses the order of the bytes.
-Finally, it writes the bytes to *dest*.
-
-pb_decode_fixed64
------------------
-Decode a *fixed64*, *sfixed64* or *double* value. ::
-
- bool pb_decode_fixed64(pb_istream_t *stream, void *dest);
-
-:stream: Input stream to read from. 8 bytes will be read.
-:dest: Pointer to destination *int64_t*, *uint64_t* or *double*.
-:returns: True on success, false on IO errors.
-
-Same as `pb_decode_fixed32`_, except this reads 8 bytes.
-
-pb_decode_double_as_float
--------------------------
-Decodes a 64-bit `double` value into a 32-bit `float` variable.
-Counterpart of `pb_encode_float_as_double`_. ::
-
- bool pb_decode_double_as_float(pb_istream_t *stream, float *dest);
-
-:stream: Input stream to read from. 8 bytes will be read.
-:dest: Pointer to destination *float*.
-:returns: True on success, false on IO errors.
-
-pb_make_string_substream
-------------------------
-Decode the length for a field with wire type *PB_WT_STRING* and create a substream for reading the data. ::
-
- bool pb_make_string_substream(pb_istream_t *stream, pb_istream_t *substream);
-
-:stream: Original input stream to read the length and data from.
-:substream: New substream that has limited length. Filled in by the function.
-:returns: True on success, false if reading the length fails.
-
-This function uses `pb_decode_varint`_ to read an integer from the stream. This is interpreted as a number of bytes, and the substream is set up so that its `bytes_left` is initially the same as the length, and its callback function and state the same as the parent stream.
-
-pb_close_string_substream
--------------------------
-Close the substream created with `pb_make_string_substream`_. ::
-
- void pb_close_string_substream(pb_istream_t *stream, pb_istream_t *substream);
-
-:stream: Original input stream to read the length and data from.
-:substream: Substream to close
-
-This function copies back the state from the substream to the parent stream.
-It must be called after done with the substream.
-
-
-
-pb_common.h
-===========
-
-pb_field_iter_begin
--------------------
-Begins iterating over the fields in a message type::
-
- bool pb_field_iter_begin(pb_field_iter_t *iter, const pb_msgdesc_t *desc, void *message);
-
-:iter: Pointer to destination `pb_field_iter_t`_ variable.
-:desc: Autogenerated message descriptor.
-:message: Pointer to message structure.
-:returns: True on success, false if the message type has no fields.
-
-pb_field_iter_next
-------------------
-Advance to the next field in the message::
-
- bool pb_field_iter_next(pb_field_iter_t *iter);
-
-:iter: Pointer to `pb_field_iter_t`_ previously initialized by `pb_field_iter_begin`_.
-:returns: True on success, false after last field in the message.
-
-When the last field in the message has been processed, this function will return
-false and initialize `iter` back to the first field in the message.
-
-pb_field_iter_find
-------------------
-Find a field specified by tag number in the message::
-
- bool pb_field_iter_find(pb_field_iter_t *iter, uint32_t tag);
-
-:iter: Pointer to `pb_field_iter_t`_ previously initialized by `pb_field_iter_begin`_.
-:tag: Tag number to search for.
-:returns: True if field was found, false otherwise.
-
-This function is functionally identical to calling `pb_field_iter_next()` until
-`iter.tag` equals the searched value. Internally this function avoids fully
-processing the descriptor for intermediate fields.
-
-pb_validate_utf8
-----------------
-Validates an UTF8 encoded string::
-
- bool pb_validate_utf8(const char *s);
-
-:s: Pointer to beginning of a string.
-:returns: True, if string is valid UTF-8, false otherwise.
-
-The protobuf standard requires that `string` fields only contain valid UTF-8
-encoded text, while `bytes` fields can contain arbitrary data. When the
-compilation option `PB_VALIDATE_UTF8` is defined, nanopb will automatically
-validate strings on both encoding and decoding.
-
-User code can call this function to validate strings in e.g. custom callbacks.
diff --git a/docs/security.md b/docs/security.md
new file mode 100644
index 0000000..044a533
--- /dev/null
+++ b/docs/security.md
@@ -0,0 +1,89 @@
+# Nanopb: Security model
+
+Importance of security in a Protocol Buffers library
+----------------------------------------------------
+
+In the context of protocol buffers, security comes into play when
+decoding untrusted data. Naturally, if the attacker can modify the
+contents of a protocol buffers message, he can feed the application any
+values possible. Therefore the application itself must be prepared to
+receive untrusted values.
+
+Where nanopb plays a part is preventing the attacker from running
+arbitrary code on the target system. Mostly this means that there must
+not be any possibility to cause buffer overruns, memory corruption or
+invalid pointers by the means of crafting a malicious message.
+
+Division of trusted and untrusted data
+--------------------------------------
+
+The following data is regarded as **trusted**. It must be under the
+control of the application writer. Malicious data in these structures
+could cause security issues, such as execution of arbitrary code:
+
+1. Callback, pointer and extension fields in message structures given
+ to pb_encode() and pb_decode(). These fields are memory pointers,
+ and are generated depending on the message definition in the .proto
+ file.
+2. The automatically generated field definitions, i.e.
+ `pb_msgdesc_t`.
+3. Contents of the `pb_istream_t` and `pb_ostream_t` structures
+ (this does not mean the contents of the stream itself, just the
+ stream definition).
+
+The following data is regarded as **untrusted**. Invalid/malicious data
+in these will cause "garbage in, garbage out" behaviour. It will not
+cause buffer overflows, information disclosure or other security
+problems:
+
+1. All data read from `pb_istream_t`.
+2. All fields in message structures, except:
+ - callbacks (`pb_callback_t` structures)
+ - pointer fields and `_count` fields for pointers
+ - extensions (`pb_extension_t` structures)
+
+Invariants
+----------
+
+The following invariants are maintained during operation, even if the
+untrusted data has been maliciously crafted:
+
+1. Nanopb will never read more than `bytes_left` bytes from
+ `pb_istream_t`.
+2. Nanopb will never write more than `max_size` bytes to
+ `pb_ostream_t`.
+3. Nanopb will never access memory out of bounds of the message
+ structure.
+4. After `pb_decode()` returns successfully, the message structure will
+ be internally consistent:
+ - The `count` fields of arrays will not exceed the array size.
+ - The `size` field of bytes will not exceed the allocated size.
+ - All string fields will have null terminator.
+ - bool fields will have valid true/false values (since
+ nanopb-0.3.9.4)
+5. After `pb_encode()` returns successfully, the resulting message is a
+ valid protocol buffers message. (Except if user-defined callbacks
+ write incorrect data.)
+
+Further considerations
+----------------------
+
+Even if the nanopb library is free of any security issues, there are
+still several possible attack vectors that the application author must
+consider. The following list is not comprehensive:
+
+1. Stack usage may depend on the contents of the message. The message
+ definition places an upper bound on how much stack will be used.
+ Tests should be run with all fields present, to record the maximum
+ possible stack usage.
+2. Callbacks can do anything. The code for the callbacks must be
+ carefully checked if they are used with untrusted data.
+3. If using stream input, a maximum size should be set in
+ `pb_istream_t` to stop a denial of service attack from using an
+ infinite message.
+4. If using network sockets as streams, a timeout should be set to stop
+ denial of service attacks.
+5. If using `malloc()` support, some method of limiting memory use
+ should be employed. This can be done by defining custom
+ `pb_realloc()` function. Nanopb will properly detect and handle
+ failed memory allocations.
diff --git a/docs/security.rst b/docs/security.rst
deleted file mode 100644
index ddc3587..0000000
--- a/docs/security.rst
+++ /dev/null
@@ -1,85 +0,0 @@
-======================
-Nanopb: Security model
-======================
-
-.. include :: menu.rst
-
-.. contents ::
-
-
-
-Importance of security in a Protocol Buffers library
-====================================================
-In the context of protocol buffers, security comes into play when decoding
-untrusted data. Naturally, if the attacker can modify the contents of a
-protocol buffers message, he can feed the application any values possible.
-Therefore the application itself must be prepared to receive untrusted values.
-
-Where nanopb plays a part is preventing the attacker from running arbitrary
-code on the target system. Mostly this means that there must not be any
-possibility to cause buffer overruns, memory corruption or invalid pointers
-by the means of crafting a malicious message.
-
-Division of trusted and untrusted data
-======================================
-The following data is regarded as **trusted**. It must be under the control of
-the application writer. Malicious data in these structures could cause
-security issues, such as execution of arbitrary code:
-
-1. Callback, pointer and extension fields in message structures given to
- pb_encode() and pb_decode(). These fields are memory pointers, and are
- generated depending on the message definition in the .proto file.
-2. The automatically generated field definitions, i.e. *pb_msgdesc_t*.
-3. Contents of the *pb_istream_t* and *pb_ostream_t* structures (this does not
- mean the contents of the stream itself, just the stream definition).
-
-The following data is regarded as **untrusted**. Invalid/malicious data in
-these will cause "garbage in, garbage out" behaviour. It will not cause
-buffer overflows, information disclosure or other security problems:
-
-1. All data read from *pb_istream_t*.
-2. All fields in message structures, except:
-
- - callbacks (*pb_callback_t* structures)
- - pointer fields (malloc support) and *_count* fields for pointers
- - extensions (*pb_extension_t* structures)
-
-Invariants
-==========
-The following invariants are maintained during operation, even if the
-untrusted data has been maliciously crafted:
-
-1. Nanopb will never read more than *bytes_left* bytes from *pb_istream_t*.
-2. Nanopb will never write more than *max_size* bytes to *pb_ostream_t*.
-3. Nanopb will never access memory out of bounds of the message structure.
-4. After pb_decode() returns successfully, the message structure will be
- internally consistent:
-
- - The *count* fields of arrays will not exceed the array size.
- - The *size* field of bytes will not exceed the allocated size.
- - All string fields will have null terminator.
- - bool fields will have valid true/false values (since nanopb-0.3.9.4)
-
-5. After pb_encode() returns successfully, the resulting message is a valid
- protocol buffers message. (Except if user-defined callbacks write incorrect
- data.)
-
-Further considerations
-======================
-Even if the nanopb library is free of any security issues, there are still
-several possible attack vectors that the application author must consider.
-The following list is not comprehensive:
-
-1. Stack usage may depend on the contents of the message. The message
- definition places an upper bound on how much stack will be used. Tests
- should be run with all fields present, to record the maximum possible
- stack usage.
-2. Callbacks can do anything. The code for the callbacks must be carefully
- checked if they are used with untrusted data.
-3. If using stream input, a maximum size should be set in *pb_istream_t* to
- stop a denial of service attack from using an infinite message.
-4. If using network sockets as streams, a timeout should be set to stop
- denial of service attacks.
-5. If using *malloc()* support, some method of limiting memory use should be
- employed. This can be done by defining custom *pb_realloc()* function.
- Nanopb will properly detect and handle failed memory allocations.
diff --git a/docs/whats_new.md b/docs/whats_new.md
new file mode 100644
index 0000000..3d942c2
--- /dev/null
+++ b/docs/whats_new.md
@@ -0,0 +1,173 @@
+# Nanopb: New features in nanopb 0.4
+
+## What's new in nanopb 0.4
+
+Long in the making, nanopb 0.4 has seen some wide reaching improvements
+in reaction to the development of the rest of the protobuf ecosystem.
+This document showcases features that are not immediately visible, but
+that you may want to take advantage of.
+
+A lot of effort has been spent in retaining backwards and forwards
+compatibility with previous nanopb versions. For a list of breaking
+changes, see [migration document](migration.html)
+
+### New field descriptor format
+
+The basic design of nanopb has always been that the information about
+messages is stored in a compact descriptor format, which is iterated in
+runtime. Initially it was very tightly tied with encoder and decoder
+logic.
+
+In nanopb-0.3.0 the field iteration logic was separated to
+`pb_common.c`. Already at that point it was clear that the old format
+was getting too limited, but it wasn't extended at that time.
+
+Now in 0.4, the descriptor format was completely decoupled from the
+encoder and decoder logic, and redesigned to meet new demands.
+Previously each field was stored as `pb_field_t` struct, which was
+between 8 and 32 bytes in size, depending on compilation options and
+platform. Now information about fields is stored as a variable length
+sequence of `uint32_t` data words. There are 1, 2, 4 and 8 word formats,
+with the 8 word format containing plenty of space for future
+extensibility.
+
+One benefit of the variable length format is that most messages now take
+less storage space. Most fields use 2 words, while simple fields in
+small messages require only 1 word. Benefit is larger if code previously
+required `PB_FIELD_16BIT` or `PB_FIELD_32BIT` options. In
+the `AllTypes` test case, 0.3 had data size of 1008 bytes in
+8-bit configuration and 1408 bytes in 16-bit configuration. New format
+in 0.4 takes 896 bytes for either of these.
+
+In addition, the new decoupling has allowed moving most of the field
+descriptor data into FLASH on Harvard architectures, such as AVR.
+Previously nanopb was quite RAM-heavy on AVR, which cannot put normal
+constants in flash like most other platforms do.
+
+### Python packaging for generator
+
+Nanopb generator is now available as a Python package, installable using
+`pip` package manager. This will reduce the need for binary
+packages, as if you have Python already installed you can just
+`pip install nanopb` and have the generator available on path as
+`nanopb_generator`.
+
+The generator can also take advantage of the Python-based `protoc`
+available in `grpcio-tools` Python package. If you also install that,
+there is no longer a need to have binary `protoc` available.
+
+### Generator now automatically calls protoc
+
+Initially, nanopb generator was used in two steps: first calling
+`protoc` to parse the `.proto` file into `.pb` binary
+format, and then calling `nanopb_generator.py` to output the
+`.pb.h` and `.pb.c` files.
+
+Nanopb 0.2.3 added support for running as a `protoc` plugin, which
+allowed single-step generation using `--nanopb_out` parameter. However,
+the plugin mode has two complications: passing options to nanopb
+generator itself becomes more difficult, and the generator does not know
+the actual path of input files. The second limitation has been
+particularly problematic for locating `.options` files.
+
+Both of these older methods still work and will remain supported.
+However, now `nanopb_generator` can also take `.proto` files
+directly and it will transparently call `protoc` in the background.
+
+### Callbacks bound by function name
+
+Since its very beginnings, nanopb has supported field callbacks to allow
+processing structures that are larger than what could fit in memory at
+once. So far the callback functions have been stored in the message
+structure in a `pb_callback_t` struct.
+
+Storing pointers along with user data is somewhat risky from a security
+point of view. In addition it has caused problems with `oneof` fields,
+which reuse the same storage space for multiple submessages. Because
+there is no separate area for each submessage, there is no space to
+store the callback pointers either.
+
+Nanopb-0.4.0 introduces callbacks that are referenced by the function
+name instead of setting the pointers separately. This should work well
+for most applications that have a single callback function for each
+message type. For more complex needs, `pb_callback_t` will also remain
+supported.
+
+Function name callbacks also allow specifying custom data types for
+inclusion in the message structure. For example, you could have
+`MyObject*` pointer along with other message fields, and then process
+that object in custom way in your callback.
+
+This feature is demonstrated in
+[tests/oneof_callback](https://github.com/nanopb/nanopb/tree/master/tests/oneof_callback) test case and
+[examples/network_server](https://github.com/nanopb/nanopb/tree/master/examples/network_server) example.
+
+### Message level callback for oneofs
+
+As mentioned above, callbacks inside submessages inside oneofs have been
+problematic to use. To make using `pb_callback_t`-style callbacks there
+possible, a new generator option `submsg_callback` was added.
+
+Setting this option to true will cause a new message level callback to
+be added before the `which_field` of the oneof. This callback will be
+called when the submessage tag number is known, but before the actual
+message is decoded. The callback can either choose to set callback
+pointers inside the submessage, or just completely decode the submessage
+there and then. If any unread data remains after the callback returns,
+normal submessage decoding will continue.
+
+There is an example of this in [tests/oneof_callback](https://github.com/nanopb/nanopb/tree/master/tests/oneof_callback) test case.
+
+### Binding message types to custom structures
+
+It is often said that good C code is chock full of macros. Or maybe I
+got it wrong. But since nanopb 0.2, the field descriptor generation has
+heavily relied on macros. This allows it to automatically adapt to
+differences in type alignment on different platforms, and to decouple
+the Python generation logic from how the message descriptors are
+implemented on the C side.
+
+Now in 0.4.0, I've made the macros even more abstract. Time will tell
+whether this was such a great idea that I think it is, but now the
+complete list of fields in each message is available in `.pb.h` file.
+This allows a kind of metaprogramming using [X-macros]()
+
+One feature that this can be used for is binding the message descriptor
+to a custom structure or C++ class type. You could have a bunch of other
+fields in the structure and even the datatypes can be different to an
+extent, and nanopb will automatically detect the size and position of
+each field. The generated `.pb.c` files now just have calls of
+`PB_BIND(msgname, structname, width)`. Adding a similar
+call to your own code will bind the message to your own structure.
+
+### UTF-8 validation
+
+Protobuf format defines that strings should consist of valid UTF-8
+codepoints. Previously nanopb has not enforced this, requiring extra
+care in the user code. Now optional UTF-8 validation is available with
+compilation option `PB_VALIDATE_UTF8`.
+
+### Double to float conversion
+
+Some platforms such as `AVR` do not support the `double`
+datatype, instead making it an alias for `float`. This has resulted in
+problems when trying to process message types containing `double` fields
+generated on other machines. There has been an example on how to
+manually perform the conversion between `double` and
+`float`.
+
+Now that example is integrated as an optional feature in nanopb core. By
+defining `PB_CONVERT_DOUBLE_FLOAT`, the required conversion between 32-
+and 64-bit floating point formats happens automatically on decoding and
+encoding.
+
+### Improved testing
+
+Testing on embedded platforms has been integrated in the continuous
+testing environment. Now all of the 80+ test cases are automatically run
+on STM32 and AVR targets. Previously only a few specialized test cases
+were manually tested on embedded systems.
+
+Nanopb fuzzer has also been integrated in Google's [OSSFuzz](https://google.github.io/oss-fuzz/)
+platform, giving a huge boost in the CPU power available for randomized
+testing.
diff --git a/docs/whats_new.rst b/docs/whats_new.rst
deleted file mode 100644
index e080c82..0000000
--- a/docs/whats_new.rst
+++ /dev/null
@@ -1,175 +0,0 @@
-====================
-Nanopb: New features
-====================
-
-.. include :: menu.rst
-
-.. contents ::
-
-What's new in nanopb 0.4
-========================
-Long in the making, nanopb 0.4 has seen some wide reaching improvements in
-reaction to the development of the rest of the protobuf ecosystem. This document
-showcases features that are not immediately visible, but that you may want to
-take advantage of.
-
-A lot of effort has been spent in retaining backwards and forwards compatibility
-with previous nanopb versions. For a list of breaking changes, see `migration document`_
-
-.. _`migration document`: migration.html
-
-New field descriptor format
----------------------------
-The basic design of nanopb has always been that the information about messages
-is stored in a compact descriptor format, which is iterated in runtime.
-Initially it was very tightly tied with encoder and decoder logic.
-
-In nanopb-0.3.0 the field iteration logic was separated to `pb_common.c`.
-Already at that point it was clear that the old format was getting too limited,
-but it wasn't extended at that time.
-
-Now in 0.4, the descriptor format was completely decoupled from the encoder
-and decoder logic, and redesigned to meet new demands. Previously each field
-was stored as `pb_field_t` struct, which was between 8 and 32 bytes in size,
-depending on compilation options and platform. Now information about fields is
-stored as a variable length sequence of `uint32_t` data words. There
-are 1, 2, 4 and 8 word formats, with the 8 word format containing plenty of
-space for future extensibility.
-
-One benefit of the variable length format is that most messages now take less
-storage space. Most fields use 2 words, while simple fields in small messages
-require only 1 word. Benefit is larger if code previously required
-`PB_FIELD_16BIT` or `PB_FIELD_32BIT` options. In the `AllTypes` test case, 0.3
-had data size of 1008 bytes in 8-bit configuration and 1408 bytes in 16-bit
-configuration. New format in 0.4 takes 896 bytes for either of these.
-
-In addition, the new decoupling has allowed moving most of the field descriptor
-data into FLASH on Harvard architectures, such as AVR. Previously nanopb was
-quite RAM-heavy on AVR, which cannot put normal constants in flash like most
-other platforms do.
-
-Python packaging for generator
-------------------------------
-Nanopb generator is now available as a Python package, installable using `pip`
-package manager. This will reduce the need for binary packages, as if you have
-Python already installed you can just `pip install nanopb` and have the
-generator available on path as `nanopb_generator`.
-
-The generator can also take advantage of the Python-based `protoc` available in
-`grpcio-tools` Python package. If you also install that, there is no longer
-a need to have binary `protoc` available.
-
-Generator now automatically calls protoc
-----------------------------------------
-Initially, nanopb generator was used in two steps: first calling `protoc` to
-parse the `.proto` file into `.pb` binary format, and then calling `nanopb_generator`
-to output the `.pb.h` and `.pb.c` files.
-
-Nanopb 0.2.3 added support for running as a `protoc` plugin, which allowed
-single-step generation using `--nanopb_out` parameter. However, the plugin
-mode has two complications: passing options to nanopb generator itself becomes
-more difficult, and the generator does not know the actual path of input files.
-The second limitation has been particularly problematic for locating `.options`
-files.
-
-Both of these older methods still work and will remain supported. However, now
-`nanopb_generator` can also take `.proto` files directly and it will transparently
-call `protoc` in the background.
-
-Callbacks bound by function name
---------------------------------
-Since its very beginnings, nanopb has supported field callbacks to allow processing
-structures that are larger than what could fit in memory at once. So far the
-callback functions have been stored in the message structure in a `pb_callback_t`
-struct.
-
-Storing pointers along with user data is somewhat risky from a security point of
-view. In addition it has caused problems with `oneof` fields, which reuse the
-same storage space for multiple submessages. Because there is no separate area
-for each submessage, there is no space to store the callback pointers either.
-
-Nanopb-0.4.0 introduces callbacks that are referenced by the function name
-instead of setting the pointers separately. This should work well for most
-applications that have a single callback function for each message type.
-For more complex needs, `pb_callback_t` will also remain supported.
-
-Function name callbacks also allow specifying custom data types for inclusion
-in the message structure. For example, you could have `MyObject*` pointer along
-with other message fields, and then process that object in custom way in your
-callback.
-
-This feature is demonstrated in `tests/oneof_callback`_ test case and
-`examples/network_server`_ example.
-
-.. _`examples/network_server`: https://github.com/nanopb/nanopb/tree/master/examples/network_server
-
-Message level callback for oneofs
----------------------------------
-As mentioned above, callbacks inside submessages inside oneofs have been
-problematic to use. To make using `pb_callback_t`-style callbacks there possible,
-a new generator option `submsg_callback` was added.
-
-Setting this option to true will cause a new message level callback to be added
-before the `which_field` of the oneof. This callback will be called when the
-submessage tag number is known, but before the actual message is decoded. The
-callback can either choose to set callback pointers inside the submessage, or
-just completely decode the submessage there and then. If any unread data remains
-after the callback returns, normal submessage decoding will continue.
-
-There is an example of this in `tests/oneof_callback`_ test case.
-
-.. _`tests/oneof_callback`: https://github.com/nanopb/nanopb/tree/master/tests/oneof_callback
-
-Binding message types to custom structures
-------------------------------------------
-It is often said that good C code is chock full of macros. Or maybe I got it
-wrong. But since nanopb 0.2, the field descriptor generation has heavily relied
-on macros. This allows it to automatically adapt to differences in type alignment
-on different platforms, and to decouple the Python generation logic from how
-the message descriptors are implemented on the C side.
-
-Now in 0.4.0, I've made the macros even more abstract. Time will tell whether
-this was such a great idea that I think it is, but now the complete list of
-fields in each message is available in `.pb.h` file. This allows a kind of
-metaprogramming using `X-macros`_
-
-.. _`X-macros`: https://en.wikipedia.org/wiki/X_Macro
-
-One feature that this can be used for is binding the message descriptor to a
-custom structure or C++ class type. You could have a bunch of other fields in
-the structure and even the datatypes can be different to an extent, and nanopb
-will automatically detect the size and position of each field. The generated
-`.pb.c` files now just have calls of `PB_BIND(msgname, structname, width)`.
-Adding a similar call to your own code will bind the message to your own structure.
-
-UTF-8 validation
-----------------
-Protobuf format defines that strings should consist of valid UTF-8 codepoints.
-Previously nanopb has not enforced this, requiring extra care in the user code.
-Now optional UTF-8 validation is available with compilation option `PB_VALIDATE_UTF8`.
-
-Double to float conversion
---------------------------
-Some platforms such as `AVR` do not support the `double` datatype, instead making
-it an alias for `float`. This has resulted in problems when trying to process
-message types containing `double` fields generated on other machines. There
-has been an example on how to manually perform the conversion between `double`
-and `float`.
-
-Now that example is integrated as an optional feature in nanopb core. By defining
-`PB_CONVERT_DOUBLE_FLOAT`, the required conversion between 32- and 64-bit floating
-point formats happens automatically on decoding and encoding.
-
-Improved testing
-----------------
-Testing on embedded platforms has been integrated in the continuous testing
-environment. Now all of the 80+ test cases are automatically run on STM32 and
-AVR targets. Previously only a few specialized test cases were manually tested
-on embedded systems.
-
-Nanopb fuzzer has also been integrated in Google's `OSSFuzz`_ platform, giving
-a huge boost in the CPU power available for randomized testing.
-
-.. _`OSSFuzz`: https://google.github.io/oss-fuzz/
-
-
