=====================================
Nanopb: Migration from older versions
=====================================

.. include :: menu.rst

This document details all the breaking changes that have been made to nanopb
since its initial release. For each change, the rationale and required
modifications of user applications are explained. Also any error indications
are included, in order to make it easier to find this document.

.. contents ::

Nanopb-0.4.2 (2020-xx-xx)
=========================

Generator now uses Python 3 by default
--------------------------------------

**Rationale:** Previously `nanopb-generator.py` had hashbang of
`#!/usr/bin/env python`, which would execute with Python 2 on most systems.
Python 2 is now deprecated and many libraries are dropping support for it, which
makes installing dependencies difficult. While `nanopb_generator.py` has worked
with Python 3 for years now, and overriding the python version was possible
with virtualenv, that was an extra complication.

**Changes:** Hashbang now uses `#!/usr/bin/env python3`. New file
`nanopb_generator.py2` can be used to run with Python 2, if necessary.

**Required actions:** If possible, just verify Python 3 is installed and necessary
dependencies are installed for it. For example `pip3 install protobuf grpcio-tools`
should take care of it. If this is not possible, call `nanopb_generator.py2` from
your build scripts instead.

**Error indications:** `python3: command not found` if Python 3 is not installed.
`Could not import the Google protobuf Python libraries` if dependencies are only
installed for Python 2.

Nanopb-0.4.0 (2019-12-20)
=========================

New field descriptor format
---------------------------

**Rationale:** Previously information about struct fields was stored as an array
of `pb_field_t` structures. This was a straightforward method, but required
allocating space for e.g. submessage type and array size for all fields, even
though most fields are not submessages nor arrays.

**Changes:** Now field information is encoded more efficiently in `uint32_t`
array in a variable-length format. Old `pb_field_t` structure has been removed
and it is now a typedef for `pb_field_iter_t`. This retains compatibility with
most old callback definitions. The field definitions in `.pb.h` files are now
of type `pb_msgdesc_t`.

**Required actions:** If your own code accesses the low-level field information
in `pb_field_t`, it must be modified to do so only through the functions declared
in `pb_common.h`.

**Error indications:** `incompatible pointer type` errors relating to `pb_field_t`


Changes to generator default options
------------------------------------

**Rationale:** Previously nanopb_generator added a timestamp header to generated
files and used only basename of files in `#include` directives. This is different
than what the `protoc` C++ backend does.

**Changes:** Now default options are `--no-timestamp` and `--no-strip-path`.

**Required actions:** If old behaviour is desired, add `--timestamp` and
`--strip-path` options to `nanopb_generator.py` or on `protoc` command line
as `--nanopb_out=--timestamp,--strip-path:outdir`.

**Error indications:** Compiler error: cannot find include file `mymessage.pb.h`
when compiling `mymessage.pb.c`.

Removal of bundled plugin.proto
-------------------------------

**Rationale:** Google's Python protobuf library, which is used in nanopb
generator, has included `plugin_pb2` with it since version 3.1.0. It is
not necessary to bundle it with nanopb anymore.

**Required actions:** Update `python-protobuf` to version 3.1.0 or newer.

**Error indications:** `ImportError: No module named compiler.plugin_pb2`

.options file is now always case-sensitive
------------------------------------------

**Rationale:** Previously field names in `.options` file were case-sensitive
on Linux and case-insensitive on Windows. This was by accident. Because
`.proto` files are case-sensitive, `.options` files should be too.

**Changes:** Now field names in `.options` are always case-sensitive, and
matched by `fnmatchcase()` instead of `fnmatch()`.

**Required actions:** If field names in `.options` are not capitalized the
same as in `.proto`, they must be updated.

*CHAR_BIT* define is now needed
-------------------------------
**Rationale:** To check whether the platform has 8-bit or larger chars, the
C standard *CHAR_BIT* macro is needed.

**Changes:** *pb.h* now includes *limits.h* for this macro.

**Required actions:** If your platform doesn't have *limits.h* available, you
can define the macro in *pb_syshdr.h*. There is an example in *extra* directory.

**Error indications:** "Cannot find include file <limits.h>." or "Undefined
identifier: CHAR_BIT."

Strings must now always be null-terminated
------------------------------------------
**Rationale:** Previously *pb_encode()* would accept non-terminated strings and
assume that they are the full length of the defined array. However, *pb_decode()*
would reject such messages because null terminator wouldn't fit in the array.

**Changes:** *pb_encode()* will now return an error if null terminator is missing.
Maximum encoded message size calculation is changed accordingly so that at most
*max_size-1* strings are assumed. New field option *max_length* can be used to
define the maximum string length, instead of the array size.

**Required actions:** If your strings were previously filling the whole allocated
array, increase the size of the field by 1.

**Error indications:** *pb_encode()* returns error *unterminated string*.

Removal of per-field default value constants
--------------------------------------------
**Rationale:** Previously nanopb declared a `fieldname_default` constant variable
for each field with a default value, and used these internally to initialize messages.
This however used unnecessarily large amount of storage for the values. The variables
were mostly for internal usage, but were available in the header file.

**Changes:** Default values are now stored as an encoded protobuf message.

**Required actions:** If your code previously used default constants, it will have to
be adapted to take the default value in some other way, such as by accessing
`static const MyMessage msg_default = MyMessage_init_default;` and `msg_default.fieldname`.

**Error indications:** Compiler error about `fieldname_default` being undeclared.

Zero tag in message now raises error by default
-----------------------------------------------
**Rationale:** Previously nanopb has allowed messages to be terminated by a null byte,
which is read as zero tag value. Most other protobuf implementations don't support this,
so it is not very useful feature. It has also been noted that this can complicate
debugging issues with corrupted messages.

**Changes:** `pb_decode()` now gives error when it encounters zero tag value. A new
function `pb_decode_ex()` supports flag `PB_DECODE_NULLTERMINATED` that supports
decoding null terminated messages.

**Required actions:** If application uses null termination for messages, switch it to
use `pb_decode_ex()` and `pb_encode_ex()`. If compatibility with 0.3.9.x is needed,
there are also `pb_decode_nullterminated()` and `pb_encode_nullterminated()` macros,
which work both in 0.4.0 and 0.3.9.

**Error indications:** Error message from `pb_decode()`: 'zero_tag'.

Submessages now have has_field in proto3 mode
---------------------------------------------
**Rationale:** Previously nanopb considered proto3 submessages as 'present' only
when their contents was non-zero. Most other protobuf libraries allow explicit
'null' state for submessages.

**Changes:** Submessages now have separate `has_field` in proto3 mode also.

**Required actions:** When using submessages in proto3 mode, user code must now
set `mymsg.has_submsg = true` for each submessage that is present. Alternatively,
the field option `proto3_singular_msgs` can be used to restore the old behavior.

**Error indications:** Submessages do not get encoded.

PB_OLD_CALLBACK_STYLE option has been removed
---------------------------------------------
**Rationale:** Back in 2013, function signature for callbacks was changed. The
`PB_OLD_CALLBACK_STYLE` option allowed compatibility with old code, but
complicated code and testing because of the different options.

**Changes:** `PB_OLD_CALLBACK_STYLE` option no-longer has any effect.

**Required actions:** If `PB_OLD_CALLBACK_STYLE` option was in use previously,
function signatures must be updated to use double pointers (`void**` and
`void * const *`).

**Error indications:** Assignment from incompatible pointer type.

protoc insertion points are no longer included by default
---------------------------------------------------------
**Rationale:** Protoc allows including comments in form `@@protoc_insertion_point`
to identify locations for other plugins to insert their own extra content.
Previously these were included by default, but they clutter the generated files
and are rarely used.

**Changes:** Insertion points are now included only when `--protoc-insertion-points`
option is passed to the generator.

Nanopb-0.3.9.4, 0.4.0 (2019-xx-xx)
==================================

Fix generation of min/max defines for enum types
------------------------------------------------

**Rationale:** Nanopb generator makes #defines for enum minimum and maximum
value. Previously these defines incorrectly had the first and last enum value,
instead of the actual minimum and maximum. (issue #405)

**Changes:** Minimum define now always has the smallest value, and maximum
define always has the largest value.

**Required actions:** If these defines are used and enum values in .proto file
are not defined in ascending order, user code behaviour may change. Check that
user code doesn't expect the old, incorrect first/last behaviour.

Fix undefined behavior related to bool fields
---------------------------------------------

**Rationale:** In C99, `bool` variables are not allowed to have other values
than `true` and `false`. Compilers use this fact in optimization, and constructs
like `int foo = msg.has_field ? 100 : 0` will give unexpected results otherwise.
Previously nanopb didn't enforce that decoded bool fields had valid values.

**Changes:** Bool fields are now handled separately as `PB_LTYPE_BOOL`. The
`LTYPE` descriptor numbers for other field types were renumbered.

**Required actions:** Source code files must be recompiled, but regenerating
`.pb.h`/`.pb.c` files from `.proto` is not required. If user code directly uses
the nanopb internal field representation (search for `PB_LTYPE_VARINT` in source),
it may need updating.

Nanopb-0.3.9.1, 0.4.0 (2018-04-14)
==================================

Fix handling of string and bytes default values
-----------------------------------------------

**Rationale:** Previously nanopb didn't properly decode special character
escapes like \\200 emitted by protoc. This caused these escapes to end up
verbatim in the default values in .pb.c file.

**Changes:** Escapes are now decoded, and e.g. "\\200" or "\\x80" results in
{0x80} for bytes field and "\\x80" for string field.

**Required actions:** If code has previously relied on '\\' in default value
being passed through verbatim, it must now be changed to '\\\\'.

Nanopb-0.3.8 (2017-03-05)
=========================

Fully drain substreams before closing
-------------------------------------

**Rationale:** If the substream functions were called directly and the caller
did not completely empty the substring before closing it, the parent stream
would be put into an incorrect state.

**Changes:** *pb_close_string_substream* can now error and returns a boolean.

**Required actions:** Add error checking onto any call to
*pb_close_string_substream*.

Change oneof format in .pb.c files
----------------------------------

**Rationale:** Previously two oneofs in a single message would be erroneously
handled as part of the same union.

**Changes:** Oneofs fields now use special *PB_DATAOFFSET_UNION* offset type
in generated .pb.c files to distinguish whether they are the first or following
field inside an union.

**Required actions:** Regenerate *.pb.c/.pb.h* files with new nanopb version if
oneofs are used.

Nanopb-0.3.5 (2016-02-13)
=========================

Add support for platforms without uint8_t
-----------------------------------------
**Rationale:** Some platforms cannot access 8-bit sized values directly, and
do not define *uint8_t*. Nanopb previously didn't support these platforms.

**Changes:** References to *uint8_t* were replaced with several alternatives,
one of them being a new *pb_byte_t* typedef. This in turn uses *uint_least8_t*
which means the smallest available type.

**Required actions:** If your platform does not have a standards-compliant
*stdint.h*, it may lack the definition for *[u]int_least8_t*. This must be
added manually, example can be found in *extra/pb_syshdr.h*.

**Error indications:** Compiler error: "unknown type name 'uint_least8_t'".

Nanopb-0.3.2 (2015-01-24)
=========================

Add support for OneOfs
----------------------
**Rationale:** Previously nanopb did not support the *oneof* construct in
*.proto* files. Those fields were generated as regular *optional* fields.

**Changes:** OneOfs are now generated as C unions. Callback fields are not
supported inside oneof and generator gives an error.

**Required actions:** The generator option *no_unions* can be used to restore old
behaviour and to allow callbacks to be used. To use unions, one change is
needed: use *which_xxxx* field to detect which field is present, instead
of *has_xxxx*. Compare the value against *MyStruct_myfield_tag*.

**Error indications:** Generator error: "Callback fields inside of oneof are
not supported". Compiler error: "Message" has no member named "has_xxxx".

Nanopb-0.3.0 (2014-08-26)
=========================

Separate field iterator logic to pb_common.c
--------------------------------------------
**Rationale:** Originally, the field iteration logic was simple enough to be
duplicated in *pb_decode.c* and *pb_encode.c*. New field types have made the
logic more complex, which required the creation of a new file to contain the
common functionality.

**Changes:** There is a new file, *pb_common.c*, which must be included in
builds.

**Required actions:** Add *pb_common.c* to build rules. This file is always
required. Either *pb_decode.c* or *pb_encode.c* can still be left out if some
functionality is not needed.

**Error indications:** Linker error: undefined reference to
*pb_field_iter_begin*, *pb_field_iter_next* or similar.

Change data type of field counts to pb_size_t
---------------------------------------------
**Rationale:** Often nanopb is used with small arrays, such as 255 items or
less. Using a full *size_t* field to store the array count wastes memory if
there are many arrays. There already exists parameters *PB_FIELD_16BIT* and
*PB_FIELD_32BIT* which tell nanopb what is the maximum size of arrays in use.

**Changes:** Generator will now use *pb_size_t* for the array *_count* fields.
The size of the type will be controlled by the *PB_FIELD_16BIT* and
*PB_FIELD_32BIT* compilation time options.

**Required actions:** Regenerate all *.pb.h* files. In some cases casts to the
*pb_size_t* type may need to be added in the user code when accessing the
*_count* fields.

**Error indications:** Incorrect data at runtime, crashes. But note that other
changes in the same version already require regenerating the files and have
better indications of errors, so this is only an issue for development
versions.

Renamed some macros and identifiers
-----------------------------------
**Rationale:** Some names in nanopb core were badly chosen and conflicted with
ISO C99 reserved names or lacked a prefix. While they haven't caused trouble
so far, it is reasonable to switch to non-conflicting names as these are rarely
used from user code.

**Changes:** The following identifier names have changed:

  * Macros:
  
    * STATIC_ASSERT(x) -> PB_STATIC_ASSERT(x)
    * UNUSED(x) -> PB_UNUSED(x)
  
  * Include guards:
  
    * _PB_filename_ -> PB_filename_INCLUDED
  
  * Structure forward declaration tags:
  
    * _pb_field_t -> pb_field_s
    * _pb_bytes_array_t -> pb_bytes_array_s
    * _pb_callback_t -> pb_callback_s
    * _pb_extension_type_t -> pb_extension_type_s
    * _pb_extension_t -> pb_extension_s
    * _pb_istream_t -> pb_istream_s
    * _pb_ostream_t -> pb_ostream_s

**Required actions:** Regenerate all *.pb.c* files. If you use any of the above
identifiers in your application code, perform search-replace to the new name.

**Error indications:** Compiler errors on lines with the macro/type names.

Nanopb-0.2.9 (2014-08-09)
=========================

Change semantics of generator -e option
---------------------------------------
**Rationale:** Some compilers do not accept filenames with two dots (like
in default extension .pb.c). The *-e* option to the generator allowed changing
the extension, but not skipping the extra dot.

**Changes:** The *-e* option in generator will no longer add the prepending
dot. The default value has been adjusted accordingly to *.pb.c* to keep the
default behaviour the same as before.

**Required actions:** Only if using the generator -e option. Add dot before
the parameter value on the command line.

**Error indications:** File not found when trying to compile generated files.

Nanopb-0.2.7 (2014-04-07)
=========================

Changed pointer-type bytes field datatype
-----------------------------------------
**Rationale:** In the initial pointer encoding support since nanopb-0.2.5,
the bytes type used a separate *pb_bytes_ptr_t* type to represent *bytes*
fields. This made it easy to encode data from a separate, user-allocated
buffer. However, it made the internal logic more complex and was inconsistent
with the other types.

**Changes:** Dynamically allocated bytes fields now have the *pb_bytes_array_t*
type, just like statically allocated ones.

**Required actions:** Only if using pointer-type fields with the bytes datatype.
Change any access to *msg->field.size* to *msg->field->size*. Change any
allocation to reserve space of amount *PB_BYTES_ARRAY_T_ALLOCSIZE(n)*. If the
data pointer was begin assigned from external source, implement the field using
a callback function instead.

**Error indications:** Compiler error: unknown type name *pb_bytes_ptr_t*.

Nanopb-0.2.4 (2013-11-07)
=========================

Remove the NANOPB_INTERNALS compilation option
----------------------------------------------
**Rationale:** Having the option in the headers required the functions to
be non-static, even if the option is not used. This caused errors on some
static analysis tools.

**Changes:** The *#ifdef* and associated functions were removed from the
header.

**Required actions:** Only if the *NANOPB_INTERNALS* option was previously
used. Actions are as listed under nanopb-0.1.3 and nanopb-0.1.6.

**Error indications:** Compiler warning: implicit declaration of function
*pb_dec_string*, *pb_enc_string*, or similar.

Nanopb-0.2.1 (2013-04-14)
=========================

Callback function signature
---------------------------
**Rationale:** Previously the auxilary data to field callbacks was passed
as *void\**. This allowed passing of any data, but made it unnecessarily
complex to return a pointer from callback.

**Changes:** The callback function parameter was changed to *void\*\**.

**Required actions:** You can continue using the old callback style by
defining *PB_OLD_CALLBACK_STYLE*. Recommended action is to:

  * Change the callback signatures to contain *void\*\** for decoders and
    *void \* const \** for encoders.
  * Change the callback function body to use *\*arg* instead of *arg*.

**Error indications:** Compiler warning: assignment from incompatible
pointer type, when initializing *funcs.encode* or *funcs.decode*.

Nanopb-0.2.0 (2013-03-02)
=========================

Reformatted generated .pb.c file using macros
---------------------------------------------
**Rationale:** Previously the generator made a list of C *pb_field_t*
initializers in the .pb.c file. This led to a need to regenerate all .pb.c
files after even small changes to the *pb_field_t* definition.

**Changes:** Macros were added to pb.h which allow for cleaner definition
of the .pb.c contents. By changing the macro definitions, changes to the
field structure are possible without breaking compatibility with old .pb.c
files.

**Required actions:** Regenerate all .pb.c files from the .proto sources.

**Error indications:** Compiler warning: implicit declaration of function
*pb_delta_end*.

Changed pb_type_t definitions
-----------------------------
**Rationale:** The *pb_type_t* was previously an enumeration type. This
caused warnings on some compilers when using bitwise operations to set flags
inside the values.

**Changes:** The *pb_type_t* was changed to *typedef uint8_t*. The values
were changed to *#define*. Some value names were changed for consistency.

**Required actions:** Only if you directly access the `pb_field_t` contents
in your own code, something which is not usually done. Needed changes:

  * Change *PB_HTYPE_ARRAY* to *PB_HTYPE_REPEATED*.
  * Change *PB_HTYPE_CALLBACK* to *PB_ATYPE()* and *PB_ATYPE_CALLBACK*.

**Error indications:** Compiler error: *PB_HTYPE_ARRAY* or *PB_HTYPE_CALLBACK*
undeclared.

Nanopb-0.1.6 (2012-09-02)
=========================

Refactored field decoder interface
----------------------------------
**Rationale:** Similarly to field encoders in nanopb-0.1.3.

**Changes:** New functions with names *pb_decode_\** were added.

**Required actions:** By defining NANOPB_INTERNALS, you can still keep using
the old functions. Recommended action is to replace any calls with the newer
*pb_decode_\** equivalents.

**Error indications:** Compiler warning: implicit declaration of function
*pb_dec_string*, *pb_dec_varint*, *pb_dec_submessage* or similar.

Nanopb-0.1.3 (2012-06-12)
=========================

Refactored field encoder interface
----------------------------------
**Rationale:** The old *pb_enc_\** functions were designed mostly for the
internal use by the core. Because they are internally accessed through
function pointers, their signatures had to be common. This led to a confusing
interface for external users.

**Changes:** New functions with names *pb_encode_\** were added. These have
easier to use interfaces. The old functions are now only thin wrappers for
the new interface.

**Required actions:** By defining NANOPB_INTERNALS, you can still keep using
the old functions. Recommended action is to replace any calls with the newer
*pb_encode_\** equivalents.

**Error indications:** Compiler warning: implicit declaration of function
*pb_enc_string*, *pb_enc_varint, *pb_enc_submessage* or similar.