This document details all the breaking changes that have been made to nanopb since its initial release. For each change, the rationale and required modifications of user applications are explained. Also any error indications are included, in order to make it easier to find this document.
Rationale: Previously CMake rules primarily looked for protoc
in system path. This was often an outdated version installed from package manager, and not necessarily compatible with python-protobuf
version installed from pip
.
Changes: CMake rules now default to using generator/protoc
, which in turn uses grpc_tools
Python package if available. If it is not available, system path is searched for protoc
.
Required actions: For most users, no actions are needed. In case of version incompatibilities, pip install --user --upgrade grpcio-tools protobuf
is recommended. If needed, PROTOBUF_PROTOC_EXECUTABLE
can be set to override the default.
Error indications: Failed to import generator/proto/nanopb_pb2.py
if versions of protoc
selected by CMake is different than installed python-protobuf
.
Rationale: Previously pb_byte_t
was always defined as uint8_least_t
. This could be annoying on some platforms without this define, or when some compiles might warn on conversion from uint8_t
. However not all platforms support uint8_t
sized access.
Changes: The stdint.h
header will define UINT8_MAX
exactly if uint8_t
is available. Use it to select which type to typedef.
Required actions: Usually none. If any compiler warnings are generated, they can either be fixed or PB_BYTE_T_OVERRIDE
can be defined to uint_least8_t
to restore old behavior.
Error indications: Implicit conversion from uint_least8_t
to uint8_t
.
Rationale: Due to the shortcomings of the WORKSPACE system, Bzlmod is going to replace the legacy WORKSPACE system in future Bazel releases. Therefore, nanopb has been migrated to use bzlmod to better support newer bazel versions.
Changes
Required actions: bazel build using WORKSPACE has been deprecated. To use bzlmod, adding below content to your MODULE.bazel
bazel_dep(name = "nanopb", version = "0.4.9") git_override( module_name = "nanopb", remote = "https://github.com/nanopb/nanopb.git", commit = "<commit>", )
noted that the name of the module has been changed to nanopb
, to better fit the convention of bzlmod. If the old name com_github_nanopb_nanopb
is preferred, can add repo_name
parameter to indicate the repo name.
bazel_dep(name = "nanopb", version = "0.4.9", repo_name="com_github_nanopb_nanopb")
Rationale: Nanopb-0.4.7 extended int_size
option to affect enums. This is only supported by C++11 and C23 compilers. The generation used #ifdef
to limit size option to use on C++ compilers. This caused binary incompatibility when project mixed C and C++ files.
Changes: enum_intsize
is now a separate option, and does not use #ifdef
. If compiler does not support the setting, compilation will fail.
Required actions: If using the recently introduced int_size
option on enums, update to use enum_intsize
instead.
Error indications: Enum integer sizes use defaults as the old setting is ignored.
Rationale: Previously CMakeLists.txt
installed nanopb Python module under name proto
and include file directly as /usr/include/pb.h
. These names have potential to conflict with other libraries.
Changes: Python module is installed as nanopb
and include files under /usr/include/nanopb
.
Required actions: Only affects users who install nanopb using the cmake
build system. Does not affect use of FindNanopb.cmake
. Calling nanopb generator should work as before. Include path may need adjustment if not using nanopb-targets.cmake
to determine it.
Error indications: Include file pb.h
not found when compiling against a system-wide installation done with CMake.
This option was separated to enum_intsize
in nanopb-0.4.9. This migration notice has been updated to match.
Rationale: The packed_enum
option does not work with MSVC due to #pragma pack
not supporting enums with MSVC. To workaround this, enum sizes can be specified with the new int_size
option. Note that this is only supported when generating C++.
Changes: The int_size
enum_intsize
option can be specified for enums.
Required actions: Any users concerned about the size of the generated C++ enums and are setting the int_size of enums via a wildcard (e.g. MyMessage.* int_size=IS_8
) will need to instead set the int_size
option for individual fields.
Error indications: The size of generated C++ enums has changed.
Changes: The include path passed to protoc
by the CMake rules was updated.
Required actions: No changes needed for most users. In some specific cases it could change the directory hierarchy generated by protoc
. More details in pull request #822.
Error indications: Generated .pb.c
or .pb.h
file not found when building with CMake rules.
Changes: To ease NANOPB_VERSION
macro usage, the value is directly a string.
Required actions: Most nanopb users probably never used that macro. If so, you certainly use the #
preprocessor to convert it as string. You, now, only have to call it directly, like this for example: strcpy(myvar, NANOPB_VERSION);
Changes: The default options passing method now uses --plugin-opt
which is supported by protoc 3.6.0 and newer (released in 2018).
Required actions: Update protoc
if needed, or alternatively install grpcio-tools
package from pip
. If neither is possible, the NANOPB_PROTOC_OLDER_THAN_3_6_0
cmake option can be used to restore the old style option passing. Note that it has problems with special characters such as :
.
Error indications: “protoc: Unknown flag: --nanopb_opt
”
Rationale: The nanopb generated headers use static assertions to catch errors at compile time. There are several mechanisms to implement this. The most widely supported is C11 _Static_assert
keyword. Previously the code used negative size array definition trick, which is supported already in C99 but does not work with every compiler and can produce confusing error messages.
Changes: Now _Static_assert
is used by default.
Required actions: If the keyword is not recognized, set the compiler to C11 standard mode if available. If it is not available, define either PB_C99_STATIC_ASSERT
or PB_NO_STATIC_ASSERT
in pb.h
or on compiler command line.
Error indications: Undefined identifier _Static_assert
Changes: Back in 2018, it was considered in pull request #241 to move nanopb generator options to a separate namespace. For this reason, a transitional file was added. It was later abandoned and is now removed to avoid confusion.
Required actions: Most nanopb users probably never used that transitional file at all. If your .proto
files import it, change to using generator/proto/nanopb.proto
.
Error indications: Errors about missing file options.proto
when running the generator.
Changes: New fields required_field_count
and largest_tag
were added to pb_msgdesc_t
and existing fields were reordered.
Required actions: All .pb.c
files must be recompiled. Regeneration is not needed.
Error indications: Messages may fail to encode or decode, or the code can crash inside load_descriptor_values()
in pb_common.c
.
Rationale: Previously nanopb-generator.py
had hashbang of #!/usr/bin/env python
, which would execute with Python 2 on most systems. Python 2 is now deprecated and many libraries are dropping support for it, which makes installing dependencies difficult. While nanopb_generator.py
has worked with Python 3 for years now, and overriding the python version was possible with virtualenv, that was an extra complication.
Changes: Hashbang now uses #!/usr/bin/env python3
. New file nanopb_generator.py2
can be used to run with Python 2, if necessary.
Required actions: If possible, just verify Python 3 is installed and necessary dependencies are installed for it. For example pip3 install protobuf grpcio-tools
should take care of it. If this is not possible, call nanopb_generator.py2
from your build scripts instead.
Error indications: python3: command not found
if Python 3 is not installed. Could not import the Google protobuf Python libraries
if dependencies are only installed for Python 2.
Rationale: Previously information about struct fields was stored as an array of pb_field_t
structures. This was a straightforward method, but required allocating space for e.g. submessage type and array size for all fields, even though most fields are not submessages nor arrays.
Changes: Now field information is encoded more efficiently in uint32_t
array in a variable-length format. Old pb_field_t
structure has been removed and it is now a typedef for pb_field_iter_t
. This retains compatibility with most old callback definitions. The field definitions in .pb.h
files are now of type pb_msgdesc_t
.
Required actions: If your own code accesses the low-level field information in pb_field_t
, it must be modified to do so only through the functions declared in pb_common.h
.
Error indications: incompatible pointer type
errors relating to pb_field_t
Rationale: Previously nanopb_generator added a timestamp header to generated files and used only basename of files in #include
directives. This is different than what the protoc
C++ backend does.
Changes: Now default options are --no-timestamp
and --no-strip-path
.
Required actions: If old behaviour is desired, add --timestamp
and --strip-path
options to nanopb_generator.py
or on protoc
command line as --nanopb_out=--timestamp,--strip-path:outdir
.
Error indications: Compiler error: cannot find include file mymessage.pb.h
when compiling mymessage.pb.c
.
Rationale: Google's Python protobuf library, which is used in nanopb generator, has included plugin_pb2
with it since version 3.1.0. It is not necessary to bundle it with nanopb anymore.
Required actions: Update python-protobuf
to version 3.1.0 or newer.
Error indications: ImportError: No module named compiler.plugin_pb2
Rationale: Previously field names in .options
file were case-sensitive on Linux and case-insensitive on Windows. This was by accident. Because .proto
files are case-sensitive, .options
files should be too.
Changes: Now field names in .options
are always case-sensitive, and matched by fnmatchcase()
instead of fnmatch()
.
Required actions: If field names in .options
are not capitalized the same as in .proto
, they must be updated.
CHAR_BIT
define is now neededRationale: To check whether the platform has 8-bit or larger chars, the C standard CHAR_BIT
macro is needed.
Changes: pb.h
now includes limits.h
for this macro.
Required actions: If your platform doesn't have limits.h
available, you can define the macro in pb_syshdr.h
. There is an example in extra
directory.
Error indications: "Cannot find include file <limits.h>."
or "Undefined identifier: CHAR_BIT."
Rationale: Previously pb_encode()
would accept non-terminated strings and assume that they are the full length of the defined array. However, pb_decode()
would reject such messages because null terminator wouldn't fit in the array.
Changes: pb_encode()
will now return an error if null terminator is missing. Maximum encoded message size calculation is changed accordingly so that at most max_size-1
strings are assumed. New field option max_length
can be used to define the maximum string length, instead of the array size.
Required actions: If your strings were previously filling the whole allocated array, increase the size of the field by 1.
Error indications: pb_encode()
returns error unterminated string
.
Rationale: Previously nanopb declared a fieldname_default
constant variable for each field with a default value, and used these internally to initialize messages. This however used unnecessarily large amount of storage for the values. The variables were mostly for internal usage, but were available in the header file.
Changes: Default values are now stored as an encoded protobuf message.
Required actions: If your code previously used default constants, it will have to be adapted to take the default value in some other way, such as by defining static const MyMessage msg_default = MyMessage_init_default;
and accessing msg_default.fieldname
.
Error indications: Compiler error about fieldname_default
being undeclared.
Rationale: Previously nanopb has allowed messages to be terminated by a null byte, which is read as zero tag value. Most other protobuf implementations don't support this, so it is not very useful feature. It has also been noted that this can complicate debugging issues with corrupted messages.
Changes: pb_decode()
now gives error when it encounters zero tag value. A new function pb_decode_ex()
supports flag PB_DECODE_NULLTERMINATED
that supports decoding null terminated messages.
Required actions: If application uses null termination for messages, switch it to use pb_decode_ex()
and pb_encode_ex()
. If compatibility with 0.3.9.x is needed, there are also pb_decode_nullterminated()
and pb_encode_nullterminated()
macros, which work both in 0.4.0 and 0.3.9.
Error indications: Error message from pb_decode()
: zero_tag
.
Rationale: Previously nanopb considered proto3 submessages as present only when their contents was non-zero. Most other protobuf libraries allow explicit null state for submessages.
Changes: Submessages now have separate has_field
in proto3 mode also.
Required actions: When using submessages in proto3 mode, user code must now set mymsg.has_submsg = true
for each submessage that is present. Alternatively, the field option proto3_singular_msgs
can be used to restore the old behavior.
Error indications: Submessages do not get encoded.
Rationale: Back in 2013, function signature for callbacks was changed. The PB_OLD_CALLBACK_STYLE
option allowed compatibility with old code, but complicated code and testing because of the different options.
Changes: PB_OLD_CALLBACK_STYLE
option no-longer has any effect.
Required actions: If PB_OLD_CALLBACK_STYLE
option was in use previously, function signatures must be updated to use double pointers (void**
and void * const *
).
Error indications: Assignment from incompatible pointer type.
Rationale: Protoc allows including comments in form @@protoc_insertion_point
to identify locations for other plugins to insert their own extra content. Previously these were included by default, but they clutter the generated files and are rarely used.
Changes: Insertion points are now included only when --protoc-insertion-points
option is passed to the generator.
Rationale: Nanopb generator makes #defines for enum minimum and maximum value. Previously these defines incorrectly had the first and last enum value, instead of the actual minimum and maximum. (issue #405)
Changes: Minimum define now always has the smallest value, and maximum define always has the largest value.
Required actions: If these defines are used and enum values in .proto file are not defined in ascending order, user code behaviour may change. Check that user code doesn't expect the old, incorrect first/last behaviour.
Rationale: In C99, bool
variables are not allowed to have other values than true
and false
. Compilers use this fact in optimization, and constructs like int foo = msg.has_field ? 100 : 0;
will give unexpected results otherwise. Previously nanopb didn't enforce that decoded bool fields had valid values.
Changes: Bool fields are now handled separately as PB_LTYPE_BOOL
. The LTYPE
descriptor numbers for other field types were renumbered.
Required actions: Source code files must be recompiled, but regenerating .pb.h
/.pb.c
files from .proto
is not required. If user code directly uses the nanopb internal field representation (search for PB_LTYPE_VARINT
in source), it may need updating.
Rationale: Previously nanopb didn't properly decode special character escapes like \200
emitted by protoc. This caused these escapes to end up verbatim in the default values in .pb.c file.
Changes: Escapes are now decoded, and e.g. \200
or \x80
results in {0x80} for bytes field and "\x80"
for string field.
Required actions: If code has previously relied on \
in default value being passed through verbatim, it must now be changed to \\
.
Rationale: If the substream functions were called directly and the caller did not completely empty the substring before closing it, the parent stream would be put into an incorrect state.
Changes: pb_close_string_substream
can now error and returns a boolean.
Required actions: Add error checking onto any call to pb_close_string_substream
.
Rationale: Previously two oneofs in a single message would be erroneously handled as part of the same union.
Changes: Oneofs fields now use special PB_DATAOFFSET_UNION
offset type in generated .pb.c files to distinguish whether they are the first or following field inside an union.
Required actions: Regenerate .pb.c/.pb.h
files with new nanopb version if oneofs are used.
Rationale: Some platforms cannot access 8-bit sized values directly, and do not define uint8_t
. Nanopb previously didn't support these platforms.
Changes: References to uint8_t
were replaced with several alternatives, one of them being a new pb_byte_t
typedef. This in turn uses uint_least8_t
which means the smallest available type.
Required actions: If your platform does not have a standards-compliant stdint.h
, it may lack the definition for [u]int_least8_t
. This must be added manually, example can be found in extra/pb_syshdr.h
.
Error indications: Compiler error: "unknown type name 'uint_least8_t'"
.
Rationale: Previously nanopb did not support the oneof
construct in .proto
files. Those fields were generated as regular optional
fields.
Changes: OneOfs are now generated as C unions. Callback fields are not supported inside oneof and generator gives an error.
Required actions: The generator option no_unions
can be used to restore old behaviour and to allow callbacks to be used. To use unions, one change is needed: use which_xxxx
field to detect which field is present, instead of has_xxxx
. Compare the value against MyStruct_myfield_tag
.
Error indications: Generator error: "Callback fields inside of oneof are not supported"
. Compiler error: "Message"
has no member named "has_xxxx"
.
Rationale: Originally, the field iteration logic was simple enough to be duplicated in pb_decode.c
and pb_encode.c
. New field types have made the logic more complex, which required the creation of a new file to contain the common functionality.
Changes: There is a new file, pb_common.c
, which must be included in builds.
Required actions: Add pb_common.c
to build rules. This file is always required. Either pb_decode.c
or pb_encode.c
can still be left out if some functionality is not needed.
Error indications: Linker error: undefined reference to pb_field_iter_begin
, pb_field_iter_next
or similar.
Rationale: Often nanopb is used with small arrays, such as 255 items or less. Using a full size_t
field to store the array count wastes memory if there are many arrays. There already exists parameters PB_FIELD_16BIT
and PB_FIELD_32BIT
which tell nanopb what is the maximum size of arrays in use.
Changes: Generator will now use pb_size_t
for the array _count
fields. The size of the type will be controlled by the PB_FIELD_16BIT
and PB_FIELD_32BIT
compilation time options.
Required actions: Regenerate all .pb.h
files. In some cases casts to the pb_size_t
type may need to be added in the user code when accessing the _count
fields.
Error indications: Incorrect data at runtime, crashes. But note that other changes in the same version already require regenerating the files and have better indications of errors, so this is only an issue for development versions.
Rationale: Some names in nanopb core were badly chosen and conflicted with ISO C99 reserved names or lacked a prefix. While they haven't caused trouble so far, it is reasonable to switch to non-conflicting names as these are rarely used from user code.
Changes: The following identifier names have changed:
Required actions: Regenerate all .pb.c
files. If you use any of the above identifiers in your application code, perform search-replace to the new name.
Error indications: Compiler errors on lines with the macro/type names.
Rationale: Some compilers do not accept filenames with two dots (like in default extension .pb.c). The -e
option to the generator allowed changing the extension, but not skipping the extra dot.
Changes: The -e
option in generator will no longer add the prepending dot. The default value has been adjusted accordingly to .pb.c
to keep the default behaviour the same as before.
Required actions: Only if using the generator -e option. Add dot before the parameter value on the command line.
Error indications: File not found when trying to compile generated files.
Rationale: In the initial pointer encoding support since nanopb-0.2.5, the bytes type used a separate pb_bytes_ptr_t
type to represent bytes
fields. This made it easy to encode data from a separate, user-allocated buffer. However, it made the internal logic more complex and was inconsistent with the other types.
Changes: Dynamically allocated bytes fields now have the pb_bytes_array_t
type, just like statically allocated ones.
Required actions: Only if using pointer-type fields with the bytes datatype. Change any access to msg->field.size
to msg->field->size
. Change any allocation to reserve space of amount PB_BYTES_ARRAY_T_ALLOCSIZE(n)
. If the data pointer was begin assigned from external source, implement the field using a callback function instead.
Error indications: Compiler error: unknown type name pb_bytes_ptr_t
.
Rationale: Having the option in the headers required the functions to be non-static, even if the option is not used. This caused errors on some static analysis tools.
Changes: The \#ifdef
and associated functions were removed from the header.
Required actions: Only if the NANOPB_INTERNALS
option was previously used. Actions are as listed under nanopb-0.1.3 and nanopb-0.1.6.
Error indications: Compiler warning: implicit declaration of function pb_dec_string
, pb_enc_string
, or similar.
Rationale: Previously the auxiliary data to field callbacks was passed as void*
. This allowed passing of any data, but made it unnecessarily complex to return a pointer from callback.
Changes: The callback function parameter was changed to void**
.
Required actions: You can continue using the old callback style by defining PB_OLD_CALLBACK_STYLE
. Recommended action is to:
void**
for decoders and void * const *
for encoders.instead of
arg`.Error indications: Compiler warning: assignment from incompatible pointer type, when initializing funcs.encode
or funcs.decode
.
Rationale: Previously the generator made a list of C pb_field_t
initializers in the .pb.c file. This led to a need to regenerate all .pb.c files after even small changes to the pb_field_t
definition.
Changes: Macros were added to pb.h which allow for cleaner definition of the .pb.c contents. By changing the macro definitions, changes to the field structure are possible without breaking compatibility with old .pb.c files.
Required actions: Regenerate all .pb.c files from the .proto sources.
Error indications: Compiler warning: implicit declaration of function pb_delta_end
.
Rationale: The pb_type_t
was previously an enumeration type. This caused warnings on some compilers when using bitwise operations to set flags inside the values.
Changes: The pb_type_t
was changed to typedef uint8_t. The values were changed to #define
. Some value names were changed for consistency.
Required actions: Only if you directly access the pb_field_t
contents in your own code, something which is not usually done. Needed changes:
PB_HTYPE_ARRAY
to PB_HTYPE_REPEATED
.PB_HTYPE_CALLBACK
to PB_ATYPE()
and PB_ATYPE_CALLBACK
.Error indications: Compiler error: PB_HTYPE_ARRAY
or PB_HTYPE_CALLBACK
undeclared.
Rationale: Similarly to field encoders in nanopb-0.1.3.
Changes: New functions with names pb_decode_*
were added.
Required actions: By defining NANOPB_INTERNALS, you can still keep using the old functions. Recommended action is to replace any calls with the newer pb_decode_*
equivalents.
Error indications: Compiler warning: implicit declaration of function pb_dec_string
, pb_dec_varint
, pb_dec_submessage
or similar.
Rationale: The old pb_enc_*
functions were designed mostly for the internal use by the core. Because they are internally accessed through function pointers, their signatures had to be common. This led to a confusing interface for external users.
Changes: New functions with names pb_encode_*
were added. These have easier to use interfaces. The old functions are now only thin wrappers for the new interface.
Required actions: By defining NANOPB_INTERNALS, you can still keep using the old functions. Recommended action is to replace any calls with the newer pb_encode_*
equivalents.
Error indications: Compiler warning: implicit declaration of function pb_enc_string
, *pb_enc_varint,pb_enc_submessage\
or similar.