Bluetooth: Mesh: Use separate workq for storing mesh settings

Currently mesh settings are stored in the system workqueue context.
Most of other stack functionality, such that advertisements (incl
relay), loopback, transport sar, beacons transmission, etc. is also
processed in the system workqueue context. When a massive amount of
data needs to be stored and in particularly when page erase needs to
be triggered by GC of NVS subsystem to allocate flash pages, the
execution of the stack (and other functionality that uses the system
workqueue) will be blocked until storing is finished. For example,
right after the provisioning of a erased device, a node may not be
responsive for up to 400ms before it can continue sending messages.
The waiting time may increase if there is a GATT connection in the
mean time.

When write or erase operation is triggered, the flash driver waits for
Bluetooth controller to allocate a time needed to perform the operation.
During the whole operation, the context from which the operation was
triggered is put to sleep. This allows other threads to run until
Bluetooth controller finds the time for the flash driver. In other words,
every settings_save_one or settings_delete should be considered as
rescheduling points.

Considering this, Bluetooth mesh can use another thread to store its
settings, thus releasing the system workqueue for other tasks including
the operation of the stack itself.

The consistency of the data to be stored is guaranteed by the current
implementation where the data is copied to another struct before calling
settings_save_one. The pending flag of a particular module is dropped in
settings.c before starting to store the corresponding data. Thus, if
during the sleep the node receives a message that triggers a change in a
module which data is currently being stored, the pending flag will be
restored and the new change will be stored eventually.

Having this option enabled including with the partial erase, will make
the node more responsive in the described situations.

Signed-off-by: Pavel Vasilyev <pavel.vasilyev@nordicsemi.no>
3 files changed