pw_tokenizer: Support unaligned databases

- Update the binary token database class to work correctly on unaligned
  data. Previously, pw_tokenizer binary databases had an unstated
  alignment requirement.
- Avoid aliasing TokenDatabase::RawEntry.
- Various improvements to the TokenDatabase container and iterator.

Fixes: b/291288142
Change-Id: I750ccd6559b0d97702440b26858c769a4de08035
Reviewed-on: https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/163333
Pigweed-Auto-Submit: Wyatt Hepler <hepler@google.com>
Presubmit-Verified: CQ Bot Account <pigweed-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Alexei Frolov <frolv@google.com>
Commit-Queue: Auto-Submit <auto-submit@pigweed-service-accounts.iam.gserviceaccount.com>
5 files changed
tree: 1dfc94750b950d0799bf3efaa5614915b192779c
  1. .allstar/
  2. .vscode/
  3. build_overrides/
  4. docker/
  5. docs/
  6. pw_alignment/
  7. pw_allocator/
  8. pw_analog/
  9. pw_android_toolchain/
  10. pw_arduino_build/
  11. pw_assert/
  12. pw_assert_basic/
  13. pw_assert_log/
  14. pw_assert_tokenized/
  15. pw_assert_zephyr/
  16. pw_async/
  17. pw_async_basic/
  18. pw_base64/
  19. pw_bloat/
  20. pw_blob_store/
  21. pw_bluetooth/
  22. pw_bluetooth_hci/
  23. pw_bluetooth_profiles/
  24. pw_boot/
  25. pw_boot_cortex_m/
  26. pw_build/
  27. pw_build_info/
  28. pw_build_mcuxpresso/
  29. pw_bytes/
  30. pw_checksum/
  31. pw_chrono/
  32. pw_chrono_embos/
  33. pw_chrono_freertos/
  34. pw_chrono_stl/
  35. pw_chrono_threadx/
  36. pw_chrono_zephyr/
  37. pw_cli/
  38. pw_compilation_testing/
  39. pw_console/
  40. pw_containers/
  41. pw_cpu_exception/
  42. pw_cpu_exception_cortex_m/
  43. pw_crypto/
  44. pw_digital_io/
  45. pw_digital_io_mcuxpresso/
  46. pw_docgen/
  47. pw_doctor/
  48. pw_env_setup/
  49. pw_file/
  50. pw_function/
  51. pw_fuzzer/
  52. pw_hdlc/
  53. pw_hex_dump/
  54. pw_i2c/
  55. pw_i2c_linux/
  56. pw_i2c_mcuxpresso/
  57. pw_ide/
  58. pw_interrupt/
  59. pw_interrupt_cortex_m/
  60. pw_interrupt_xtensa/
  61. pw_interrupt_zephyr/
  62. pw_intrusive_ptr/
  63. pw_kvs/
  64. pw_libc/
  65. pw_log/
  66. pw_log_android/
  67. pw_log_basic/
  68. pw_log_null/
  69. pw_log_rpc/
  70. pw_log_string/
  71. pw_log_tokenized/
  72. pw_log_zephyr/
  73. pw_malloc/
  74. pw_malloc_freelist/
  75. pw_metric/
  76. pw_minimal_cpp_stdlib/
  77. pw_module/
  78. pw_multisink/
  79. pw_package/
  80. pw_perf_test/
  81. pw_persistent_ram/
  82. pw_polyfill/
  83. pw_preprocessor/
  84. pw_presubmit/
  85. pw_protobuf/
  86. pw_protobuf_compiler/
  87. pw_random/
  88. pw_result/
  89. pw_ring_buffer/
  90. pw_router/
  91. pw_rpc/
  92. pw_rpc_transport/
  93. pw_rust/
  94. pw_snapshot/
  95. pw_software_update/
  96. pw_span/
  97. pw_spi/
  98. pw_spi_mcuxpresso/
  99. pw_status/
  100. pw_stm32cube_build/
  101. pw_stream/
  102. pw_stream_shmem_mcuxpresso/
  103. pw_stream_uart_linux/
  104. pw_stream_uart_mcuxpresso/
  105. pw_string/
  106. pw_symbolizer/
  107. pw_sync/
  108. pw_sync_baremetal/
  109. pw_sync_embos/
  110. pw_sync_freertos/
  111. pw_sync_stl/
  112. pw_sync_threadx/
  113. pw_sync_zephyr/
  114. pw_sys_io/
  115. pw_sys_io_arduino/
  116. pw_sys_io_baremetal_lm3s6965evb/
  117. pw_sys_io_baremetal_stm32f429/
  118. pw_sys_io_emcraft_sf2/
  119. pw_sys_io_mcuxpresso/
  120. pw_sys_io_pico/
  121. pw_sys_io_stdio/
  122. pw_sys_io_stm32cube/
  123. pw_sys_io_zephyr/
  124. pw_system/
  125. pw_target_runner/
  126. pw_thread/
  127. pw_thread_embos/
  128. pw_thread_freertos/
  129. pw_thread_stl/
  130. pw_thread_threadx/
  131. pw_thread_zephyr/
  132. pw_tls_client/
  133. pw_tls_client_boringssl/
  134. pw_tls_client_mbedtls/
  135. pw_tokenizer/
  136. pw_tool/
  137. pw_toolchain/
  138. pw_trace/
  139. pw_trace_tokenized/
  140. pw_transfer/
  141. pw_unit_test/
  142. pw_unit_test_zephyr/
  143. pw_varint/
  144. pw_watch/
  145. pw_web/
  146. pw_work_queue/
  147. seed/
  148. targets/
  149. third_party/
  150. ts/
  151. zephyr/
  152. .bazelignore
  153. .bazelrc
  154. .black.toml
  155. .clang-format
  156. .clang-tidy
  157. .eslintrc.json
  158. .git-blame-ignore-revs
  159. .gitattributes
  160. .gitignore
  161. .gn
  162. .mypy.ini
  163. .prettierrc.js
  164. .pw_ide.yaml
  165. .pylintrc
  166. activate.bat
  167. Android.bp
  168. AUTHORS
  169. bootstrap.bat
  170. bootstrap.sh
  171. BUILD.bazel
  172. BUILD.gn
  173. BUILDCONFIG.gn
  174. CMakeLists.txt
  175. jest.config.ts
  176. Kconfig.zephyr
  177. LICENSE
  178. modules.gni
  179. OWNERS
  180. package-lock.json
  181. package.json
  182. pigweed.json
  183. PIGWEED_MODULES
  184. README.md
  185. rollup.config.js
  186. tsconfig.json
  187. WORKSPACE
README.md

Pigweed

Pigweed is an open source collection of embedded-targeted libraries–or as we like to call them, modules. These modules are building blocks and infrastructure that enable faster and more reliable development on small-footprint MMU-less 32-bit microcontrollers like the STMicroelectronics STM32L452 or the Nordic nRF52832.

For more information please see our website: https://pigweed.dev/.

Links