pw_tokenizer: Deduplicate entries when reading from ELF

- Rework detokenization tests for ELF sections.
- Expand test coverage to include duplicate identical tokens. The test
  ELF file already contained several of these, so the tests did not
  initially pass.
- Support comparing FormatString objects.
- Check for duplicates before adding entries to the detokenizer when
  reading from ELF.

Fixes: b/413729254
Change-Id: I7af2d8d7e75e6c7ec7d5fbb2e819334c6cd5e084
Reviewed-on: https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/289412
Reviewed-by: Dave Roth <davidroth@google.com>
Commit-Queue: Wyatt Hepler <hepler@google.com>
Lint: Lint 🤖 <android-build-ayeaye@system.gserviceaccount.com>
Docs-Not-Needed: Wyatt Hepler <hepler@google.com>
Pigweed-Auto-Submit: Wyatt Hepler <hepler@google.com>
3 files changed
tree: e72af82eb8788b9563eb6f5430ced04a760ae46c
  1. .allstar/
  2. .github/
  3. .vscode/
  4. build_overrides/
  5. docker/
  6. docs/
  7. kudzu/
  8. pw_alignment/
  9. pw_allocator/
  10. pw_analog/
  11. pw_android_toolchain/
  12. pw_arduino_build/
  13. pw_assert/
  14. pw_assert_basic/
  15. pw_assert_fuchsia/
  16. pw_assert_log/
  17. pw_assert_tokenized/
  18. pw_assert_trap/
  19. pw_assert_zephyr/
  20. pw_async/
  21. pw_async2/
  22. pw_async2_basic/
  23. pw_async2_epoll/
  24. pw_async_basic/
  25. pw_async_fuchsia/
  26. pw_atomic/
  27. pw_base64/
  28. pw_bloat/
  29. pw_blob_store/
  30. pw_bluetooth/
  31. pw_bluetooth_hci/
  32. pw_bluetooth_profiles/
  33. pw_bluetooth_proxy/
  34. pw_bluetooth_sapphire/
  35. pw_boot/
  36. pw_boot_cortex_m/
  37. pw_build/
  38. pw_build_android/
  39. pw_build_info/
  40. pw_build_mcuxpresso/
  41. pw_bytes/
  42. pw_channel/
  43. pw_checksum/
  44. pw_chre/
  45. pw_chrono/
  46. pw_chrono_embos/
  47. pw_chrono_freertos/
  48. pw_chrono_rp2040/
  49. pw_chrono_stl/
  50. pw_chrono_threadx/
  51. pw_chrono_zephyr/
  52. pw_cli/
  53. pw_cli_analytics/
  54. pw_clock_tree/
  55. pw_clock_tree_mcuxpresso/
  56. pw_compilation_testing/
  57. pw_config_loader/
  58. pw_console/
  59. pw_containers/
  60. pw_cpu_exception/
  61. pw_cpu_exception_cortex_m/
  62. pw_cpu_exception_risc_v/
  63. pw_crypto/
  64. pw_digital_io/
  65. pw_digital_io_linux/
  66. pw_digital_io_mcuxpresso/
  67. pw_digital_io_rp2040/
  68. pw_digital_io_zephyr/
  69. pw_display/
  70. pw_dma_mcuxpresso/
  71. pw_docgen/
  72. pw_doctor/
  73. pw_elf/
  74. pw_emu/
  75. pw_env_setup/
  76. pw_env_setup_zephyr/
  77. pw_file/
  78. pw_flatbuffers/
  79. pw_format/
  80. pw_function/
  81. pw_fuzzer/
  82. pw_grpc/
  83. pw_hdlc/
  84. pw_hex_dump/
  85. pw_i2c/
  86. pw_i2c_linux/
  87. pw_i2c_mcuxpresso/
  88. pw_i2c_rp2040/
  89. pw_ide/
  90. pw_interrupt/
  91. pw_interrupt_cortex_m/
  92. pw_interrupt_freertos/
  93. pw_interrupt_zephyr/
  94. pw_intrusive_ptr/
  95. pw_json/
  96. pw_kernel/
  97. pw_kvs/
  98. pw_libc/
  99. pw_libcxx/
  100. pw_log/
  101. pw_log_android/
  102. pw_log_basic/
  103. pw_log_fuchsia/
  104. pw_log_null/
  105. pw_log_rpc/
  106. pw_log_string/
  107. pw_log_tokenized/
  108. pw_log_zephyr/
  109. pw_malloc/
  110. pw_malloc_freelist/
  111. pw_malloc_freertos/
  112. pw_metric/
  113. pw_minimal_cpp_stdlib/
  114. pw_module/
  115. pw_multibuf/
  116. pw_multisink/
  117. pw_numeric/
  118. pw_package/
  119. pw_perf_test/
  120. pw_persistent_ram/
  121. pw_polyfill/
  122. pw_preprocessor/
  123. pw_presubmit/
  124. pw_protobuf/
  125. pw_protobuf_compiler/
  126. pw_random/
  127. pw_random_fuchsia/
  128. pw_result/
  129. pw_ring_buffer/
  130. pw_router/
  131. pw_rpc/
  132. pw_rpc_transport/
  133. pw_rust/
  134. pw_sensor/
  135. pw_snapshot/
  136. pw_software_update/
  137. pw_span/
  138. pw_spi/
  139. pw_spi_linux/
  140. pw_spi_mcuxpresso/
  141. pw_spi_rp2040/
  142. pw_status/
  143. pw_stm32cube_build/
  144. pw_stream/
  145. pw_stream_shmem_mcuxpresso/
  146. pw_stream_uart_linux/
  147. pw_stream_uart_mcuxpresso/
  148. pw_string/
  149. pw_symbolizer/
  150. pw_sync/
  151. pw_sync_baremetal/
  152. pw_sync_embos/
  153. pw_sync_freertos/
  154. pw_sync_stl/
  155. pw_sync_threadx/
  156. pw_sync_zephyr/
  157. pw_sys_io/
  158. pw_sys_io_ambiq_sdk/
  159. pw_sys_io_arduino/
  160. pw_sys_io_baremetal_lm3s6965evb/
  161. pw_sys_io_baremetal_stm32f429/
  162. pw_sys_io_emcraft_sf2/
  163. pw_sys_io_mcuxpresso/
  164. pw_sys_io_rp2040/
  165. pw_sys_io_stdio/
  166. pw_sys_io_stm32cube/
  167. pw_sys_io_zephyr/
  168. pw_system/
  169. pw_target_runner/
  170. pw_thread/
  171. pw_thread_embos/
  172. pw_thread_freertos/
  173. pw_thread_stl/
  174. pw_thread_threadx/
  175. pw_thread_zephyr/
  176. pw_tls_client/
  177. pw_tls_client_boringssl/
  178. pw_tls_client_mbedtls/
  179. pw_tokenizer/
  180. pw_toolchain/
  181. pw_trace/
  182. pw_trace_tokenized/
  183. pw_transfer/
  184. pw_uart/
  185. pw_uart_mcuxpresso/
  186. pw_unit_test/
  187. pw_unit_test_zephyr/
  188. pw_varint/
  189. pw_watch/
  190. pw_web/
  191. pw_work_queue/
  192. seed/
  193. targets/
  194. third_party/
  195. ts/
  196. zephyr/
  197. .bazelignore
  198. .bazelrc
  199. .bazelversion
  200. .black.toml
  201. .clang-format
  202. .clang-tidy
  203. .eslintrc.cjs
  204. .git-blame-ignore-revs
  205. .gitattributes
  206. .gitignore
  207. .gn
  208. .mypy.ini
  209. .prettierignore
  210. .prettierrc.cjs
  211. .pw_ide.yaml
  212. .pylintrc
  213. .ruff.toml
  214. activate.bat
  215. Android.bp
  216. AUTHORS
  217. bootstrap.bat
  218. bootstrap.fish
  219. bootstrap.sh
  220. BUILD.bazel
  221. BUILD.gn
  222. BUILDCONFIG.gn
  223. CMakeLists.txt
  224. go.mod
  225. go.sum
  226. jest.config.ts
  227. Kconfig.zephyr
  228. LICENSE
  229. MODULE.bazel
  230. MODULE.bazel.lock
  231. modules.gni
  232. OWNERS
  233. package-lock.json
  234. package.json
  235. pigweed.json
  236. PIGWEED_MODULES
  237. pnpm-lock.yaml
  238. README.md
  239. REPO.bazel
  240. rollup-protos.config.js
  241. rollup.config.js
  242. tsconfig.json
  243. WORKSPACE
  244. WORKSPACE_OWNERS
README.md

Pigweed

Pigweed is an open source collection of embedded-targeted libraries–or as we like to call them, modules. These modules are building blocks and infrastructure that enable faster and more reliable development on small-footprint MMU-less 32-bit microcontrollers like the STMicroelectronics STM32L452 or the Nordic nRF52832.

For more information please see our website: https://pigweed.dev/.

Links