Google Git
Sign in
pigweed / pigweed / pigweed / 13bbf6ba4242316a53f6f08598b53d46f6f814cb
commit13bbf6ba4242316a53f6f08598b53d46f6f814cb[log] [tgz]
authorMaya Hegde <mhegde@google.com>Thu Mar 13 13:41:15 2025 -0700
committerCQ Bot Account <pigweed-scoped@luci-project-accounts.iam.gserviceaccount.com>Thu Mar 13 13:41:15 2025 -0700
tree8af0733aca18ddb775a5f497d11ba02b0de353df
parent6a635aa33c7b301089ff2da4a1637707a66fb08f [diff]
pw_tokenizer: Add support for nested tokenization in C++

Use regex to perform token search for nested tokens, implement recursive
search, and check for doubly- nested tokens.

Change-Id: I9a6f4dea10161c6839e9b861cc8ca6fb4abfff8b
Reviewed-on: https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/258773
Reviewed-by: Wyatt Hepler <hepler@google.com>
Commit-Queue: Maya Hegde <mhegde@google.com>
Lint: Lint 🤖 <android-build-ayeaye@system.gserviceaccount.com>
7 files changed
tree: 8af0733aca18ddb775a5f497d11ba02b0de353df
  1. .allstar/
  2. .github/
  3. .vscode/
  4. build_overrides/
  5. docker/
  6. docs/
  7. kudzu/
  8. pw_alignment/
  9. pw_allocator/
  10. pw_analog/
  11. pw_android_toolchain/
  12. pw_arduino_build/
  13. pw_assert/
  14. pw_assert_basic/
  15. pw_assert_fuchsia/
  16. pw_assert_log/
  17. pw_assert_tokenized/
  18. pw_assert_trap/
  19. pw_assert_zephyr/
  20. pw_async/
  21. pw_async2/
  22. pw_async2_basic/
  23. pw_async2_epoll/
  24. pw_async_basic/
  25. pw_async_fuchsia/
  26. pw_atomic/
  27. pw_base64/
  28. pw_bloat/
  29. pw_blob_store/
  30. pw_bluetooth/
  31. pw_bluetooth_hci/
  32. pw_bluetooth_profiles/
  33. pw_bluetooth_proxy/
  34. pw_bluetooth_sapphire/
  35. pw_boot/
  36. pw_boot_cortex_m/
  37. pw_build/
  38. pw_build_android/
  39. pw_build_info/
  40. pw_build_mcuxpresso/
  41. pw_bytes/
  42. pw_channel/
  43. pw_checksum/
  44. pw_chre/
  45. pw_chrono/
  46. pw_chrono_embos/
  47. pw_chrono_freertos/
  48. pw_chrono_rp2040/
  49. pw_chrono_stl/
  50. pw_chrono_threadx/
  51. pw_chrono_zephyr/
  52. pw_cli/
  53. pw_cli_analytics/
  54. pw_clock_tree/
  55. pw_clock_tree_mcuxpresso/
  56. pw_compilation_testing/
  57. pw_config_loader/
  58. pw_console/
  59. pw_containers/
  60. pw_cpu_exception/
  61. pw_cpu_exception_cortex_m/
  62. pw_cpu_exception_risc_v/
  63. pw_crypto/
  64. pw_digital_io/
  65. pw_digital_io_linux/
  66. pw_digital_io_mcuxpresso/
  67. pw_digital_io_rp2040/
  68. pw_display/
  69. pw_dma_mcuxpresso/
  70. pw_docgen/
  71. pw_doctor/
  72. pw_elf/
  73. pw_emu/
  74. pw_env_setup/
  75. pw_env_setup_zephyr/
  76. pw_file/
  77. pw_flatbuffers/
  78. pw_format/
  79. pw_function/
  80. pw_fuzzer/
  81. pw_grpc/
  82. pw_hdlc/
  83. pw_hex_dump/
  84. pw_i2c/
  85. pw_i2c_linux/
  86. pw_i2c_mcuxpresso/
  87. pw_i2c_rp2040/
  88. pw_ide/
  89. pw_interrupt/
  90. pw_interrupt_cortex_m/
  91. pw_interrupt_freertos/
  92. pw_interrupt_xtensa/
  93. pw_interrupt_zephyr/
  94. pw_intrusive_ptr/
  95. pw_json/
  96. pw_kernel/
  97. pw_kvs/
  98. pw_libc/
  99. pw_libcxx/
  100. pw_log/
  101. pw_log_android/
  102. pw_log_basic/
  103. pw_log_fuchsia/
  104. pw_log_null/
  105. pw_log_rpc/
  106. pw_log_string/
  107. pw_log_tokenized/
  108. pw_log_zephyr/
  109. pw_malloc/
  110. pw_malloc_freelist/
  111. pw_malloc_freertos/
  112. pw_metric/
  113. pw_minimal_cpp_stdlib/
  114. pw_module/
  115. pw_multibuf/
  116. pw_multisink/
  117. pw_numeric/
  118. pw_package/
  119. pw_perf_test/
  120. pw_persistent_ram/
  121. pw_polyfill/
  122. pw_preprocessor/
  123. pw_presubmit/
  124. pw_protobuf/
  125. pw_protobuf_compiler/
  126. pw_random/
  127. pw_random_fuchsia/
  128. pw_result/
  129. pw_ring_buffer/
  130. pw_router/
  131. pw_rpc/
  132. pw_rpc_transport/
  133. pw_rust/
  134. pw_sensor/
  135. pw_snapshot/
  136. pw_software_update/
  137. pw_span/
  138. pw_spi/
  139. pw_spi_linux/
  140. pw_spi_mcuxpresso/
  141. pw_spi_rp2040/
  142. pw_status/
  143. pw_stm32cube_build/
  144. pw_stream/
  145. pw_stream_shmem_mcuxpresso/
  146. pw_stream_uart_linux/
  147. pw_stream_uart_mcuxpresso/
  148. pw_string/
  149. pw_symbolizer/
  150. pw_sync/
  151. pw_sync_baremetal/
  152. pw_sync_embos/
  153. pw_sync_freertos/
  154. pw_sync_stl/
  155. pw_sync_threadx/
  156. pw_sync_zephyr/
  157. pw_sys_io/
  158. pw_sys_io_ambiq_sdk/
  159. pw_sys_io_arduino/
  160. pw_sys_io_baremetal_lm3s6965evb/
  161. pw_sys_io_baremetal_stm32f429/
  162. pw_sys_io_emcraft_sf2/
  163. pw_sys_io_mcuxpresso/
  164. pw_sys_io_rp2040/
  165. pw_sys_io_stdio/
  166. pw_sys_io_stm32cube/
  167. pw_sys_io_zephyr/
  168. pw_system/
  169. pw_target_runner/
  170. pw_thread/
  171. pw_thread_embos/
  172. pw_thread_freertos/
  173. pw_thread_stl/
  174. pw_thread_threadx/
  175. pw_thread_zephyr/
  176. pw_tls_client/
  177. pw_tls_client_boringssl/
  178. pw_tls_client_mbedtls/
  179. pw_tokenizer/
  180. pw_toolchain/
  181. pw_trace/
  182. pw_trace_tokenized/
  183. pw_transfer/
  184. pw_uart/
  185. pw_uart_mcuxpresso/
  186. pw_unit_test/
  187. pw_unit_test_zephyr/
  188. pw_varint/
  189. pw_watch/
  190. pw_web/
  191. pw_work_queue/
  192. seed/
  193. targets/
  194. third_party/
  195. ts/
  196. zephyr/
  197. .dockerignore ⇨ .gitignore
  198. activate.fish ⇨ bootstrap.fish
  199. activate.sh ⇨ bootstrap.sh
  200. .bazelignore
  201. .bazelrc
  202. .bazelversion
  203. .black.toml
  204. .clang-format
  205. .clang-tidy
  206. .eslintrc.cjs
  207. .git-blame-ignore-revs
  208. .gitattributes
  209. .gitignore
  210. .gn
  211. .mypy.ini
  212. .prettierignore
  213. .prettierrc.cjs
  214. .pw_ide.yaml
  215. .pylintrc
  216. .ruff.toml
  217. activate.bat
  218. Android.bp
  219. AUTHORS
  220. bootstrap.bat
  221. bootstrap.fish
  222. bootstrap.sh
  223. BUILD.bazel
  224. BUILD.gn
  225. BUILDCONFIG.gn
  226. CMakeLists.txt
  227. go.mod
  228. go.sum
  229. jest.config.ts
  230. Kconfig.zephyr
  231. LICENSE
  232. MODULE.bazel
  233. MODULE.bazel.lock
  234. modules.gni
  235. OWNERS
  236. package-lock.json
  237. package.json
  238. pigweed.json
  239. PIGWEED_MODULES
  240. pnpm-lock.yaml
  241. README.md
  242. REPO.bazel
  243. rollup-protos.config.js
  244. rollup.config.js
  245. tsconfig.json
  246. WORKSPACE
  247. WORKSPACE_OWNERS
README.md

Pigweed

Pigweed is an open source collection of embedded-targeted libraries–or as we like to call them, modules. These modules are building blocks and infrastructure that enable faster and more reliable development on small-footprint MMU-less 32-bit microcontrollers like the STMicroelectronics STM32L452 or the Nordic nRF52832.

For more information please see our website: https://pigweed.dev/.

Links