pw_tokenizer: Variable length hash

Renames pw_tokenizer_65599_fixed_length_hash to pw_tokenizer_65599_hash
and changes the function to do variable length tokenization by default.
This makes it easier to generate unbounded length tokens in addition to
fixed-length tokens.

Change-Id: I7dac54fbf39efa523708e9c1db49a28859078240
Reviewed-on: https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/78881
Reviewed-by: Wyatt Hepler <hepler@google.com>
Pigweed-Auto-Submit: Armando Montanez <amontanez@google.com>
Commit-Queue: Auto-Submit <auto-submit@pigweed.google.com.iam.gserviceaccount.com>
4 files changed
tree: bb028c57c249553bc0ab7e0107af8cff997a556f
  1. build_overrides/
  2. docker/
  3. docs/
  4. pw_allocator/
  5. pw_analog/
  6. pw_android_toolchain/
  7. pw_arduino_build/
  8. pw_assert/
  9. pw_assert_basic/
  10. pw_assert_log/
  11. pw_assert_zephyr/
  12. pw_base64/
  13. pw_bloat/
  14. pw_blob_store/
  15. pw_bluetooth_hci/
  16. pw_boot/
  17. pw_boot_cortex_m/
  18. pw_build/
  19. pw_build_info/
  20. pw_build_mcuxpresso/
  21. pw_bytes/
  22. pw_checksum/
  23. pw_chrono/
  24. pw_chrono_embos/
  25. pw_chrono_freertos/
  26. pw_chrono_stl/
  27. pw_chrono_threadx/
  28. pw_chrono_zephyr/
  29. pw_cli/
  30. pw_console/
  31. pw_containers/
  32. pw_cpu_exception/
  33. pw_cpu_exception_cortex_m/
  34. pw_crypto/
  35. pw_docgen/
  36. pw_doctor/
  37. pw_env_setup/
  38. pw_file/
  39. pw_function/
  40. pw_fuzzer/
  41. pw_hdlc/
  42. pw_hex_dump/
  43. pw_i2c/
  44. pw_interrupt/
  45. pw_interrupt_cortex_m/
  46. pw_interrupt_zephyr/
  47. pw_kvs/
  48. pw_libc/
  49. pw_log/
  50. pw_log_basic/
  51. pw_log_null/
  52. pw_log_rpc/
  53. pw_log_tokenized/
  54. pw_log_zephyr/
  55. pw_malloc/
  56. pw_malloc_freelist/
  57. pw_metric/
  58. pw_minimal_cpp_stdlib/
  59. pw_module/
  60. pw_multisink/
  61. pw_package/
  62. pw_persistent_ram/
  63. pw_polyfill/
  64. pw_preprocessor/
  65. pw_presubmit/
  66. pw_protobuf/
  67. pw_protobuf_compiler/
  68. pw_random/
  69. pw_result/
  70. pw_ring_buffer/
  71. pw_router/
  72. pw_rpc/
  73. pw_snapshot/
  74. pw_software_update/
  75. pw_span/
  76. pw_spi/
  77. pw_status/
  78. pw_stm32cube_build/
  79. pw_stream/
  80. pw_string/
  81. pw_symbolizer/
  82. pw_sync/
  83. pw_sync_baremetal/
  84. pw_sync_embos/
  85. pw_sync_freertos/
  86. pw_sync_stl/
  87. pw_sync_threadx/
  88. pw_sync_zephyr/
  89. pw_sys_io/
  90. pw_sys_io_arduino/
  91. pw_sys_io_baremetal_lm3s6965evb/
  92. pw_sys_io_baremetal_stm32f429/
  93. pw_sys_io_mcuxpresso/
  94. pw_sys_io_stdio/
  95. pw_sys_io_stm32cube/
  96. pw_sys_io_zephyr/
  97. pw_system/
  98. pw_target_runner/
  99. pw_thread/
  100. pw_thread_embos/
  101. pw_thread_freertos/
  102. pw_thread_stl/
  103. pw_thread_threadx/
  104. pw_tls_client/
  105. pw_tls_client_boringssl/
  106. pw_tls_client_mbedtls/
  107. pw_tokenizer/
  108. pw_tool/
  109. pw_toolchain/
  110. pw_trace/
  111. pw_trace_tokenized/
  112. pw_transfer/
  113. pw_unit_test/
  114. pw_varint/
  115. pw_watch/
  116. pw_web_ui/
  117. pw_work_queue/
  118. targets/
  119. third_party/
  120. zephyr/
  121. .bazelignore
  122. .bazelrc
  123. .clang-format
  124. .clang-tidy
  125. .eslintrc.json
  126. .gitattributes
  127. .gitignore
  128. .gn
  129. .prettierrc.js
  130. .pylintrc
  131. AUTHORS
  132. Android.bp
  133. BUILD.bazel
  134. BUILD.gn
  135. BUILDCONFIG.gn
  136. CMakeLists.txt
  137. Kconfig.zephyr
  138. LICENSE
  139. OWNERS
  140. PW_PLUGINS
  141. README.md
  142. WORKSPACE
  143. activate.bat
  144. bootstrap.bat
  145. bootstrap.sh
  146. modules.gni
  147. package.json
  148. tsconfig.json
  149. yarn.lock
README.md

Pigweed

Pigweed is an open source collection of embedded-targeted libraries–or as we like to call them, modules. These modules are building blocks and infrastructure that enable faster and more reliable development on small-footprint MMU-less 32-bit microcontrollers like the STMicroelectronics STM32L452 or the Nordic nRF52832.

For more information please see our website: https://pigweed.dev/

Links