| :tocdepth: 2 |
| |
| .. _module-pw_tokenizer-api: |
| |
| ============= |
| API reference |
| ============= |
| .. pigweed-module-subpage:: |
| :name: pw_tokenizer |
| |
| ----- |
| C/C++ |
| ----- |
| Moved: :cc:`pw_tokenizer` |
| |
| ---- |
| Rust |
| ---- |
| See `Crate pw_tokenizer </rustdoc/pw_tokenizer/>`_. |
| |
| ------ |
| Python |
| ------ |
| |
| Tokenization |
| ============ |
| .. autofunction:: pw_tokenizer.encode.encode_token_and_args |
| .. autofunction:: pw_tokenizer.tokens.pw_tokenizer_65599_hash |
| |
| Detokenization |
| ============== |
| .. automodule:: pw_tokenizer.detokenize |
| :members: |
| .. automodule:: pw_tokenizer.proto |
| :members: |
