:tocdepth: 2

.. _module-pw_tokenizer-api:

=============
API reference
=============
.. pigweed-module-subpage::
   :name: pw_tokenizer

-----
C/C++
-----
Moved: :cc:`pw_tokenizer`

----
Rust
----
See `Crate pw_tokenizer </rustdoc/pw_tokenizer/>`_.

------
Python
------

Tokenization
============
.. autofunction:: pw_tokenizer.encode.encode_token_and_args
.. autofunction:: pw_tokenizer.tokens.pw_tokenizer_65599_hash

Detokenization
==============
.. automodule:: pw_tokenizer.detokenize
   :members:
.. automodule:: pw_tokenizer.proto
   :members: