| .. _module-pw_tokenizer-get-started: |
| |
| ============================= |
| Get started with pw_tokenizer |
| ============================= |
| .. pigweed-module-subpage:: |
| :name: pw_tokenizer |
| |
| .. _module-pw_tokenizer-get-started-overview: |
| |
| -------- |
| Overview |
| -------- |
| There are two sides to ``pw_tokenizer``, which we call tokenization and |
| detokenization. |
| |
| * **Tokenization** converts string literals in the source code to binary tokens |
| at compile time. If the string has printf-style arguments, these are encoded |
| to compact binary form at runtime. |
| * **Detokenization** converts tokenized strings back to the original |
| human-readable strings. |
| |
| Here's an overview of what happens when ``pw_tokenizer`` is used: |
| |
| 1. During compilation, the ``pw_tokenizer`` module hashes string literals to |
| generate stable 32-bit tokens. |
| 2. The tokenization macro removes these strings by declaring them in an ELF |
| section that is excluded from the final binary. |
| 3. After compilation, strings are extracted from the ELF to build a database of |
| tokenized strings for use by the detokenizer. The ELF file may also be used |
| directly. |
| 4. During operation, the device encodes the string token and its arguments, if |
| any. |
| 5. The encoded tokenized strings are sent off-device or stored. |
| 6. Off-device, the detokenizer tools use the token database to decode the |
| strings to human-readable form. |
| |
| .. _module-pw_tokenizer-get-started-integration: |
| |
| Integrating with Bazel / GN / CMake projects |
| ============================================ |
| Integrating ``pw_tokenizer`` requires a few steps beyond building the code. This |
| section describes one way ``pw_tokenizer`` might be integrated with a project. |
| These steps can be adapted as needed. |
| |
| #. Add ``pw_tokenizer`` to your build. Build files for GN, CMake, and Bazel are |
| provided. For Make or other build systems, add the files specified in the |
| BUILD.gn's ``pw_tokenizer`` target to the build. |
| #. Use the tokenization macros in your code. See |
| :ref:`module-pw_tokenizer-tokenization`. |
| #. Ensure the ``.pw_tokenizer.*`` sections are included in your output ELF file: |
| |
| * In GN and CMake, this is done automatically. |
| * In Bazel, include ``"@pigweed//pw_tokenizer:linker_script"`` in the |
| ``deps`` of your main binary rule (assuming you're already overriding the |
| default linker script). |
| * If your binary does not use a custom linker script, you can pass |
| ``add_tokenizer_sections_to_default_script.ld`` to the linker which will |
| augment the default linker script (rather than override it). |
| * Alternatively, add the contents of ``pw_tokenizer_linker_sections.ld`` to |
| your project's linker script. |
| |
| #. Compile your code to produce an ELF file. |
| #. Run ``database.py create`` on the ELF file to generate a CSV token |
| database. See :ref:`module-pw_tokenizer-managing-token-databases`. |
| #. Commit the token database to your repository. See notes in |
| :ref:`module-pw_tokenizer-database-management`. |
| #. Integrate a ``database.py add`` command to your build to automatically update |
| the committed token database. In GN, use the ``pw_tokenizer_database`` |
| template to do this. See :ref:`module-pw_tokenizer-update-token-database`. |
| #. Integrate ``detokenize.py`` or the C++ detokenization library with your tools |
| to decode tokenized logs. See :ref:`module-pw_tokenizer-detokenization`. |
| |
| Using with Zephyr |
| ================= |
| When building ``pw_tokenizer`` with Zephyr, 3 Kconfigs can be used currently: |
| |
| * ``CONFIG_PIGWEED_TOKENIZER`` will automatically link ``pw_tokenizer`` as well |
| as any dependencies. |
| * ``CONFIG_PIGWEED_TOKENIZER_BASE64`` will automatically link |
| ``pw_tokenizer.base64`` as well as any dependencies. |
| * ``CONFIG_PIGWEED_DETOKENIZER`` will automatically link |
| ``pw_tokenizer.decoder`` as well as any dependencies. |
| |
| Once enabled, the tokenizer headers can be included like any Zephyr headers: |
| |
| .. code-block:: cpp |
| |
| #include <pw_tokenizer/tokenize.h> |
| |
| .. note:: |
| Zephyr handles the additional linker sections via |
| ``pw_tokenizer_zephyr.ld`` which is added to the end of the linker file |
| via a call to ``zephyr_linker_sources(SECTIONS ...)``. |