Automatically update tokenizer database

Change-Id: If1e5ae69cb5fb47ce2ceedf9d28ed3538ebf322e
Reviewed-on: https://pigweed-review.googlesource.com/c/pigweed/sample_project/+/17940
Reviewed-by: Wyatt Hepler <hepler@google.com>
Commit-Queue: Carlos Chinchilla <cachinchilla@google.com>
2 files changed
tree: 621feb7f57db9bde8435d840859cdcf8f9eb511e
  1. .gitignore
  2. .gitmodules
  3. .gn
  4. BUILD.gn
  5. BUILDCONFIG.gn
  6. README.md
  7. activate.bat
  8. banner.txt
  9. bootstrap.bat
  10. bootstrap.sh
  11. build_overrides/
  12. source/
  13. targets/
  14. third_party/
README.md

Pigweed Sample Project

This repository outlines the recommended way of using Pigweed in a new or existing project. Feel free to fork this repository, or read it as a reference.

For more information see the Pigweed Getting started guide

Check back for more complex examples and features coming soon!

Repository setup

The preferred method to add the Pigweed source is to use git submodules and place the Pigweed source repository in third_party/pigweed. If you forked this repository, don't forget to run git submodule init and git submodule update.

Environment Setup

The scripts bootstrap.bat for Windows, and bootstrap.sh for Unix systems call the respective Pigweed bootstrap scripts. Feel free to modify them to set up your development environment for the first time.

After the initial setup, use the activate.bat in Windows or activate.sh in Unix to enter the environment in a shell.

Banner

Make the project yours with your own banner. Create your own banner and place it in banner.txt.

Building

Generate the build files with gn gen out once, unless the build configuration has changed. Then, use ninja to build everything with ninja -C out.

Sample Application

The sample application in source/main.cc uses the sample module simple_counter. Look at source/BUILD.gn and source/simple_counter/BUILD.gn to see how these are built respectively. The key part is in the root BUILD.gn, which creates the host target using the host toolchain. A toolchain is required for each target.

Build the project and run the application. ./out/host_clang_debug/obj/source/bin/hello_world

Sample Test

The simple_counter module has tests defined in source/simple_counter_tests.cc. Look at source/simple_counter/BUILD.gn for an example of how a test is defined. The root BUILD.gn groups all the host tests together.

Build the project and run the tests. ./out/host_clang_debug_tests/obj/source/simple_counter/test/simple_counter_test

Tokenized Logging

Log entries in the sample app are tokenized to save binary space. The included tokens database, source/tokenizer_database.csv, is updated on each build. See the Pigweed pw_tokenizer for more information.

Optionally, the database can be created manually using the binary or the .elf file. python -m pw_tokenizer.database create --database source/tokenizer_database.csv out/host_clang_debug/obj/source/bin/hello_world

Running the app shows log entries similiar to $kgjLdg==. These can be saved to a file and then detokenized.

python -m pw_tokenizer.detokenize base64 source/tokenizer_database.csv -i log.txt

Or can be detokenized in realtime. ./out/host_clang_debug/obj/source/bin/hello_world | python -m pw_tokenizer.detokenize base64 source/tokenizer_database.csv