pw_tokenizer: Allow globs in GN database template
The paths argument for pw_tokenizer_database is passed directly as paths
or globs to the pw_tokenizer.database. This enables, for example,
collecting tokens from all .elf files in the output directory.
Change-Id: I04428294cc276c110561db2ef7c0dd790d1c43f9
Reviewed-on: https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/37080
Pigweed-Auto-Submit: Wyatt Hepler <hepler@google.com>
Commit-Queue: Auto-Submit <auto-submit@pigweed.google.com.iam.gserviceaccount.com>
Reviewed-by: Armando Montanez <amontanez@google.com>
diff --git a/pw_tokenizer/database.gni b/pw_tokenizer/database.gni
index 8b7b894..846c329 100644
--- a/pw_tokenizer/database.gni
+++ b/pw_tokenizer/database.gni
@@ -34,6 +34,9 @@
# these targets are added to deps
# optional_targets: GN targets from which to add tokens, if the output files
# already exist; these targets are NOT added to deps
+# paths: Paths or globs from which to add tokens. A common use for this is to
+# find all .elf files in the output directory (e.g.
+# "$root_build_dir/**/*.elf").
# input_databases: paths to other database files from which to add tokens
# deps: GN targets to build prior to generating the database; artifacts from
# these targets are NOT implicitly used for database generation
@@ -129,6 +132,10 @@
args += [ "<TARGET_FILE_IF_EXISTS($target)>$_domain" ]
}
+ if (defined(invoker.paths)) {
+ args += rebase_path(invoker.paths)
+ }
+
deps = _targets
if (defined(invoker.deps)) {
diff --git a/pw_tokenizer/docs.rst b/pw_tokenizer/docs.rst
index 121f025..3b1d0d2 100644
--- a/pw_tokenizer/docs.rst
+++ b/pw_tokenizer/docs.rst
@@ -500,22 +500,16 @@
GN integration
^^^^^^^^^^^^^^
Token databases may be updated or created as part of a GN build. The
-``pw_tokenizer_database`` template provided by ``dir_pw_tokenizer/database.gni``
-automatically updates an in-source tokenized strings database or creates a new
-database with artifacts from one or more GN targets or other database files.
+``pw_tokenizer_database`` template provided by
+``$dir_pw_tokenizer/database.gni`` automatically updates an in-source tokenized
+strings database or creates a new database with artifacts from one or more GN
+targets or other database files.
To create a new database, set the ``create`` variable to the desired database
type (``"csv"`` or ``"binary"``). The database will be created in the output
directory. To update an existing database, provide the path to the database with
the ``database`` variable.
-Each database in the source tree can only be updated from a single
-``pw_tokenizer_database`` rule. Updating the same database in multiple rules
-results in ``Duplicate output file`` GN errors or ``multiple rules generate
-<file>`` Ninja errors. To avoid these errors, ``pw_tokenizer_database`` rules
-should be defined in the default toolchain, and the input targets should be
-referenced with specific toolchains.
-
.. code-block::
import("//build_overrides/pigweed.gni")
@@ -528,6 +522,17 @@
input_databases = [ "other_database.csv" ]
}
+Instead of specifying GN targets, paths or globs to output files may be provided
+with the ``paths`` option.
+
+.. code-block::
+
+ pw_tokenizer_database("my_database") {
+ database = "database_in_the_source_tree.csv"
+ deps = [ ":apps" ]
+ paths = [ "$root_build_dir/**/*.elf" ]
+ }
+
Detokenization
==============
Detokenization is the process of expanding a token to the string it represents
