sanitycheck help: how to spot tests that were not run and just built
There are a few reasons why sanitycheck will only build a test and not
run it: list them in the developer guide. Also lists the --options that
provide that information and update their --help message.
A couple other --help fixes.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
diff --git a/scripts/sanitycheck b/scripts/sanitycheck
index 019ab07..5fce4b4 100755
--- a/scripts/sanitycheck
+++ b/scripts/sanitycheck
@@ -3,6 +3,8 @@
# SPDX-License-Identifier: Apache-2.0
"""Zephyr Sanity Tests
+Also check the "User and Developer Guides" at https://docs.zephyrproject.org/
+
This script scans for the set of unit test applications in the git
repository and attempts to execute them. By default, it tries to
build each test case on one platform per architecture, using a precedence
@@ -42,8 +44,8 @@
when building or running the test case.
build_only: <True|False> (default False)
- If true, don't try to run the test under QEMU even if the
- selected platform supports it.
+ If true, don't try to run the test even if the selected platform
+ supports it.
build_on_all: <True|False> (default False)
If true, attempt to build test on all available platforms.
@@ -2723,8 +2725,8 @@
"""
Artificially long but functional example:
$ ./scripts/sanitycheck -v \\
- --testcase-root tests/ \\
- --testcase-root mytests/ \\
+ --testcase-root tests/ztest/base \\
+ --testcase-root tests/kernel \\
--test tests/ztest/base/testing.ztest.verbose_0 \\
--test tests/kernel/fifo/fifo_api/kernel.fifo.poll
@@ -2776,7 +2778,7 @@
test_xor_subtest.add_argument(
"-s", "--test", action="append",
help="Run only the specified test cases. These are named by "
- "path/relative/to/Zephyr/base/section.name.in.testcase.yaml>")
+ "<path/relative/to/Zephyr/base/section.name.in.testcase.yaml>")
test_xor_subtest.add_argument(
"--sub-test", action="append",
@@ -2794,7 +2796,9 @@
parser.add_argument(
"-o", "--testcase-report",
- help="Output a CSV spreadsheet containing results of the test run")
+ help="""Output a CSV spreadsheet containing results of the test run.
+ The handler_time column is left blank for tests that were only
+ compiled and not run.""")
parser.add_argument(
"-d", "--discard-report",
help="Output a CSV spreadsheet showing tests that were skipped "
@@ -2844,7 +2848,11 @@
parser.add_argument("--detailed-report",
action="store",
metavar="FILENAME",
- help="Generate a junit report with detailed testcase results.")
+ help="""Generate a junit report with detailed testcase results.
+ Unlike the CSV file produced by --testcase-report, this XML
+ report includes only tests which have run and none which were
+ merely built. If an image with multiple tests crashes early then
+ later tests are not accounted for either.""")
parser.add_argument("--timestamps",
action="store_true",
