blob: abd06589e6e6812cfb4207f9b73216db0bc5ff74 [file] [log] [blame]
Zephyr Sanity Tests
###################
This script scans for the set of unit test applications in the git repository
and attempts to execute them. By default, it tries to build each test
case on boards marked as default in the board definition file.
The default options will build the majority of the tests on a defined set of
boards and will run in an emulated environment (QEMU) if available for the
architecture or configuration being tested.
In normal use, sanitycheck runs a limited set of kernel tests (inside
QEMU). Because of its limited text execution coverage, sanitycheck
cannot guarantee local changes will succeed in the full build
environment, but it does sufficient testing by building samples and
tests for different boards and different configurations to help keep the
full code tree buildable.
To run the script in the local tree, follow the steps below:
::
$ source zephyr-env.sh
$ ./scripts/sanitycheck
If you have a system with a large number of cores, you can build and run
all possible tests using the following options:
::
$ ./scripts/sanitycheck --all --enable-slow
This will build for all available boards and run all applicable tests in
a simulated (QEMU) environment.
The sanitycheck script accepts the following optional arguments:
-h, --help
show this help message and exit
-p PLATFORM, --platform PLATFORM
Platform filter for testing. This option may be used
multiple times. Testcases will only be built/run on
the platforms specified. If this option is not used,
then platforms marked as default in the platform
metadata file will be chosen to build and test.
-L N, --platform-limit N
Controls what platforms are tested if --platform or
--all are not used. For each architecture specified by
--arch (defaults to all of them), choose the first N
platforms to test in the arch-specific .yaml file
'platforms' list. Defaults to 1.
-a ARCH, --arch ARCH Arch filter for testing. Takes precedence over
--platform. If unspecified, test all arches. Multiple
invocations are treated as a logical 'or' relationship
-t TAG, --tag TAG Specify tags to restrict which tests to run by tag
value. Default is to not do any tag filtering.
Multiple invocations are treated as a logical 'or'
relationship
-e EXCLUDE_TAG, --exclude-tag EXCLUDE_TAG
Specify tags of tests that should not run. Default is
to run all tests with all tags.
-f, --only-failed Run only those tests that failed the previous sanity
check invocation.
-c CONFIG, --config CONFIG
Specify platform configuration values filtering. This
can be specified two ways: <config>=<value> or just
<config>. The defconfig for all platforms will be
checked. For the <config>=<value> case, only match
defconfig that have that value defined. For the
<config> case, match defconfig that have that value
assigned to any value. Prepend a '!' to invert the
match.
-s TEST, --test TEST Run only the specified test cases. These are named by
<path to test project relative to --testcase-
root>/<testcase.yaml section name>
-l, --all Build/test on all platforms. Any --platform arguments
ignored.
-o TESTCASE_REPORT, --testcase-report TESTCASE_REPORT
Output a CSV spreadsheet containing results of the
test run
-d DISCARD_REPORT, --discard-report DISCARD_REPORT
Output a CSV spreadsheet showing tests that were
skipped and why
--compare-report COMPARE_REPORT
Use this report file for size comparison
--ccache Enable the use of ccache when building
-B SUBSET, --subset SUBSET
Only run a subset of the tests, 1/4 for running the
first 25%, 3/5 means run the 3rd fifth of the total.
This option is useful when running a large number of
tests on different hosts to speed up execution time.
-y, --dry-run Create the filtered list of test cases, but don't
actually run them. Useful if you're just interested in
--discard-report
-r, --release Update the benchmark database with the results of this
test run. Intended to be run by CI when tagging an
official release. This database is used as a basis for
comparison when looking for deltas in metrics such as
footprint
-w, --warnings-as-errors
Treat warning conditions as errors
-v, --verbose Emit debugging information, call multiple times to
increase verbosity
-i, --inline-logs Upon test failure, print relevant log data to stdout
instead of just a path to it
--log-file FILENAME log also to file
-m, --last-metrics Instead of comparing metrics from the last --release,
compare with the results of the previous sanity check
invocation
-u, --no-update do not update the results of the last run of the
sanity checks
-b, --build-only Only build the code, do not execute any of it in QEMU
-j JOBS, --jobs JOBS Number of cores to use when building, defaults to
number of CPUs * 2
-H FOOTPRINT_THRESHOLD, --footprint-threshold FOOTPRINT_THRESHOLD
When checking test case footprint sizes, warn the user
if the new app size is greater then the specified
percentage from the last release. Default is 5. 0 to
warn on any increase on app size
-D, --all-deltas Show all footprint deltas, positive or negative.
Implies --footprint-threshold=0
-O OUTDIR, --outdir OUTDIR
Output directory for logs and binaries.
-n, --no-clean Do not delete the outdir before building. Will result
in faster compilation since builds will be incremental
-T TESTCASE_ROOT, --testcase-root TESTCASE_ROOT
Base directory to recursively search for test cases.
All testcase.yaml files under here will be processed.
May be called multiple times. Defaults to the
'samples' and 'tests' directories in the Zephyr tree.
-A ARCH_ROOT, --arch-root ARCH_ROOT
Directory to search for arch configuration files. All
.yaml files in the directory will be processed.
-z SIZE, --size SIZE Don't run sanity checks. Instead, produce a report to
stdout detailing RAM/ROM sizes on the specified
filenames. All other command line arguments ignored.
-S, --enable-slow Execute time-consuming test cases that have been
marked as 'slow' in testcase.yaml. Normally these are
only built.
-R, --enable-asserts Build all test cases with assertions enabled.
-Q, --error-on-deprecations
Error on deprecation warnings.
-x EXTRA_ARGS, --extra-args EXTRA_ARGS
Extra arguments to pass to the build when compiling
test cases. May be called multiple times. These will
be passed in after any sanitycheck-supplied options.
-C, --coverage Scan for unit test coverage with gcov + lcov.
Board Configuration
*******************
To build tests for a specific board and to execute some of the tests on real
hardware or in an emulation environment such as QEMU a board configuration file
is required which is generic enough to be used for other tasks that require a
board inventory with details about the board and its configuration that is only
available during build time otherwise.
The board metadata file is located in the board directory and is structured
using the YAML markup language. The example below shows a board with a data
required for best test coverage for this specific board:
.. code-block:: yaml
identifier: quark_d2000_crb
name: Quark D2000 Devboard
type: mcu
arch: x86
toolchain:
- zephyr
- issm
ram: 8
flash: 32
testing:
default: true
ignore_tags:
- net
- bluetooth
identifier:
A string that matches how the board is defined in the build system. This same
string is used when building, for example when calling 'cmake'::
# cmake -DBOARD=quark_d2000_crb ..
name:
The actual name of the board as it appears in marketing material.
type:
Type of the board or configuration, currently we support 2 types: mcu, qemu
arch:
Architecture of the board
toolchain:
The list of supported toolchains that can build this board. This should match
one of the values used for 'ZEPHYR_GCC_VARIANT' when building on the command line
ram:
Available RAM on the board (specified in KB). This is used to match testcase
requirements. If not specified we default to 128KB.
flash:
Available FLASH on the board (specified in KB). This is used to match testcase
requirements. If not specified we default to 512KB.
supported:
A list of features this board supports. This can be specified as a single word
feature or as a variant of a feature class. For example:
::
supported:
- pci
This indicates the board does support PCI. You can make a testcase build or
run only on such boards, or:
::
supported:
- netif:eth
- sensor:bmi16
A testcase can both depend on 'eth' to only test ethernet or on 'netif' to run
on any board with a networking interface.
testing:
testing relating keywords to provide best coverage for the features of this
board.
default: [True|False]:
This is a default board, it will tested with the highest priority and is
covered when invoking the simplified sanitycheck without any additional
arguments.
ignore_tags:
Do not attempt to build (and therefore run) tests marked with this list of
tags.
Test Cases
**********
Test cases are detected by the presence of a 'testcase.yaml' or a 'sample.yaml'
files in the application's project directory. This file may contain one or more
entries in the test section each identifying a test scenario. The name of
the test case only needs to be unique for the test cases specified in
that testcase meta-data.
Test cases are written using the YAML syntax and share the same structure as
samples. The following is an example test with a few options that are
explained in this document.
::
tests:
test:
build_only: true
platform_whitelist: qemu_cortex_m3 qemu_x86 arduino_101
tags: bluetooth
test_br:
build_only: true
extra_args: CONF_FILE="prj_br.conf"
filter: not CONFIG_DEBUG
platform_exclude: quark_d2000_crb
platform_whitelist: qemu_cortex_m3 qemu_x86
tags: bluetooth
A sample with tests will have the same structure with additional information
related to the sample and what is being demonstrated:
::
sample:
name: hello world
description: Hello World sample, the simplest Zephyr application
platforms: all
tests:
test:
build_only: true
tags: samples tests
min_ram: 16
singlethread:
build_only: true
extra_args: CONF_FILE=prj_single.conf
filter: not CONFIG_BT and not CONFIG_GPIO_SCH
tags: samples tests
min_ram: 16
The full canonical name for each test case is:
::
<path to test case>/<test entry>
Each test block in the testcase meta data can define the following key/value
pairs:
tags: <list of tags> (required)
A set of string tags for the testcase. Usually pertains to
functional domains but can be anything. Command line invocations
of this script can filter the set of tests to run based on tag.
skip: <True|False> (default False)
skip testcase unconditionally. This can be used for broken tests.
slow: <True|False> (default False)
Don't run this test case unless --enable-slow was passed in on the
command line. Intended for time-consuming test cases that are only
run under certain circumstances, like daily builds. These test cases
are still compiled.
extra_args: <list of extra arguments>
Extra arguments to pass to Make when building or running the
test case.
extra_configs: <list of extra configurations>
Extra configuration options to be merged with a master prj.conf
when building or running the test case. For example::
common:
tags: drivers adc
tests:
test:
depends_on: adc
test_resolution_6:
extra_configs:
- CONFIG_ADC_QMSI_SAMPLE_WIDTH=6
platform_whitelist: quark_se_c1000_ss_devboard
tags: hwtest
build_only: <True|False> (default False)
If true, don't try to run the test under QEMU even if the
selected platform supports it.
build_on_all: <True|False> (default False)
If true, attempt to build test on all available platforms.
depends_on: <list of features>
A board or platform can announce what features it supports, this option
will enable the test only those platforms that provide this feature.
min_ram: <integer>
minimum amount of RAM needed for this test to build and run. This is
compared with information provided by the board metadata.
min_flash: <integer>
minimum amount of ROM needed for this test to build and run. This is
compared with information provided by the board metadata.
timeout: <number of seconds>
Length of time to run test in QEMU before automatically killing it.
Default to 60 seconds.
arch_whitelist: <list of arches, such as x86, arm, arc>
Set of architectures that this test case should only be run for.
arch_exclude: <list of arches, such as x86, arm, arc>
Set of architectures that this test case should not run on.
platform_whitelist: <list of platforms>
Set of platforms that this test case should only be run for.
platform_exclude: <list of platforms>
Set of platforms that this test case should not run on.
extra_sections: <list of extra binary sections>
When computing sizes, sanitycheck will report errors if it finds
extra, unexpected sections in the Zephyr binary unless they are named
here. They will not be included in the size calculation.
filter: <expression>
Filter whether the testcase should be run by evaluating an expression
against an environment containing the following values:
::
{ ARCH : <architecture>,
PLATFORM : <platform>,
<all CONFIG_* key/value pairs in the test's generated defconfig>,
*<env>: any environment variable available
}
The grammar for the expression language is as follows:
expression ::= expression "and" expression
| expression "or" expression
| "not" expression
| "(" expression ")"
| symbol "==" constant
| symbol "!=" constant
| symbol "<" number
| symbol ">" number
| symbol ">=" number
| symbol "<=" number
| symbol "in" list
| symbol ":" string
| symbol
list ::= "[" list_contents "]"
list_contents ::= constant
| list_contents "," constant
constant ::= number
| string
For the case where expression ::= symbol, it evaluates to true
if the symbol is defined to a non-empty string.
Operator precedence, starting from lowest to highest:
or (left associative)
and (left associative)
not (right associative)
all comparison operators (non-associative)
arch_whitelist, arch_exclude, platform_whitelist, platform_exclude
are all syntactic sugar for these expressions. For instance
arch_exclude = x86 arc
Is the same as:
filter = not ARCH in ["x86", "arc"]
The ':' operator compiles the string argument as a regular expression,
and then returns a true value only if the symbol's value in the environment
matches. For example, if CONFIG_SOC="quark_se" then
filter = CONFIG_SOC : "quark.*"
Would match it.
The set of test cases that actually run depends on directives in the testcase
filed and options passed in on the command line. If there is any confusion,
running with -v or --discard-report can help show why particular test cases
were skipped.
Metrics (such as pass/fail state and binary size) for the last code
release are stored in scripts/sanity_chk/sanity_last_release.csv.
To update this, pass the --all --release options.
To load arguments from a file, write '+' before the file name, e.g.,
+file_name. File content must be one or more valid arguments separated by
line break instead of white spaces.
Most everyday users will run with no arguments.