Andrew Boie | 08ce5a5 | 2016-02-22 13:28:10 -0800 | [diff] [blame] | 1 | #!/usr/bin/env python3 |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 2 | # vim: set syntax=python ts=4 : |
Anas Nashif | 3ae5262 | 2019-04-06 09:08:09 -0400 | [diff] [blame] | 3 | # SPDX-License-Identifier: Apache-2.0 |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 4 | """Zephyr Sanity Tests |
| 5 | |
Marc Herbert | e5cedca | 2019-04-08 14:02:34 -0700 | [diff] [blame] | 6 | Also check the "User and Developer Guides" at https://docs.zephyrproject.org/ |
| 7 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 8 | This script scans for the set of unit test applications in the git |
| 9 | repository and attempts to execute them. By default, it tries to |
| 10 | build each test case on one platform per architecture, using a precedence |
Paul Sokolovsky | ff70add | 2017-06-16 01:31:54 +0300 | [diff] [blame] | 11 | list defined in an architecture configuration file, and if possible |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 12 | run the tests in any available emulators or simulators on the system. |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 13 | |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 14 | Test cases are detected by the presence of a 'testcase.yaml' or a sample.yaml |
| 15 | files in the application's project directory. This file may contain one or more |
| 16 | blocks, each identifying a test scenario. The title of the block is a name for |
| 17 | the test case, which only needs to be unique for the test cases specified in |
| 18 | that testcase meta-data. The full canonical name for each test case is <path to |
| 19 | test case>/<block>. |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 20 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 21 | Each test block in the testcase meta data can define the following key/value |
| 22 | pairs: |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 23 | |
Anas Nashif | fa695d2 | 2017-10-04 16:14:27 -0400 | [diff] [blame] | 24 | tags: <list of tags> (required) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 25 | A set of string tags for the testcase. Usually pertains to |
| 26 | functional domains but can be anything. Command line invocations |
| 27 | of this script can filter the set of tests to run based on tag. |
| 28 | |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 29 | skip: <True|False> (default False) |
Anas Nashif | 2bd99bc | 2015-10-12 13:10:57 -0400 | [diff] [blame] | 30 | skip testcase unconditionally. This can be used for broken tests. |
| 31 | |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 32 | slow: <True|False> (default False) |
Andy Ross | dc4151f | 2019-01-03 14:17:43 -0800 | [diff] [blame] | 33 | Don't build or run this test case unless --enable-slow was passed |
| 34 | in on the command line. Intended for time-consuming test cases |
| 35 | that are only run under certain circumstances, like daily |
| 36 | builds. |
Andrew Boie | 6bb087c | 2016-02-10 13:39:00 -0800 | [diff] [blame] | 37 | |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 38 | extra_args: <list of extra arguments> |
Sebastian Bøe | c218261 | 2017-11-09 12:25:02 +0100 | [diff] [blame] | 39 | Extra cache entries to pass to CMake when building or running the |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 40 | test case. |
| 41 | |
Anas Nashif | ebc329d | 2017-10-17 09:00:33 -0400 | [diff] [blame] | 42 | extra_configs: <list of extra configurations> |
| 43 | Extra configuration options to be merged with a master prj.conf |
| 44 | when building or running the test case. |
| 45 | |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 46 | build_only: <True|False> (default False) |
Marc Herbert | e5cedca | 2019-04-08 14:02:34 -0700 | [diff] [blame] | 47 | If true, don't try to run the test even if the selected platform |
| 48 | supports it. |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 49 | |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 50 | build_on_all: <True|False> (default False) |
| 51 | If true, attempt to build test on all available platforms. |
| 52 | |
| 53 | depends_on: <list of features> |
| 54 | A board or platform can announce what features it supports, this option |
| 55 | will enable the test only those platforms that provide this feature. |
| 56 | |
| 57 | min_ram: <integer> |
| 58 | minimum amount of RAM needed for this test to build and run. This is |
| 59 | compared with information provided by the board metadata. |
| 60 | |
| 61 | min_flash: <integer> |
| 62 | minimum amount of ROM needed for this test to build and run. This is |
| 63 | compared with information provided by the board metadata. |
| 64 | |
| 65 | timeout: <number of seconds> |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 66 | Length of time to run test in emulator before automatically killing it. |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 67 | Default to 60 seconds. |
| 68 | |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 69 | arch_whitelist: <list of arches, such as x86, arm, arc> |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 70 | Set of architectures that this test case should only be run for. |
| 71 | |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 72 | arch_exclude: <list of arches, such as x86, arm, arc> |
Anas Nashif | 30d1387 | 2015-10-05 10:02:45 -0400 | [diff] [blame] | 73 | Set of architectures that this test case should not run on. |
| 74 | |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 75 | platform_whitelist: <list of platforms> |
Anas Nashif | 30d1387 | 2015-10-05 10:02:45 -0400 | [diff] [blame] | 76 | Set of platforms that this test case should only be run for. |
| 77 | |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 78 | platform_exclude: <list of platforms> |
Anas Nashif | 30d1387 | 2015-10-05 10:02:45 -0400 | [diff] [blame] | 79 | Set of platforms that this test case should not run on. |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 80 | |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 81 | extra_sections: <list of extra binary sections> |
Andrew Boie | 52fef67 | 2016-11-29 12:21:59 -0800 | [diff] [blame] | 82 | When computing sizes, sanitycheck will report errors if it finds |
| 83 | extra, unexpected sections in the Zephyr binary unless they are named |
| 84 | here. They will not be included in the size calculation. |
| 85 | |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 86 | filter: <expression> |
Andrew Boie | 3ea7892 | 2016-03-24 14:46:00 -0700 | [diff] [blame] | 87 | Filter whether the testcase should be run by evaluating an expression |
| 88 | against an environment containing the following values: |
| 89 | |
| 90 | { ARCH : <architecture>, |
| 91 | PLATFORM : <platform>, |
Javier B Perez | 7941454 | 2016-08-08 12:24:59 -0500 | [diff] [blame] | 92 | <all CONFIG_* key/value pairs in the test's generated defconfig>, |
Anas Nashif | 45a9786 | 2019-01-09 08:46:42 -0500 | [diff] [blame] | 93 | <all DT_* key/value pairs in the test's generated device tree file>, |
| 94 | <all CMake key/value pairs in the test's generated CMakeCache.txt file>, |
Javier B Perez | 7941454 | 2016-08-08 12:24:59 -0500 | [diff] [blame] | 95 | *<env>: any environment variable available |
Andrew Boie | 3ea7892 | 2016-03-24 14:46:00 -0700 | [diff] [blame] | 96 | } |
| 97 | |
| 98 | The grammar for the expression language is as follows: |
| 99 | |
| 100 | expression ::= expression "and" expression |
| 101 | | expression "or" expression |
| 102 | | "not" expression |
| 103 | | "(" expression ")" |
| 104 | | symbol "==" constant |
| 105 | | symbol "!=" constant |
| 106 | | symbol "<" number |
| 107 | | symbol ">" number |
| 108 | | symbol ">=" number |
| 109 | | symbol "<=" number |
| 110 | | symbol "in" list |
Andrew Boie | 7a87f9f | 2016-06-02 12:27:54 -0700 | [diff] [blame] | 111 | | symbol ":" string |
Andrew Boie | 3ea7892 | 2016-03-24 14:46:00 -0700 | [diff] [blame] | 112 | | symbol |
| 113 | |
| 114 | list ::= "[" list_contents "]" |
| 115 | |
| 116 | list_contents ::= constant |
| 117 | | list_contents "," constant |
| 118 | |
| 119 | constant ::= number |
| 120 | | string |
| 121 | |
| 122 | |
| 123 | For the case where expression ::= symbol, it evaluates to true |
| 124 | if the symbol is defined to a non-empty string. |
| 125 | |
| 126 | Operator precedence, starting from lowest to highest: |
| 127 | |
| 128 | or (left associative) |
| 129 | and (left associative) |
| 130 | not (right associative) |
| 131 | all comparison operators (non-associative) |
| 132 | |
| 133 | arch_whitelist, arch_exclude, platform_whitelist, platform_exclude |
| 134 | are all syntactic sugar for these expressions. For instance |
| 135 | |
| 136 | arch_exclude = x86 arc |
| 137 | |
| 138 | Is the same as: |
| 139 | |
| 140 | filter = not ARCH in ["x86", "arc"] |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 141 | |
Andrew Boie | 7a87f9f | 2016-06-02 12:27:54 -0700 | [diff] [blame] | 142 | The ':' operator compiles the string argument as a regular expression, |
| 143 | and then returns a true value only if the symbol's value in the environment |
Anas Nashif | 578ae40 | 2019-07-12 07:54:35 -0700 | [diff] [blame] | 144 | matches. For example, if CONFIG_SOC="stm32f107xc" then |
Andrew Boie | 7a87f9f | 2016-06-02 12:27:54 -0700 | [diff] [blame] | 145 | |
Anas Nashif | 578ae40 | 2019-07-12 07:54:35 -0700 | [diff] [blame] | 146 | filter = CONFIG_SOC : "stm.*" |
Andrew Boie | 7a87f9f | 2016-06-02 12:27:54 -0700 | [diff] [blame] | 147 | |
| 148 | Would match it. |
| 149 | |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 150 | The set of test cases that actually run depends on directives in the testcase |
| 151 | filed and options passed in on the command line. If there is any confusion, |
Anas Nashif | 12d8cce | 2019-11-20 03:47:27 -0800 | [diff] [blame] | 152 | running with -v or examining the discard report (sanitycheck_discard.csv) |
| 153 | can help show why particular test cases were skipped. |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 154 | |
| 155 | Metrics (such as pass/fail state and binary size) for the last code |
| 156 | release are stored in scripts/sanity_chk/sanity_last_release.csv. |
| 157 | To update this, pass the --all --release options. |
| 158 | |
Genaro Saucedo Tejada | 28bba92 | 2016-10-24 18:00:58 -0500 | [diff] [blame] | 159 | To load arguments from a file, write '+' before the file name, e.g., |
| 160 | +file_name. File content must be one or more valid arguments separated by |
| 161 | line break instead of white spaces. |
| 162 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 163 | Most everyday users will run with no arguments. |
Andy Ross | dc4151f | 2019-01-03 14:17:43 -0800 | [diff] [blame] | 164 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 165 | """ |
| 166 | |
Sebastian Bøe | 56d7471 | 2019-01-21 15:48:46 +0100 | [diff] [blame] | 167 | import os |
Anas Nashif | aae71d7 | 2018-04-21 22:26:48 -0500 | [diff] [blame] | 168 | import contextlib |
Anas Nashif | a4c368e | 2018-10-15 09:45:59 -0400 | [diff] [blame] | 169 | import string |
Anas Nashif | aae71d7 | 2018-04-21 22:26:48 -0500 | [diff] [blame] | 170 | import mmap |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 171 | import argparse |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 172 | import sys |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 173 | import re |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 174 | import subprocess |
| 175 | import multiprocessing |
| 176 | import select |
| 177 | import shutil |
Marc Herbert | af1090c | 2019-04-30 14:11:29 -0700 | [diff] [blame] | 178 | import shlex |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 179 | import signal |
| 180 | import threading |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 181 | import concurrent.futures |
| 182 | from threading import BoundedSemaphore |
| 183 | import queue |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 184 | import time |
Anas Nashif | 654ec598 | 2019-04-11 08:38:21 -0400 | [diff] [blame] | 185 | import datetime |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 186 | import csv |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 187 | import yaml |
Andrew Boie | 5d4eb78 | 2015-10-02 10:04:56 -0700 | [diff] [blame] | 188 | import glob |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 189 | import serial |
Daniel Leung | 6b17007 | 2016-04-07 12:10:25 -0700 | [diff] [blame] | 190 | import concurrent |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 191 | import xml.etree.ElementTree as ET |
Anas Nashif | 035799f | 2017-05-13 21:31:53 -0400 | [diff] [blame] | 192 | from collections import OrderedDict |
| 193 | from itertools import islice |
Anas Nashif | e24350c | 2018-07-11 15:09:22 -0500 | [diff] [blame] | 194 | from pathlib import Path |
| 195 | from distutils.spawn import find_executable |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 196 | |
Kumar Gala | 7733b94 | 2019-09-12 17:08:43 -0500 | [diff] [blame] | 197 | ZEPHYR_BASE = os.getenv("ZEPHYR_BASE") |
| 198 | if not ZEPHYR_BASE: |
| 199 | sys.exit("$ZEPHYR_BASE environment variable undefined") |
| 200 | |
| 201 | sys.path.insert(0, os.path.join(ZEPHYR_BASE, "scripts", "dts")) |
| 202 | import edtlib |
| 203 | |
Inaky Perez-Gonzalez | 662dde6 | 2017-07-24 10:24:35 -0700 | [diff] [blame] | 204 | import logging |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 205 | |
| 206 | |
| 207 | hw_map_local = threading.Lock() |
Anas Nashif | c1ea452 | 2019-10-11 07:32:45 -0700 | [diff] [blame] | 208 | report_lock = threading.Lock() |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 209 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 210 | |
Inaky Perez-Gonzalez | 662dde6 | 2017-07-24 10:24:35 -0700 | [diff] [blame] | 211 | log_format = "%(levelname)s %(name)s::%(module)s.%(funcName)s():%(lineno)d: %(message)s" |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 212 | logging.basicConfig(format=log_format, level=30) |
Inaky Perez-Gonzalez | 662dde6 | 2017-07-24 10:24:35 -0700 | [diff] [blame] | 213 | |
Marc Herbert | 1c8632c | 2019-04-15 17:58:45 -0700 | [diff] [blame] | 214 | # Use this for internal comparisons; that's what canonicalization is |
| 215 | # for. Don't use it when invoking other components of the build system |
| 216 | # to avoid confusing and hard to trace inconsistencies in error messages |
| 217 | # and logs, generated Makefiles, etc. compared to when users invoke these |
| 218 | # components directly. |
| 219 | # Note "normalization" is different from canonicalization, see os.path. |
| 220 | canonical_zephyr_base = os.path.realpath(ZEPHYR_BASE) |
| 221 | |
Andrew Boie | 3ea7892 | 2016-03-24 14:46:00 -0700 | [diff] [blame] | 222 | sys.path.insert(0, os.path.join(ZEPHYR_BASE, "scripts/")) |
| 223 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 224 | from sanity_chk import scl |
| 225 | from sanity_chk import expr_parser |
| 226 | |
Andrew Boie | 3ea7892 | 2016-03-24 14:46:00 -0700 | [diff] [blame] | 227 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 228 | VERBOSE = 0 |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 229 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 230 | RELEASE_DATA = os.path.join(ZEPHYR_BASE, "scripts", "sanity_chk", |
| 231 | "sanity_last_release.csv") |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 232 | |
| 233 | if os.isatty(sys.stdout.fileno()): |
| 234 | TERMINAL = True |
| 235 | COLOR_NORMAL = '\033[0m' |
| 236 | COLOR_RED = '\033[91m' |
| 237 | COLOR_GREEN = '\033[92m' |
| 238 | COLOR_YELLOW = '\033[93m' |
| 239 | else: |
| 240 | TERMINAL = False |
| 241 | COLOR_NORMAL = "" |
| 242 | COLOR_RED = "" |
| 243 | COLOR_GREEN = "" |
| 244 | COLOR_YELLOW = "" |
| 245 | |
Anas Nashif | 45a9786 | 2019-01-09 08:46:42 -0500 | [diff] [blame] | 246 | class CMakeCacheEntry: |
| 247 | '''Represents a CMake cache entry. |
| 248 | |
| 249 | This class understands the type system in a CMakeCache.txt, and |
| 250 | converts the following cache types to Python types: |
| 251 | |
| 252 | Cache Type Python type |
| 253 | ---------- ------------------------------------------- |
| 254 | FILEPATH str |
| 255 | PATH str |
| 256 | STRING str OR list of str (if ';' is in the value) |
| 257 | BOOL bool |
| 258 | INTERNAL str OR list of str (if ';' is in the value) |
| 259 | ---------- ------------------------------------------- |
| 260 | ''' |
| 261 | |
| 262 | # Regular expression for a cache entry. |
| 263 | # |
| 264 | # CMake variable names can include escape characters, allowing a |
| 265 | # wider set of names than is easy to match with a regular |
| 266 | # expression. To be permissive here, use a non-greedy match up to |
| 267 | # the first colon (':'). This breaks if the variable name has a |
| 268 | # colon inside, but it's good enough. |
| 269 | CACHE_ENTRY = re.compile( |
| 270 | r'''(?P<name>.*?) # name |
| 271 | :(?P<type>FILEPATH|PATH|STRING|BOOL|INTERNAL) # type |
| 272 | =(?P<value>.*) # value |
| 273 | ''', re.X) |
| 274 | |
| 275 | @classmethod |
| 276 | def _to_bool(cls, val): |
| 277 | # Convert a CMake BOOL string into a Python bool. |
| 278 | # |
| 279 | # "True if the constant is 1, ON, YES, TRUE, Y, or a |
| 280 | # non-zero number. False if the constant is 0, OFF, NO, |
| 281 | # FALSE, N, IGNORE, NOTFOUND, the empty string, or ends in |
| 282 | # the suffix -NOTFOUND. Named boolean constants are |
| 283 | # case-insensitive. If the argument is not one of these |
| 284 | # constants, it is treated as a variable." |
| 285 | # |
| 286 | # https://cmake.org/cmake/help/v3.0/command/if.html |
| 287 | val = val.upper() |
| 288 | if val in ('ON', 'YES', 'TRUE', 'Y'): |
| 289 | return 1 |
| 290 | elif val in ('OFF', 'NO', 'FALSE', 'N', 'IGNORE', 'NOTFOUND', ''): |
| 291 | return 0 |
| 292 | elif val.endswith('-NOTFOUND'): |
| 293 | return 0 |
| 294 | else: |
| 295 | try: |
| 296 | v = int(val) |
| 297 | return v != 0 |
| 298 | except ValueError as exc: |
| 299 | raise ValueError('invalid bool {}'.format(val)) from exc |
| 300 | |
| 301 | @classmethod |
| 302 | def from_line(cls, line, line_no): |
| 303 | # Comments can only occur at the beginning of a line. |
| 304 | # (The value of an entry could contain a comment character). |
| 305 | if line.startswith('//') or line.startswith('#'): |
| 306 | return None |
| 307 | |
| 308 | # Whitespace-only lines do not contain cache entries. |
| 309 | if not line.strip(): |
| 310 | return None |
| 311 | |
| 312 | m = cls.CACHE_ENTRY.match(line) |
| 313 | if not m: |
| 314 | return None |
| 315 | |
| 316 | name, type_, value = (m.group(g) for g in ('name', 'type', 'value')) |
| 317 | if type_ == 'BOOL': |
| 318 | try: |
| 319 | value = cls._to_bool(value) |
| 320 | except ValueError as exc: |
| 321 | args = exc.args + ('on line {}: {}'.format(line_no, line),) |
| 322 | raise ValueError(args) from exc |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 323 | elif type_ in ['STRING','INTERNAL']: |
Anas Nashif | 45a9786 | 2019-01-09 08:46:42 -0500 | [diff] [blame] | 324 | # If the value is a CMake list (i.e. is a string which |
| 325 | # contains a ';'), convert to a Python list. |
| 326 | if ';' in value: |
| 327 | value = value.split(';') |
| 328 | |
| 329 | return CMakeCacheEntry(name, value) |
| 330 | |
| 331 | def __init__(self, name, value): |
| 332 | self.name = name |
| 333 | self.value = value |
| 334 | |
| 335 | def __str__(self): |
| 336 | fmt = 'CMakeCacheEntry(name={}, value={})' |
| 337 | return fmt.format(self.name, self.value) |
| 338 | |
| 339 | |
| 340 | class CMakeCache: |
| 341 | '''Parses and represents a CMake cache file.''' |
| 342 | |
| 343 | @staticmethod |
| 344 | def from_file(cache_file): |
| 345 | return CMakeCache(cache_file) |
| 346 | |
| 347 | def __init__(self, cache_file): |
| 348 | self.cache_file = cache_file |
| 349 | self.load(cache_file) |
| 350 | |
| 351 | def load(self, cache_file): |
| 352 | entries = [] |
| 353 | with open(cache_file, 'r') as cache: |
| 354 | for line_no, line in enumerate(cache): |
| 355 | entry = CMakeCacheEntry.from_line(line, line_no) |
| 356 | if entry: |
| 357 | entries.append(entry) |
| 358 | self._entries = OrderedDict((e.name, e) for e in entries) |
| 359 | |
| 360 | def get(self, name, default=None): |
| 361 | entry = self._entries.get(name) |
| 362 | if entry is not None: |
| 363 | return entry.value |
| 364 | else: |
| 365 | return default |
| 366 | |
| 367 | def get_list(self, name, default=None): |
| 368 | if default is None: |
| 369 | default = [] |
| 370 | entry = self._entries.get(name) |
| 371 | if entry is not None: |
| 372 | value = entry.value |
| 373 | if isinstance(value, list): |
| 374 | return value |
| 375 | elif isinstance(value, str): |
| 376 | return [value] if value else [] |
| 377 | else: |
| 378 | msg = 'invalid value {} type {}' |
| 379 | raise RuntimeError(msg.format(value, type(value))) |
| 380 | else: |
| 381 | return default |
| 382 | |
| 383 | def __contains__(self, name): |
| 384 | return name in self._entries |
| 385 | |
| 386 | def __getitem__(self, name): |
| 387 | return self._entries[name].value |
| 388 | |
| 389 | def __setitem__(self, name, entry): |
| 390 | if not isinstance(entry, CMakeCacheEntry): |
| 391 | msg = 'improper type {} for value {}, expecting CMakeCacheEntry' |
| 392 | raise TypeError(msg.format(type(entry), entry)) |
| 393 | self._entries[name] = entry |
| 394 | |
| 395 | def __delitem__(self, name): |
| 396 | del self._entries[name] |
| 397 | |
| 398 | def __iter__(self): |
| 399 | return iter(self._entries.values()) |
| 400 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 401 | class SanityCheckException(Exception): |
| 402 | pass |
| 403 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 404 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 405 | class SanityRuntimeError(SanityCheckException): |
| 406 | pass |
| 407 | |
| 408 | class ConfigurationError(SanityCheckException): |
| 409 | def __init__(self, cfile, message): |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 410 | SanityCheckException.__init__(self, cfile + ": " + message) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 411 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 412 | class BuildError(SanityCheckException): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 413 | pass |
| 414 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 415 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 416 | class ExecutionError(SanityCheckException): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 417 | pass |
| 418 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 419 | |
Inaky Perez-Gonzalez | 9a36cb6 | 2016-11-29 10:43:40 -0800 | [diff] [blame] | 420 | log_file = None |
| 421 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 422 | # Debug Functions |
Anas Nashif | 654ec598 | 2019-04-11 08:38:21 -0400 | [diff] [blame] | 423 | def info(what, show_time=True): |
| 424 | if options.timestamps and show_time: |
| 425 | date = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") |
| 426 | what = "{}: {}".format(date, what) |
Andrew Boie | 08ce5a5 | 2016-02-22 13:28:10 -0800 | [diff] [blame] | 427 | sys.stdout.write(what + "\n") |
Paul Sokolovsky | 3ba0876 | 2017-10-27 14:53:24 +0300 | [diff] [blame] | 428 | sys.stdout.flush() |
Inaky Perez-Gonzalez | 9a36cb6 | 2016-11-29 10:43:40 -0800 | [diff] [blame] | 429 | if log_file: |
| 430 | log_file.write(what + "\n") |
| 431 | log_file.flush() |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 432 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 433 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 434 | def error(what): |
Anas Nashif | 654ec598 | 2019-04-11 08:38:21 -0400 | [diff] [blame] | 435 | if options.timestamps: |
| 436 | date = datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S") |
| 437 | what = "{}: {}".format(date, what) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 438 | sys.stderr.write(COLOR_RED + what + COLOR_NORMAL + "\n") |
Inaky Perez-Gonzalez | 9a36cb6 | 2016-11-29 10:43:40 -0800 | [diff] [blame] | 439 | if log_file: |
| 440 | log_file(what + "\n") |
| 441 | log_file.flush() |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 442 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 443 | |
Andrew Boie | 08ce5a5 | 2016-02-22 13:28:10 -0800 | [diff] [blame] | 444 | def debug(what): |
| 445 | if VERBOSE >= 1: |
| 446 | info(what) |
| 447 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 448 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 449 | def verbose(what): |
| 450 | if VERBOSE >= 2: |
Andrew Boie | 08ce5a5 | 2016-02-22 13:28:10 -0800 | [diff] [blame] | 451 | info(what) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 452 | |
Anas Nashif | 576be98 | 2017-12-23 20:20:27 -0500 | [diff] [blame] | 453 | class HarnessImporter: |
| 454 | |
| 455 | def __init__(self, name): |
| 456 | sys.path.insert(0, os.path.join(ZEPHYR_BASE, "scripts/sanity_chk")) |
| 457 | module = __import__("harness") |
| 458 | if name: |
| 459 | my_class = getattr(module, name) |
| 460 | else: |
| 461 | my_class = getattr(module, "Test") |
| 462 | |
| 463 | self.instance = my_class() |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 464 | |
Jaakko Hannikainen | ca505f8 | 2016-08-22 15:03:46 +0300 | [diff] [blame] | 465 | class Handler: |
Anas Nashif | d18ec53 | 2019-04-11 23:20:39 -0400 | [diff] [blame] | 466 | def __init__(self, instance, type_str="build"): |
Jaakko Hannikainen | ca505f8 | 2016-08-22 15:03:46 +0300 | [diff] [blame] | 467 | """Constructor |
| 468 | |
Jaakko Hannikainen | ca505f8 | 2016-08-22 15:03:46 +0300 | [diff] [blame] | 469 | """ |
| 470 | self.lock = threading.Lock() |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 471 | |
Jaakko Hannikainen | ca505f8 | 2016-08-22 15:03:46 +0300 | [diff] [blame] | 472 | self.state = "waiting" |
Anas Nashif | 1377375 | 2018-07-06 18:20:23 -0500 | [diff] [blame] | 473 | self.run = False |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 474 | self.duration = 0 |
Anas Nashif | d18ec53 | 2019-04-11 23:20:39 -0400 | [diff] [blame] | 475 | self.type_str = type_str |
Jaakko Hannikainen | ca505f8 | 2016-08-22 15:03:46 +0300 | [diff] [blame] | 476 | |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 477 | self.binary = None |
Jan Kowalewski | 265895b | 2019-01-07 16:40:24 +0100 | [diff] [blame] | 478 | self.pid_fn = None |
Anas Nashif | 99f5a6c | 2018-07-07 08:45:53 -0500 | [diff] [blame] | 479 | self.call_make_run = False |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 480 | |
Anas Nashif | d3384fb | 2018-02-22 06:44:16 -0600 | [diff] [blame] | 481 | self.name = instance.name |
| 482 | self.instance = instance |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 483 | self.timeout = instance.testcase.timeout |
| 484 | self.sourcedir = instance.testcase.source_dir |
| 485 | self.build_dir = instance.build_dir |
| 486 | self.log = os.path.join(self.build_dir, "handler.log") |
Anas Nashif | d3384fb | 2018-02-22 06:44:16 -0600 | [diff] [blame] | 487 | self.returncode = 0 |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 488 | self.set_state("running", self.duration) |
Anas Nashif | d3384fb | 2018-02-22 06:44:16 -0600 | [diff] [blame] | 489 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 490 | self.args = [] |
| 491 | |
| 492 | def set_state(self, state, duration): |
Jaakko Hannikainen | ca505f8 | 2016-08-22 15:03:46 +0300 | [diff] [blame] | 493 | self.lock.acquire() |
| 494 | self.state = state |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 495 | self.duration = duration |
Jaakko Hannikainen | ca505f8 | 2016-08-22 15:03:46 +0300 | [diff] [blame] | 496 | self.lock.release() |
| 497 | |
| 498 | def get_state(self): |
| 499 | self.lock.acquire() |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 500 | ret = (self.state, self.duration) |
Jaakko Hannikainen | ca505f8 | 2016-08-22 15:03:46 +0300 | [diff] [blame] | 501 | self.lock.release() |
| 502 | return ret |
| 503 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 504 | def record(self, harness): |
| 505 | if harness.recording: |
| 506 | filename = os.path.join(options.outdir, |
| 507 | self.instance.platform.name, |
| 508 | self.instance.testcase.name, "recording.csv") |
| 509 | with open(filename, "at") as csvfile: |
| 510 | cw = csv.writer(csvfile, harness.fieldnames, lineterminator=os.linesep) |
| 511 | cw.writerow(harness.fieldnames) |
| 512 | for instance in harness.recording: |
| 513 | cw.writerow(instance) |
| 514 | |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 515 | class BinaryHandler(Handler): |
Anas Nashif | d18ec53 | 2019-04-11 23:20:39 -0400 | [diff] [blame] | 516 | def __init__(self, instance, type_str): |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 517 | """Constructor |
| 518 | |
| 519 | @param instance Test Instance |
| 520 | """ |
Anas Nashif | d18ec53 | 2019-04-11 23:20:39 -0400 | [diff] [blame] | 521 | super().__init__(instance, type_str) |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 522 | |
Alberto Escolar Piedras | 661bc82 | 2018-12-11 15:13:05 +0100 | [diff] [blame] | 523 | self.terminated = False |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 524 | |
Jan Kowalewski | 265895b | 2019-01-07 16:40:24 +0100 | [diff] [blame] | 525 | def try_kill_process_by_pid(self): |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 526 | if self.pid_fn: |
Jan Kowalewski | 265895b | 2019-01-07 16:40:24 +0100 | [diff] [blame] | 527 | pid = int(open(self.pid_fn).read()) |
| 528 | os.unlink(self.pid_fn) |
| 529 | self.pid_fn = None # clear so we don't try to kill the binary twice |
| 530 | try: |
| 531 | os.kill(pid, signal.SIGTERM) |
| 532 | except ProcessLookupError: |
| 533 | pass |
| 534 | |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 535 | def _output_reader(self, proc, harness): |
| 536 | log_out_fp = open(self.log, "wt") |
| 537 | for line in iter(proc.stdout.readline, b''): |
| 538 | verbose("OUTPUT: {0}".format(line.decode('utf-8').rstrip())) |
| 539 | log_out_fp.write(line.decode('utf-8')) |
| 540 | log_out_fp.flush() |
| 541 | harness.handle(line.decode('utf-8').rstrip()) |
| 542 | if harness.state: |
Alberto Escolar Piedras | 661bc82 | 2018-12-11 15:13:05 +0100 | [diff] [blame] | 543 | try: |
| 544 | #POSIX arch based ztests end on their own, |
| 545 | #so let's give it up to 100ms to do so |
| 546 | proc.wait(0.1) |
| 547 | except subprocess.TimeoutExpired: |
| 548 | proc.terminate() |
| 549 | self.terminated = True |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 550 | break |
| 551 | |
| 552 | log_out_fp.close() |
| 553 | |
| 554 | def handle(self): |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 555 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 556 | harness_name = self.instance.testcase.harness.capitalize() |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 557 | harness_import = HarnessImporter(harness_name) |
| 558 | harness = harness_import.instance |
| 559 | harness.configure(self.instance) |
| 560 | |
Anas Nashif | 99f5a6c | 2018-07-07 08:45:53 -0500 | [diff] [blame] | 561 | if self.call_make_run: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 562 | command = [get_generator()[0], "run"] |
Anas Nashif | 99f5a6c | 2018-07-07 08:45:53 -0500 | [diff] [blame] | 563 | else: |
| 564 | command = [self.binary] |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 565 | |
Anas Nashif | c1ea452 | 2019-10-11 07:32:45 -0700 | [diff] [blame] | 566 | run_valgrind = False |
| 567 | if options.enable_valgrind and shutil.which("valgrind"): |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 568 | command = ["valgrind", "--error-exitcode=2", |
Alberto Escolar Piedras | 3980e0c | 2018-12-12 17:41:04 +0100 | [diff] [blame] | 569 | "--leak-check=full", |
| 570 | "--suppressions="+ZEPHYR_BASE+"/scripts/valgrind.supp", |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 571 | "--log-file="+self.build_dir+"/valgrind.log" |
Alberto Escolar Piedras | 3980e0c | 2018-12-12 17:41:04 +0100 | [diff] [blame] | 572 | ] + command |
Anas Nashif | c1ea452 | 2019-10-11 07:32:45 -0700 | [diff] [blame] | 573 | run_valgrind = True |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 574 | |
Marc Herbert | af1090c | 2019-04-30 14:11:29 -0700 | [diff] [blame] | 575 | verbose("Spawning process: " + |
Jan Van Winkel | 6b9e160 | 2019-01-03 19:18:35 +0100 | [diff] [blame] | 576 | " ".join(shlex.quote(word) for word in command) + os.linesep + |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 577 | "Spawning process in directory: " + self.build_dir) |
Alberto Escolar Piedras | 649368c | 2019-07-02 09:59:35 +0200 | [diff] [blame] | 578 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 579 | start_time = time.time() |
Alberto Escolar Piedras | 649368c | 2019-07-02 09:59:35 +0200 | [diff] [blame] | 580 | |
Anas Nashif | 89c8304 | 2019-11-05 05:55:39 -0800 | [diff] [blame] | 581 | env = os.environ.copy() |
Jan Van Winkel | 21212f3 | 2019-09-12 00:03:35 +0200 | [diff] [blame] | 582 | if options.enable_asan: |
Jan Van Winkel | 21212f3 | 2019-09-12 00:03:35 +0200 | [diff] [blame] | 583 | env["ASAN_OPTIONS"] = "log_path=stdout:" + \ |
| 584 | env.get("ASAN_OPTIONS", "") |
| 585 | if not options.enable_lsan: |
| 586 | env["ASAN_OPTIONS"] += "detect_leaks=0" |
| 587 | with subprocess.Popen(command, stdout=subprocess.PIPE, |
| 588 | stderr=subprocess.PIPE, cwd=self.build_dir, env=env) as proc: |
Marc Herbert | af1090c | 2019-04-30 14:11:29 -0700 | [diff] [blame] | 589 | verbose("Spawning BinaryHandler Thread for %s" % self.name) |
Flavio Ceolin | 063ab90 | 2019-10-15 16:10:49 -0700 | [diff] [blame] | 590 | t = threading.Thread(target=self._output_reader, args=(proc, harness, ), daemon=True) |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 591 | t.start() |
| 592 | t.join(self.timeout) |
| 593 | if t.is_alive(): |
Jan Kowalewski | 265895b | 2019-01-07 16:40:24 +0100 | [diff] [blame] | 594 | self.try_kill_process_by_pid() |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 595 | proc.terminate() |
Alberto Escolar Piedras | 661bc82 | 2018-12-11 15:13:05 +0100 | [diff] [blame] | 596 | self.terminated = True |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 597 | t.join() |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 598 | proc.wait() |
| 599 | self.returncode = proc.returncode |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 600 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 601 | handler_time = time.time() - start_time |
Alberto Escolar Piedras | 649368c | 2019-07-02 09:59:35 +0200 | [diff] [blame] | 602 | |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 603 | if options.enable_coverage: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 604 | subprocess.call(["GCOV_PREFIX=" + self.build_dir, |
| 605 | "gcov", self.sourcedir, "-b", "-s", self.build_dir], shell=True) |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 606 | |
Jan Kowalewski | 265895b | 2019-01-07 16:40:24 +0100 | [diff] [blame] | 607 | self.try_kill_process_by_pid() |
| 608 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 609 | # FIXME: This is needed when killing the simulator, the console is |
Anas Nashif | 99f5a6c | 2018-07-07 08:45:53 -0500 | [diff] [blame] | 610 | # garbled and needs to be reset. Did not find a better way to do that. |
| 611 | |
| 612 | subprocess.call(["stty", "sane"]) |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 613 | self.instance.results = harness.tests |
Anas Nashif | c1ea452 | 2019-10-11 07:32:45 -0700 | [diff] [blame] | 614 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 615 | if not self.terminated and self.returncode != 0: |
Alberto Escolar Piedras | 661bc82 | 2018-12-11 15:13:05 +0100 | [diff] [blame] | 616 | #When a process is killed, the default handler returns 128 + SIGTERM |
| 617 | #so in that case the return code itself is not meaningful |
Anas Nashif | f72d190 | 2019-10-18 17:20:44 -0400 | [diff] [blame] | 618 | self.set_state("failed", handler_time) |
Anas Nashif | c1ea452 | 2019-10-11 07:32:45 -0700 | [diff] [blame] | 619 | self.instance.reason = "Handler Error" |
| 620 | elif run_valgrind and self.returncode == 2: |
| 621 | self.set_state("failed", handler_time) |
| 622 | self.instance.reason = "Valgrind error" |
Alberto Escolar Piedras | 661bc82 | 2018-12-11 15:13:05 +0100 | [diff] [blame] | 623 | elif harness.state: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 624 | self.set_state(harness.state, handler_time) |
Anas Nashif | df7ee61 | 2018-07-07 06:09:01 -0500 | [diff] [blame] | 625 | else: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 626 | self.set_state("timeout", handler_time) |
| 627 | self.instance.reason = "Handler timeout" |
| 628 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 629 | self.record(harness) |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 630 | |
| 631 | class DeviceHandler(Handler): |
| 632 | |
Anas Nashif | d18ec53 | 2019-04-11 23:20:39 -0400 | [diff] [blame] | 633 | def __init__(self, instance, type_str): |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 634 | """Constructor |
| 635 | |
| 636 | @param instance Test Instance |
| 637 | """ |
Anas Nashif | d18ec53 | 2019-04-11 23:20:39 -0400 | [diff] [blame] | 638 | super().__init__(instance, type_str) |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 639 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 640 | self.suite = None |
| 641 | |
Marti Bolivar | 5591ca2 | 2019-02-07 15:53:39 -0700 | [diff] [blame] | 642 | def monitor_serial(self, ser, halt_fileno, harness): |
Anas Nashif | 4a9f3e6 | 2018-07-06 18:58:18 -0500 | [diff] [blame] | 643 | log_out_fp = open(self.log, "wt") |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 644 | |
Marti Bolivar | 5591ca2 | 2019-02-07 15:53:39 -0700 | [diff] [blame] | 645 | ser_fileno = ser.fileno() |
| 646 | readlist = [halt_fileno, ser_fileno] |
| 647 | |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 648 | while ser.isOpen(): |
Marti Bolivar | 5591ca2 | 2019-02-07 15:53:39 -0700 | [diff] [blame] | 649 | readable, _, _ = select.select(readlist, [], [], self.timeout) |
| 650 | |
| 651 | if halt_fileno in readable: |
| 652 | verbose('halted') |
| 653 | ser.close() |
| 654 | break |
| 655 | if ser_fileno not in readable: |
| 656 | continue # Timeout. |
| 657 | |
| 658 | serial_line = None |
Anas Nashif | 61e2163 | 2018-04-08 13:30:16 -0500 | [diff] [blame] | 659 | try: |
| 660 | serial_line = ser.readline() |
| 661 | except TypeError: |
| 662 | pass |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 663 | except serial.SerialException: |
Anas Nashif | 5923760 | 2019-03-03 10:36:35 -0500 | [diff] [blame] | 664 | ser.close() |
| 665 | break |
Anas Nashif | 61e2163 | 2018-04-08 13:30:16 -0500 | [diff] [blame] | 666 | |
Marti Bolivar | 5591ca2 | 2019-02-07 15:53:39 -0700 | [diff] [blame] | 667 | # Just because ser_fileno has data doesn't mean an entire line |
| 668 | # is available yet. |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 669 | if serial_line: |
Anas Nashif | d3384fb | 2018-02-22 06:44:16 -0600 | [diff] [blame] | 670 | sl = serial_line.decode('utf-8', 'ignore') |
| 671 | verbose("DEVICE: {0}".format(sl.rstrip())) |
| 672 | |
| 673 | log_out_fp.write(sl) |
| 674 | log_out_fp.flush() |
| 675 | harness.handle(sl.rstrip()) |
Marti Bolivar | 5591ca2 | 2019-02-07 15:53:39 -0700 | [diff] [blame] | 676 | |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 677 | if harness.state: |
| 678 | ser.close() |
| 679 | break |
| 680 | |
| 681 | log_out_fp.close() |
| 682 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 683 | def device_is_available(self, device): |
| 684 | for i in self.suite.connected_hardware: |
| 685 | if i['platform'] == device and i['available'] and i['connected']: |
| 686 | return True |
| 687 | |
| 688 | return False |
| 689 | |
| 690 | def get_available_device(self, device): |
| 691 | for i in self.suite.connected_hardware: |
| 692 | if i['platform'] == device and i['available']: |
| 693 | i['available'] = False |
| 694 | i['counter'] += 1 |
| 695 | return i |
| 696 | |
| 697 | return None |
| 698 | |
| 699 | def make_device_available(self, serial): |
| 700 | with hw_map_local: |
| 701 | for i in self.suite.connected_hardware: |
| 702 | if i['serial'] == serial: |
| 703 | i['available'] = True |
| 704 | |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 705 | def handle(self): |
| 706 | out_state = "failed" |
| 707 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 708 | if options.west_flash: |
| 709 | command = ["west", "flash", "--skip-rebuild", "-d", self.build_dir] |
| 710 | if options.west_runner: |
Michael Scott | 421ce46 | 2019-06-18 09:37:46 -0700 | [diff] [blame] | 711 | command.append("--runner") |
| 712 | command.append(options.west_runner) |
Jorgen Kvalvaag | d50fb31 | 2019-09-02 15:25:20 +0200 | [diff] [blame] | 713 | # There are three ways this option is used. |
Andy Doan | 79c4884 | 2019-02-08 10:09:04 -0600 | [diff] [blame] | 714 | # 1) bare: --west-flash |
| 715 | # This results in options.west_flash == [] |
| 716 | # 2) with a value: --west-flash="--board-id=42" |
| 717 | # This results in options.west_flash == "--board-id=42" |
Jorgen Kvalvaag | d50fb31 | 2019-09-02 15:25:20 +0200 | [diff] [blame] | 718 | # 3) Multiple values: --west-flash="--board-id=42,--erase" |
| 719 | # This results in options.west_flash == "--board-id=42 --erase" |
Andy Doan | 79c4884 | 2019-02-08 10:09:04 -0600 | [diff] [blame] | 720 | if options.west_flash != []: |
| 721 | command.append('--') |
Jorgen Kvalvaag | d50fb31 | 2019-09-02 15:25:20 +0200 | [diff] [blame] | 722 | command.extend(options.west_flash.split(',')) |
Anas Nashif | d3384fb | 2018-02-22 06:44:16 -0600 | [diff] [blame] | 723 | else: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 724 | command = [get_generator()[0], "-C", self.build_dir, "flash"] |
Anas Nashif | d3384fb | 2018-02-22 06:44:16 -0600 | [diff] [blame] | 725 | |
Anas Nashif | d3384fb | 2018-02-22 06:44:16 -0600 | [diff] [blame] | 726 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 727 | while not self.device_is_available(self.instance.platform.name): |
| 728 | time.sleep(1) |
| 729 | |
| 730 | hardware = self.get_available_device(self.instance.platform.name) |
| 731 | |
| 732 | runner = hardware.get('runner', None) |
| 733 | if runner: |
Peter Bigot | da73848 | 2019-11-21 11:55:26 -0600 | [diff] [blame] | 734 | board_id = hardware.get("probe_id", hardware.get("id", None)) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 735 | product = hardware.get("product", None) |
| 736 | command = ["west", "flash", "--skip-rebuild", "-d", self.build_dir] |
| 737 | command.append("--runner") |
| 738 | command.append(hardware.get('runner', None)) |
| 739 | if runner == "pyocd": |
| 740 | command.append("--board-id") |
| 741 | command.append(board_id) |
| 742 | elif runner == "nrfjprog": |
| 743 | command.append('--') |
| 744 | command.append("--snr") |
| 745 | command.append(board_id) |
| 746 | elif runner == "openocd" and product == "STM32 STLink": |
| 747 | command.append('--') |
| 748 | command.append("--cmd-pre-init") |
| 749 | command.append("hla_serial %s" %(board_id)) |
| 750 | elif runner == "openocd" and product == "EDBG CMSIS-DAP": |
| 751 | command.append('--') |
| 752 | command.append("--cmd-pre-init") |
| 753 | command.append("cmsis_dap_serial %s" %(board_id)) |
| 754 | elif runner == "jlink": |
| 755 | command.append("--tool-opt=-SelectEmuBySN %s" %(board_id)) |
| 756 | |
| 757 | serial_device = hardware['serial'] |
| 758 | |
| 759 | try: |
| 760 | ser = serial.Serial( |
| 761 | serial_device, |
| 762 | baudrate=115200, |
| 763 | parity=serial.PARITY_NONE, |
| 764 | stopbits=serial.STOPBITS_ONE, |
| 765 | bytesize=serial.EIGHTBITS, |
| 766 | timeout=self.timeout |
| 767 | ) |
| 768 | except serial.SerialException as e: |
| 769 | self.set_state("failed", 0) |
| 770 | error("Serial device err: %s" %(str(e))) |
| 771 | self.make_device_available(serial_device) |
| 772 | return |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 773 | |
| 774 | ser.flush() |
| 775 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 776 | harness_name = self.instance.testcase.harness.capitalize() |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 777 | harness_import = HarnessImporter(harness_name) |
| 778 | harness = harness_import.instance |
| 779 | harness.configure(self.instance) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 780 | read_pipe, write_pipe = os.pipe() |
| 781 | start_time = time.time() |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 782 | |
Marti Bolivar | 5591ca2 | 2019-02-07 15:53:39 -0700 | [diff] [blame] | 783 | t = threading.Thread(target=self.monitor_serial, daemon=True, |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 784 | args=(ser, read_pipe, harness)) |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 785 | t.start() |
| 786 | |
Andy Doan | 79c4884 | 2019-02-08 10:09:04 -0600 | [diff] [blame] | 787 | logging.debug('Flash command: %s', command) |
Anas Nashif | 61e2163 | 2018-04-08 13:30:16 -0500 | [diff] [blame] | 788 | try: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 789 | if VERBOSE and not runner: |
Marti Bolivar | 303b522 | 2019-02-07 15:50:55 -0700 | [diff] [blame] | 790 | subprocess.check_call(command) |
| 791 | else: |
| 792 | subprocess.check_output(command, stderr=subprocess.PIPE) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 793 | |
Anas Nashif | 61e2163 | 2018-04-08 13:30:16 -0500 | [diff] [blame] | 794 | except subprocess.CalledProcessError: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 795 | os.write(write_pipe, b'x') # halt the thread |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 796 | |
| 797 | t.join(self.timeout) |
| 798 | if t.is_alive(): |
| 799 | out_state = "timeout" |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 800 | |
| 801 | if ser.isOpen(): |
| 802 | ser.close() |
| 803 | |
Anas Nashif | d3384fb | 2018-02-22 06:44:16 -0600 | [diff] [blame] | 804 | if out_state == "timeout": |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 805 | for c in self.instance.testcase.cases: |
Anas Nashif | d3384fb | 2018-02-22 06:44:16 -0600 | [diff] [blame] | 806 | if c not in harness.tests: |
| 807 | harness.tests[c] = "BLOCK" |
Anas Nashif | 61e2163 | 2018-04-08 13:30:16 -0500 | [diff] [blame] | 808 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 809 | handler_time = time.time() - start_time |
| 810 | |
Anas Nashif | 61e2163 | 2018-04-08 13:30:16 -0500 | [diff] [blame] | 811 | self.instance.results = harness.tests |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 812 | if harness.state: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 813 | self.set_state(harness.state, handler_time) |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 814 | else: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 815 | self.set_state(out_state, handler_time) |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 816 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 817 | self.make_device_available(serial_device) |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 818 | |
Anas Nashif | 3a00d0c | 2019-11-12 11:07:15 -0500 | [diff] [blame] | 819 | self.record(harness) |
| 820 | |
Jaakko Hannikainen | ca505f8 | 2016-08-22 15:03:46 +0300 | [diff] [blame] | 821 | class QEMUHandler(Handler): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 822 | """Spawns a thread to monitor QEMU output from pipes |
| 823 | |
Anas Nashif | 3ac7b3a | 2017-08-03 10:03:02 -0400 | [diff] [blame] | 824 | We pass QEMU_PIPE to 'make run' and monitor the pipes for output. |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 825 | We need to do this as once qemu starts, it runs forever until killed. |
| 826 | Test cases emit special messages to the console as they run, we check |
| 827 | for these to collect whether the test passed or failed. |
| 828 | """ |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 829 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 830 | |
| 831 | def __init__(self, instance, type_str): |
| 832 | """Constructor |
| 833 | |
| 834 | @param instance Test instance |
| 835 | """ |
| 836 | |
| 837 | super().__init__(instance, type_str) |
| 838 | self.fifo_fn = os.path.join(instance.build_dir, "qemu-fifo") |
| 839 | |
| 840 | self.pid_fn = os.path.join(instance.build_dir, "qemu.pid") |
| 841 | |
| 842 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 843 | @staticmethod |
Anas Nashif | 576be98 | 2017-12-23 20:20:27 -0500 | [diff] [blame] | 844 | def _thread(handler, timeout, outdir, logfile, fifo_fn, pid_fn, results, harness): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 845 | fifo_in = fifo_fn + ".in" |
| 846 | fifo_out = fifo_fn + ".out" |
| 847 | |
| 848 | # These in/out nodes are named from QEMU's perspective, not ours |
| 849 | if os.path.exists(fifo_in): |
| 850 | os.unlink(fifo_in) |
| 851 | os.mkfifo(fifo_in) |
| 852 | if os.path.exists(fifo_out): |
| 853 | os.unlink(fifo_out) |
| 854 | os.mkfifo(fifo_out) |
| 855 | |
| 856 | # We don't do anything with out_fp but we need to open it for |
| 857 | # writing so that QEMU doesn't block, due to the way pipes work |
| 858 | out_fp = open(fifo_in, "wb") |
| 859 | # Disable internal buffering, we don't |
| 860 | # want read() or poll() to ever block if there is data in there |
| 861 | in_fp = open(fifo_out, "rb", buffering=0) |
Andrew Boie | 08ce5a5 | 2016-02-22 13:28:10 -0800 | [diff] [blame] | 862 | log_out_fp = open(logfile, "wt") |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 863 | |
| 864 | start_time = time.time() |
| 865 | timeout_time = start_time + timeout |
| 866 | p = select.poll() |
| 867 | p.register(in_fp, select.POLLIN) |
Anas Nashif | 39ae72b | 2018-08-29 01:45:38 -0400 | [diff] [blame] | 868 | out_state = None |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 869 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 870 | line = "" |
Anas Nashif | 74dbe33 | 2019-01-17 17:11:37 -0500 | [diff] [blame] | 871 | timeout_extended = False |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 872 | while True: |
| 873 | this_timeout = int((timeout_time - time.time()) * 1000) |
| 874 | if this_timeout < 0 or not p.poll(this_timeout): |
Anas Nashif | 39ae72b | 2018-08-29 01:45:38 -0400 | [diff] [blame] | 875 | if not out_state: |
| 876 | out_state = "timeout" |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 877 | break |
| 878 | |
Genaro Saucedo Tejada | 2968b36 | 2016-08-09 15:11:53 -0500 | [diff] [blame] | 879 | try: |
| 880 | c = in_fp.read(1).decode("utf-8") |
| 881 | except UnicodeDecodeError: |
| 882 | # Test is writing something weird, fail |
| 883 | out_state = "unexpected byte" |
| 884 | break |
| 885 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 886 | if c == "": |
| 887 | # EOF, this shouldn't happen unless QEMU crashes |
| 888 | out_state = "unexpected eof" |
| 889 | break |
| 890 | line = line + c |
| 891 | if c != "\n": |
| 892 | continue |
| 893 | |
Jaakko Hannikainen | ca505f8 | 2016-08-22 15:03:46 +0300 | [diff] [blame] | 894 | # line contains a full line of data output from QEMU |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 895 | log_out_fp.write(line) |
| 896 | log_out_fp.flush() |
| 897 | line = line.strip() |
| 898 | verbose("QEMU: %s" % line) |
| 899 | |
Anas Nashif | 576be98 | 2017-12-23 20:20:27 -0500 | [diff] [blame] | 900 | harness.handle(line) |
| 901 | if harness.state: |
Anas Nashif | 39ae72b | 2018-08-29 01:45:38 -0400 | [diff] [blame] | 902 | # if we have registered a fail make sure the state is not |
| 903 | # overridden by a false success message coming from the |
| 904 | # testsuite |
| 905 | if out_state != 'failed': |
| 906 | out_state = harness.state |
| 907 | |
| 908 | # if we get some state, that means test is doing well, we reset |
Marc Herbert | 9e57338 | 2019-07-03 12:49:42 -0700 | [diff] [blame] | 909 | # the timeout and wait for 2 more seconds to catch anything |
| 910 | # printed late. We wait much longer if code |
Andrew Boie | 095b82a | 2019-07-02 17:03:48 -0700 | [diff] [blame] | 911 | # coverage is enabled since dumping this information can |
| 912 | # take some time. |
Anas Nashif | f29087e | 2019-01-25 09:37:38 -0500 | [diff] [blame] | 913 | if not timeout_extended or harness.capture_coverage: |
| 914 | timeout_extended= True |
| 915 | if harness.capture_coverage: |
Andrew Boie | d62e229 | 2019-07-01 15:01:48 -0700 | [diff] [blame] | 916 | timeout_time = time.time() + 30 |
Anas Nashif | f29087e | 2019-01-25 09:37:38 -0500 | [diff] [blame] | 917 | else: |
Anas Nashif | 74dbe33 | 2019-01-17 17:11:37 -0500 | [diff] [blame] | 918 | timeout_time = time.time() + 2 |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 919 | line = "" |
| 920 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 921 | handler.record(harness) |
| 922 | |
| 923 | handler_time = time.time() - start_time |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 924 | verbose("QEMU complete (%s) after %f seconds" % |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 925 | (out_state, handler_time)) |
| 926 | handler.set_state(out_state, handler_time) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 927 | |
| 928 | log_out_fp.close() |
| 929 | out_fp.close() |
| 930 | in_fp.close() |
Anas Nashif | d6476ee | 2019-04-11 11:40:09 -0400 | [diff] [blame] | 931 | if os.path.exists(pid_fn): |
| 932 | pid = int(open(pid_fn).read()) |
| 933 | os.unlink(pid_fn) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 934 | |
Anas Nashif | d6476ee | 2019-04-11 11:40:09 -0400 | [diff] [blame] | 935 | try: |
| 936 | if pid: |
| 937 | os.kill(pid, signal.SIGTERM) |
| 938 | except ProcessLookupError: |
| 939 | # Oh well, as long as it's dead! User probably sent Ctrl-C |
| 940 | pass |
Andrew Boie | 08ce5a5 | 2016-02-22 13:28:10 -0800 | [diff] [blame] | 941 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 942 | os.unlink(fifo_in) |
| 943 | os.unlink(fifo_out) |
| 944 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 945 | def handle(self): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 946 | self.results = {} |
Anas Nashif | 1377375 | 2018-07-06 18:20:23 -0500 | [diff] [blame] | 947 | self.run = True |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 948 | |
| 949 | # We pass this to QEMU which looks for fifos with .in and .out |
| 950 | # suffixes. |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 951 | self.fifo_fn = os.path.join(self.instance.build_dir, "qemu-fifo") |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 952 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 953 | self.pid_fn = os.path.join(self.instance.build_dir, "qemu.pid") |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 954 | if os.path.exists(self.pid_fn): |
| 955 | os.unlink(self.pid_fn) |
| 956 | |
Anas Nashif | 4a9f3e6 | 2018-07-06 18:58:18 -0500 | [diff] [blame] | 957 | self.log_fn = self.log |
Anas Nashif | 576be98 | 2017-12-23 20:20:27 -0500 | [diff] [blame] | 958 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 959 | harness_import = HarnessImporter(self.instance.testcase.harness.capitalize()) |
Anas Nashif | 576be98 | 2017-12-23 20:20:27 -0500 | [diff] [blame] | 960 | harness = harness_import.instance |
Anas Nashif | d3384fb | 2018-02-22 06:44:16 -0600 | [diff] [blame] | 961 | harness.configure(self.instance) |
| 962 | self.thread = threading.Thread(name=self.name, target=QEMUHandler._thread, |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 963 | args=(self, self.timeout, self.build_dir, |
Jaakko Hannikainen | ca505f8 | 2016-08-22 15:03:46 +0300 | [diff] [blame] | 964 | self.log_fn, self.fifo_fn, |
Anas Nashif | 576be98 | 2017-12-23 20:20:27 -0500 | [diff] [blame] | 965 | self.pid_fn, self.results, harness)) |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 966 | |
| 967 | self.instance.results = harness.tests |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 968 | self.thread.daemon = True |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 969 | verbose("Spawning QEMUHandler Thread for %s" % self.name) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 970 | self.thread.start() |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 971 | subprocess.call(["stty", "sane"]) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 972 | |
Stephanos Ioannidis | e1827c0 | 2019-11-12 14:11:31 +0900 | [diff] [blame] | 973 | verbose("Running %s (%s)" %(self.name, self.type_str)) |
| 974 | command = [get_generator()[0]] |
| 975 | command += ["-C", self.build_dir, "run"] |
| 976 | |
| 977 | with subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=self.build_dir) as proc: |
| 978 | verbose("Spawning QEMUHandler Thread for %s" % self.name) |
| 979 | proc.wait() |
| 980 | self.returncode = proc.returncode |
| 981 | |
| 982 | if self.returncode != 0: |
| 983 | self.set_state("failed", 0) |
| 984 | self.instance.reason = "Exited with {}".format(self.returncode) |
| 985 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 986 | def get_fifo(self): |
| 987 | return self.fifo_fn |
| 988 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 989 | class SizeCalculator: |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 990 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 991 | alloc_sections = [ |
| 992 | "bss", |
| 993 | "noinit", |
| 994 | "app_bss", |
| 995 | "app_noinit", |
| 996 | "ccm_bss", |
| 997 | "ccm_noinit" |
| 998 | ] |
| 999 | |
| 1000 | rw_sections = [ |
| 1001 | "datas", |
| 1002 | "initlevel", |
| 1003 | "exceptions", |
| 1004 | "initshell", |
| 1005 | "_static_thread_area", |
| 1006 | "_k_timer_area", |
| 1007 | "_k_mem_slab_area", |
| 1008 | "_k_mem_pool_area", |
| 1009 | "sw_isr_table", |
| 1010 | "_k_sem_area", |
| 1011 | "_k_mutex_area", |
| 1012 | "app_shmem_regions", |
| 1013 | "_k_fifo_area", |
| 1014 | "_k_lifo_area", |
| 1015 | "_k_stack_area", |
| 1016 | "_k_msgq_area", |
| 1017 | "_k_mbox_area", |
| 1018 | "_k_pipe_area", |
| 1019 | "net_if", |
| 1020 | "net_if_dev", |
| 1021 | "net_stack", |
| 1022 | "net_l2_data", |
| 1023 | "_k_queue_area", |
| 1024 | "_net_buf_pool_area", |
| 1025 | "app_datas", |
| 1026 | "kobject_data", |
| 1027 | "mmu_tables", |
| 1028 | "app_pad", |
| 1029 | "priv_stacks", |
| 1030 | "ccm_data", |
| 1031 | "usb_descriptor", |
| 1032 | "usb_data", "usb_bos_desc", |
| 1033 | 'log_backends_sections', |
| 1034 | 'log_dynamic_sections', |
| 1035 | 'log_const_sections', |
| 1036 | "app_smem", |
| 1037 | 'shell_root_cmds_sections', |
| 1038 | 'log_const_sections', |
| 1039 | "font_entry_sections", |
| 1040 | "priv_stacks_noinit", |
| 1041 | "_TEXT_SECTION_NAME_2", |
| 1042 | "_GCOV_BSS_SECTION_NAME", |
| 1043 | "gcov", |
| 1044 | "nocache" |
| 1045 | ] |
Anas Nashif | db9592a | 2018-10-08 10:19:41 -0400 | [diff] [blame] | 1046 | |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 1047 | # These get copied into RAM only on non-XIP |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1048 | ro_sections = [ |
| 1049 | "text", |
| 1050 | "ctors", |
| 1051 | "init_array", |
| 1052 | "reset", |
| 1053 | "object_access", |
| 1054 | "rodata", |
| 1055 | "devconfig", |
| 1056 | "net_l2", |
| 1057 | "vector", |
| 1058 | "sw_isr_table", |
| 1059 | "_settings_handlers_area", |
| 1060 | "_bt_channels_area", |
| 1061 | "_bt_br_channels_area", |
| 1062 | "_bt_services_area", |
| 1063 | "vectors", |
| 1064 | "net_socket_register", |
| 1065 | "net_ppp_proto" |
| 1066 | ] |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 1067 | |
Andrew Boie | 52fef67 | 2016-11-29 12:21:59 -0800 | [diff] [blame] | 1068 | def __init__(self, filename, extra_sections): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1069 | """Constructor |
| 1070 | |
Andrew Boie | bbd670c | 2015-08-17 13:16:11 -0700 | [diff] [blame] | 1071 | @param filename Path to the output binary |
| 1072 | The <filename> is parsed by objdump to determine section sizes |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1073 | """ |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1074 | # Make sure this is an ELF binary |
Andrew Boie | bbd670c | 2015-08-17 13:16:11 -0700 | [diff] [blame] | 1075 | with open(filename, "rb") as f: |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1076 | magic = f.read(4) |
| 1077 | |
Anas Nashif | b4bdd66 | 2018-08-15 17:12:28 -0500 | [diff] [blame] | 1078 | try: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1079 | if magic != b'\x7fELF': |
Anas Nashif | b4bdd66 | 2018-08-15 17:12:28 -0500 | [diff] [blame] | 1080 | raise SanityRuntimeError("%s is not an ELF binary" % filename) |
| 1081 | except Exception as e: |
| 1082 | print(str(e)) |
| 1083 | sys.exit(2) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1084 | |
| 1085 | # Search for CONFIG_XIP in the ELF's list of symbols using NM and AWK. |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1086 | # GREP can not be used as it returns an error if the symbol is not |
| 1087 | # found. |
| 1088 | is_xip_command = "nm " + filename + \ |
| 1089 | " | awk '/CONFIG_XIP/ { print $3 }'" |
| 1090 | is_xip_output = subprocess.check_output( |
| 1091 | is_xip_command, shell=True, stderr=subprocess.STDOUT).decode( |
| 1092 | "utf-8").strip() |
Anas Nashif | b4bdd66 | 2018-08-15 17:12:28 -0500 | [diff] [blame] | 1093 | try: |
| 1094 | if is_xip_output.endswith("no symbols"): |
| 1095 | raise SanityRuntimeError("%s has no symbol information" % filename) |
| 1096 | except Exception as e: |
| 1097 | print(str(e)) |
| 1098 | sys.exit(2) |
| 1099 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1100 | self.is_xip = (len(is_xip_output) != 0) |
| 1101 | |
Andrew Boie | bbd670c | 2015-08-17 13:16:11 -0700 | [diff] [blame] | 1102 | self.filename = filename |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 1103 | self.sections = [] |
| 1104 | self.rom_size = 0 |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1105 | self.ram_size = 0 |
Andrew Boie | 52fef67 | 2016-11-29 12:21:59 -0800 | [diff] [blame] | 1106 | self.extra_sections = extra_sections |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1107 | |
| 1108 | self._calculate_sizes() |
| 1109 | |
| 1110 | def get_ram_size(self): |
| 1111 | """Get the amount of RAM the application will use up on the device |
| 1112 | |
| 1113 | @return amount of RAM, in bytes |
| 1114 | """ |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 1115 | return self.ram_size |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1116 | |
| 1117 | def get_rom_size(self): |
| 1118 | """Get the size of the data that this application uses on device's flash |
| 1119 | |
| 1120 | @return amount of ROM, in bytes |
| 1121 | """ |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 1122 | return self.rom_size |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1123 | |
| 1124 | def unrecognized_sections(self): |
| 1125 | """Get a list of sections inside the binary that weren't recognized |
| 1126 | |
David B. Kinder | 29963c3 | 2017-06-16 12:32:42 -0700 | [diff] [blame] | 1127 | @return list of unrecognized section names |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1128 | """ |
| 1129 | slist = [] |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 1130 | for v in self.sections: |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1131 | if not v["recognized"]: |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 1132 | slist.append(v["name"]) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1133 | return slist |
| 1134 | |
| 1135 | def _calculate_sizes(self): |
| 1136 | """ Calculate RAM and ROM usage by section """ |
Andrew Boie | bbd670c | 2015-08-17 13:16:11 -0700 | [diff] [blame] | 1137 | objdump_command = "objdump -h " + self.filename |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1138 | objdump_output = subprocess.check_output( |
| 1139 | objdump_command, shell=True).decode("utf-8").splitlines() |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1140 | |
| 1141 | for line in objdump_output: |
| 1142 | words = line.split() |
| 1143 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1144 | if not words: # Skip lines that are too short |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1145 | continue |
| 1146 | |
| 1147 | index = words[0] |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1148 | if not index[0].isdigit(): # Skip lines that do not start |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1149 | continue # with a digit |
| 1150 | |
| 1151 | name = words[1] # Skip lines with section names |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1152 | if name[0] == '.': # starting with '.' |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1153 | continue |
| 1154 | |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 1155 | # TODO this doesn't actually reflect the size in flash or RAM as |
| 1156 | # it doesn't include linker-imposed padding between sections. |
| 1157 | # It is close though. |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1158 | size = int(words[2], 16) |
Andrew Boie | 9882dcd | 2015-10-07 14:25:51 -0700 | [diff] [blame] | 1159 | if size == 0: |
| 1160 | continue |
| 1161 | |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 1162 | load_addr = int(words[4], 16) |
| 1163 | virt_addr = int(words[3], 16) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1164 | |
| 1165 | # Add section to memory use totals (for both non-XIP and XIP scenarios) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1166 | # Unrecognized section names are not included in the calculations. |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1167 | recognized = True |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 1168 | if name in SizeCalculator.alloc_sections: |
| 1169 | self.ram_size += size |
| 1170 | stype = "alloc" |
| 1171 | elif name in SizeCalculator.rw_sections: |
| 1172 | self.ram_size += size |
| 1173 | self.rom_size += size |
| 1174 | stype = "rw" |
| 1175 | elif name in SizeCalculator.ro_sections: |
| 1176 | self.rom_size += size |
| 1177 | if not self.is_xip: |
| 1178 | self.ram_size += size |
| 1179 | stype = "ro" |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1180 | else: |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 1181 | stype = "unknown" |
Andrew Boie | 52fef67 | 2016-11-29 12:21:59 -0800 | [diff] [blame] | 1182 | if name not in self.extra_sections: |
| 1183 | recognized = False |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1184 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1185 | self.sections.append({"name": name, "load_addr": load_addr, |
| 1186 | "size": size, "virt_addr": virt_addr, |
| 1187 | "type": stype, "recognized": recognized}) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1188 | |
| 1189 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1190 | # "list" - List of strings |
| 1191 | # "list:<type>" - List of <type> |
| 1192 | # "set" - Set of unordered, unique strings |
| 1193 | # "set:<type>" - Set of <type> |
| 1194 | # "float" - Floating point |
| 1195 | # "int" - Integer |
| 1196 | # "bool" - Boolean |
| 1197 | # "str" - String |
| 1198 | |
| 1199 | # XXX Be sure to update __doc__ if you change any of this!! |
| 1200 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1201 | platform_valid_keys = { |
Anas Nashif | 924a4e7 | 2018-10-18 12:25:55 -0400 | [diff] [blame] | 1202 | "supported_toolchains": {"type": "list", "default": []}, |
| 1203 | "env": {"type": "list", "default": []} |
| 1204 | } |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1205 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1206 | testcase_valid_keys = {"tags": {"type": "set", "required": False}, |
| 1207 | "type": {"type": "str", "default": "integration"}, |
| 1208 | "extra_args": {"type": "list"}, |
| 1209 | "extra_configs": {"type": "list"}, |
| 1210 | "build_only": {"type": "bool", "default": False}, |
| 1211 | "build_on_all": {"type": "bool", "default": False}, |
| 1212 | "skip": {"type": "bool", "default": False}, |
| 1213 | "slow": {"type": "bool", "default": False}, |
| 1214 | "timeout": {"type": "int", "default": 60}, |
| 1215 | "min_ram": {"type": "int", "default": 8}, |
| 1216 | "depends_on": {"type": "set"}, |
| 1217 | "min_flash": {"type": "int", "default": 32}, |
| 1218 | "arch_whitelist": {"type": "set"}, |
| 1219 | "arch_exclude": {"type": "set"}, |
| 1220 | "extra_sections": {"type": "list", "default": []}, |
| 1221 | "platform_exclude": {"type": "set"}, |
| 1222 | "platform_whitelist": {"type": "set"}, |
| 1223 | "toolchain_exclude": {"type": "set"}, |
| 1224 | "toolchain_whitelist": {"type": "set"}, |
Anas Nashif | ab94016 | 2017-12-08 10:17:57 -0500 | [diff] [blame] | 1225 | "filter": {"type": "str"}, |
Anas Nashif | 576be98 | 2017-12-23 20:20:27 -0500 | [diff] [blame] | 1226 | "harness": {"type": "str"}, |
Praful Swarnakar | cf89e28 | 2018-09-13 02:58:28 +0530 | [diff] [blame] | 1227 | "harness_config": {"type": "map", "default": {}} |
Anas Nashif | ab94016 | 2017-12-08 10:17:57 -0500 | [diff] [blame] | 1228 | } |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1229 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1230 | class SanityConfigParser: |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1231 | """Class to read test case files with semantic checking |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1232 | """ |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1233 | |
Inaky Perez-Gonzalez | 662dde6 | 2017-07-24 10:24:35 -0700 | [diff] [blame] | 1234 | def __init__(self, filename, schema): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1235 | """Instantiate a new SanityConfigParser object |
| 1236 | |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1237 | @param filename Source .yaml file to read |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1238 | """ |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1239 | self.data = {} |
| 1240 | self.schema = schema |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1241 | self.filename = filename |
Anas Nashif | 255625b | 2017-12-05 15:08:26 -0500 | [diff] [blame] | 1242 | self.tests = {} |
| 1243 | self.common = {} |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1244 | |
| 1245 | def load(self): |
| 1246 | self.data = scl.yaml_load_verify(self.filename, self.schema) |
| 1247 | |
Anas Nashif | 255625b | 2017-12-05 15:08:26 -0500 | [diff] [blame] | 1248 | if 'tests' in self.data: |
| 1249 | self.tests = self.data['tests'] |
| 1250 | if 'common' in self.data: |
| 1251 | self.common = self.data['common'] |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1252 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1253 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1254 | def _cast_value(self, value, typestr): |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1255 | if isinstance(value, str): |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1256 | v = value.strip() |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1257 | if typestr == "str": |
| 1258 | return v |
| 1259 | |
| 1260 | elif typestr == "float": |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1261 | return float(value) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1262 | |
| 1263 | elif typestr == "int": |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1264 | return int(value) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1265 | |
| 1266 | elif typestr == "bool": |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1267 | return value |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1268 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1269 | elif typestr.startswith("list") and isinstance(value, list): |
Anas Nashif | fa695d2 | 2017-10-04 16:14:27 -0400 | [diff] [blame] | 1270 | return value |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1271 | elif typestr.startswith("list") and isinstance(value, str): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1272 | vs = v.split() |
| 1273 | if len(typestr) > 4 and typestr[4] == ":": |
| 1274 | return [self._cast_value(vsi, typestr[5:]) for vsi in vs] |
| 1275 | else: |
| 1276 | return vs |
| 1277 | |
| 1278 | elif typestr.startswith("set"): |
| 1279 | vs = v.split() |
| 1280 | if len(typestr) > 3 and typestr[3] == ":": |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1281 | return {self._cast_value(vsi, typestr[4:]) for vsi in vs} |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1282 | else: |
| 1283 | return set(vs) |
| 1284 | |
Anas Nashif | 576be98 | 2017-12-23 20:20:27 -0500 | [diff] [blame] | 1285 | elif typestr.startswith("map"): |
| 1286 | return value |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1287 | else: |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1288 | raise ConfigurationError( |
| 1289 | self.filename, "unknown type '%s'" % value) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1290 | |
Anas Nashif | b4754ed | 2017-12-05 17:27:58 -0500 | [diff] [blame] | 1291 | def get_test(self, name, valid_keys): |
| 1292 | """Get a dictionary representing the keys/values within a test |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1293 | |
Anas Nashif | b4754ed | 2017-12-05 17:27:58 -0500 | [diff] [blame] | 1294 | @param name The test in the .yaml file to retrieve data from |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1295 | @param valid_keys A dictionary representing the intended semantics |
Anas Nashif | b4754ed | 2017-12-05 17:27:58 -0500 | [diff] [blame] | 1296 | for this test. Each key in this dictionary is a key that could |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1297 | be specified, if a key is given in the .yaml file which isn't in |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1298 | here, it will generate an error. Each value in this dictionary |
| 1299 | is another dictionary containing metadata: |
| 1300 | |
| 1301 | "default" - Default value if not given |
| 1302 | "type" - Data type to convert the text value to. Simple types |
| 1303 | supported are "str", "float", "int", "bool" which will get |
| 1304 | converted to respective Python data types. "set" and "list" |
| 1305 | may also be specified which will split the value by |
| 1306 | whitespace (but keep the elements as strings). finally, |
| 1307 | "list:<type>" and "set:<type>" may be given which will |
| 1308 | perform a type conversion after splitting the value up. |
| 1309 | "required" - If true, raise an error if not defined. If false |
| 1310 | and "default" isn't specified, a type conversion will be |
| 1311 | done on an empty string |
Anas Nashif | b4754ed | 2017-12-05 17:27:58 -0500 | [diff] [blame] | 1312 | @return A dictionary containing the test key-value pairs with |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1313 | type conversion and default values filled in per valid_keys |
| 1314 | """ |
| 1315 | |
| 1316 | d = {} |
Anas Nashif | 255625b | 2017-12-05 15:08:26 -0500 | [diff] [blame] | 1317 | for k, v in self.common.items(): |
Anas Nashif | fa695d2 | 2017-10-04 16:14:27 -0400 | [diff] [blame] | 1318 | d[k] = v |
Anas Nashif | 75547e2 | 2018-02-24 08:32:14 -0600 | [diff] [blame] | 1319 | |
Anas Nashif | b4754ed | 2017-12-05 17:27:58 -0500 | [diff] [blame] | 1320 | for k, v in self.tests[name].items(): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1321 | if k not in valid_keys: |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1322 | raise ConfigurationError( |
| 1323 | self.filename, |
| 1324 | "Unknown config key '%s' in definition for '%s'" % |
Anas Nashif | b4754ed | 2017-12-05 17:27:58 -0500 | [diff] [blame] | 1325 | (k, name)) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1326 | |
Anas Nashif | fa695d2 | 2017-10-04 16:14:27 -0400 | [diff] [blame] | 1327 | if k in d: |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1328 | if isinstance(d[k], str): |
Paul Sokolovsky | 8c6d2e2 | 2019-08-29 13:20:57 +0300 | [diff] [blame] | 1329 | # By default, we just concatenate string values of keys |
| 1330 | # which appear both in "common" and per-test sections, |
| 1331 | # but some keys are handled in adhoc way based on their |
| 1332 | # semantics. |
| 1333 | if k == "filter": |
| 1334 | d[k] = "(%s) and (%s)" % (d[k], v) |
| 1335 | else: |
| 1336 | d[k] += " " + v |
Anas Nashif | fa695d2 | 2017-10-04 16:14:27 -0400 | [diff] [blame] | 1337 | else: |
| 1338 | d[k] = v |
Anas Nashif | 75547e2 | 2018-02-24 08:32:14 -0600 | [diff] [blame] | 1339 | |
Andrew Boie | 08ce5a5 | 2016-02-22 13:28:10 -0800 | [diff] [blame] | 1340 | for k, kinfo in valid_keys.items(): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1341 | if k not in d: |
| 1342 | if "required" in kinfo: |
| 1343 | required = kinfo["required"] |
| 1344 | else: |
| 1345 | required = False |
| 1346 | |
| 1347 | if required: |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1348 | raise ConfigurationError( |
| 1349 | self.filename, |
Anas Nashif | b4754ed | 2017-12-05 17:27:58 -0500 | [diff] [blame] | 1350 | "missing required value for '%s' in test '%s'" % |
| 1351 | (k, name)) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1352 | else: |
| 1353 | if "default" in kinfo: |
| 1354 | default = kinfo["default"] |
| 1355 | else: |
| 1356 | default = self._cast_value("", kinfo["type"]) |
| 1357 | d[k] = default |
| 1358 | else: |
| 1359 | try: |
| 1360 | d[k] = self._cast_value(d[k], kinfo["type"]) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1361 | except ValueError: |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1362 | raise ConfigurationError( |
Anas Nashif | b4754ed | 2017-12-05 17:27:58 -0500 | [diff] [blame] | 1363 | self.filename, "bad %s value '%s' for key '%s' in name '%s'" % |
| 1364 | (kinfo["type"], d[k], k, name)) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1365 | |
| 1366 | return d |
| 1367 | |
| 1368 | |
| 1369 | class Platform: |
| 1370 | """Class representing metadata for a particular platform |
| 1371 | |
Anas Nashif | c740608 | 2015-12-13 15:00:31 -0500 | [diff] [blame] | 1372 | Maps directly to BOARD when building""" |
Inaky Perez-Gonzalez | 662dde6 | 2017-07-24 10:24:35 -0700 | [diff] [blame] | 1373 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1374 | platform_schema = scl.yaml_load(os.path.join(ZEPHYR_BASE, |
| 1375 | "scripts","sanity_chk","platform-schema.yaml")) |
Inaky Perez-Gonzalez | 662dde6 | 2017-07-24 10:24:35 -0700 | [diff] [blame] | 1376 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1377 | def __init__(self): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1378 | """Constructor. |
| 1379 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1380 | """ |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1381 | |
| 1382 | self.name = "" |
| 1383 | self.sanitycheck = True |
| 1384 | # if no RAM size is specified by the board, take a default of 128K |
| 1385 | self.ram = 128 |
| 1386 | |
| 1387 | self.ignore_tags = [] |
| 1388 | self.default = False |
| 1389 | # if no flash size is specified by the board, take a default of 512K |
| 1390 | self.flash = 512 |
| 1391 | self.supported = set() |
| 1392 | |
| 1393 | self.arch = "" |
| 1394 | self.type = "na" |
| 1395 | self.simulation = "na" |
| 1396 | self.supported_toolchains = [] |
| 1397 | self.env = [] |
| 1398 | self.env_satisfied = True |
| 1399 | self.filter_data = dict() |
| 1400 | |
| 1401 | def load(self, platform_file): |
| 1402 | scp = SanityConfigParser(platform_file, self.platform_schema) |
| 1403 | scp.load() |
Anas Nashif | 255625b | 2017-12-05 15:08:26 -0500 | [diff] [blame] | 1404 | data = scp.data |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1405 | |
Anas Nashif | 255625b | 2017-12-05 15:08:26 -0500 | [diff] [blame] | 1406 | self.name = data['identifier'] |
Anas Nashif | f3d48e1 | 2018-07-24 08:14:42 -0500 | [diff] [blame] | 1407 | self.sanitycheck = data.get("sanitycheck", True) |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1408 | # if no RAM size is specified by the board, take a default of 128K |
Anas Nashif | 255625b | 2017-12-05 15:08:26 -0500 | [diff] [blame] | 1409 | self.ram = data.get("ram", 128) |
| 1410 | testing = data.get("testing", {}) |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1411 | self.ignore_tags = testing.get("ignore_tags", []) |
| 1412 | self.default = testing.get("default", False) |
| 1413 | # if no flash size is specified by the board, take a default of 512K |
Anas Nashif | 255625b | 2017-12-05 15:08:26 -0500 | [diff] [blame] | 1414 | self.flash = data.get("flash", 512) |
Anas Nashif | 9527693 | 2017-07-31 08:47:41 -0400 | [diff] [blame] | 1415 | self.supported = set() |
Anas Nashif | 255625b | 2017-12-05 15:08:26 -0500 | [diff] [blame] | 1416 | for supp_feature in data.get("supported", []): |
Anas Nashif | 9527693 | 2017-07-31 08:47:41 -0400 | [diff] [blame] | 1417 | for item in supp_feature.split(":"): |
| 1418 | self.supported.add(item) |
| 1419 | |
Anas Nashif | 255625b | 2017-12-05 15:08:26 -0500 | [diff] [blame] | 1420 | self.arch = data['arch'] |
Anas Nashif | cc16422 | 2017-12-26 11:02:46 -0500 | [diff] [blame] | 1421 | self.type = data.get('type', "na") |
Anas Nashif | 8acdbd7 | 2018-01-04 14:15:22 -0500 | [diff] [blame] | 1422 | self.simulation = data.get('simulation', "na") |
Anas Nashif | 255625b | 2017-12-05 15:08:26 -0500 | [diff] [blame] | 1423 | self.supported_toolchains = data.get("toolchain", []) |
Anas Nashif | 924a4e7 | 2018-10-18 12:25:55 -0400 | [diff] [blame] | 1424 | self.env = data.get("env", []) |
| 1425 | self.env_satisfied = True |
| 1426 | for env in self.env: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1427 | if not os.environ.get(env, None): |
Anas Nashif | 924a4e7 | 2018-10-18 12:25:55 -0400 | [diff] [blame] | 1428 | self.env_satisfied = False |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1429 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1430 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1431 | def __repr__(self): |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1432 | return "<%s on %s>" % (self.name, self.arch) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1433 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1434 | class TestCase(object): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1435 | """Class representing a test application |
| 1436 | """ |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1437 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1438 | |
| 1439 | def __init__(self): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1440 | """TestCase constructor. |
| 1441 | |
Anas Nashif | 877d3ca | 2017-12-05 17:39:29 -0500 | [diff] [blame] | 1442 | This gets called by TestSuite as it finds and reads test yaml files. |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1443 | Multiple TestCase instances may be generated from a single testcase.yaml, |
Anas Nashif | 877d3ca | 2017-12-05 17:39:29 -0500 | [diff] [blame] | 1444 | each one corresponds to an entry within that file. |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1445 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1446 | We need to have a unique name for every single test case. Since |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1447 | a testcase.yaml can define multiple tests, the canonical name for |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1448 | the test case is <workdir>/<name>. |
| 1449 | |
Marc Herbert | 1c8632c | 2019-04-15 17:58:45 -0700 | [diff] [blame] | 1450 | @param testcase_root os.path.abspath() of one of the --testcase-root |
| 1451 | @param workdir Sub-directory of testcase_root where the |
| 1452 | .yaml test configuration file was found |
Anas Nashif | 877d3ca | 2017-12-05 17:39:29 -0500 | [diff] [blame] | 1453 | @param name Name of this test case, corresponding to the entry name |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1454 | in the test case configuration file. For many test cases that just |
| 1455 | define one test, can be anything and is usually "test". This is |
| 1456 | really only used to distinguish between different cases when |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1457 | the testcase.yaml defines multiple tests |
Anas Nashif | 877d3ca | 2017-12-05 17:39:29 -0500 | [diff] [blame] | 1458 | @param tc_dict Dictionary with test values for this test case |
Anas Nashif | a792a3d | 2017-04-04 18:47:49 -0400 | [diff] [blame] | 1459 | from the testcase.yaml file |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1460 | """ |
Anas Nashif | f18ad9d | 2018-11-20 09:03:17 -0500 | [diff] [blame] | 1461 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1462 | self.id = "" |
| 1463 | self.source_dir = "" |
| 1464 | self.yamlfile = "" |
Anas Nashif | aae71d7 | 2018-04-21 22:26:48 -0500 | [diff] [blame] | 1465 | self.cases = [] |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1466 | self.name = "" |
Anas Nashif | bd166f4 | 2017-09-02 12:32:08 -0400 | [diff] [blame] | 1467 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1468 | self.type = None |
| 1469 | self.tags = None |
| 1470 | self.extra_args = None |
| 1471 | self.extra_configs = None |
| 1472 | self.arch_whitelist = None |
| 1473 | self.arch_exclude = None |
| 1474 | self.skip = None |
| 1475 | self.platform_exclude = None |
| 1476 | self.platform_whitelist = None |
| 1477 | self.toolchain_exclude = None |
| 1478 | self.toolchain_whitelist = None |
| 1479 | self.tc_filter = None |
| 1480 | self.timeout = 60 |
| 1481 | self.harness = "" |
| 1482 | self.harness_config = {} |
| 1483 | self.build_only = True |
| 1484 | self.build_on_all = False |
| 1485 | self.slow = False |
| 1486 | self.min_ram = None |
| 1487 | self.depends_on = None |
| 1488 | self.min_flash = None |
| 1489 | self.extra_sections = None |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1490 | |
Anas Nashif | f18ad9d | 2018-11-20 09:03:17 -0500 | [diff] [blame] | 1491 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1492 | @staticmethod |
| 1493 | def get_unique(testcase_root, workdir, name): |
Anas Nashif | f18ad9d | 2018-11-20 09:03:17 -0500 | [diff] [blame] | 1494 | |
Marc Herbert | 1c8632c | 2019-04-15 17:58:45 -0700 | [diff] [blame] | 1495 | canonical_testcase_root = os.path.realpath(testcase_root) |
Tyler Hall | bb35925 | 2019-06-19 02:11:35 -0400 | [diff] [blame] | 1496 | if Path(canonical_zephyr_base) in Path(canonical_testcase_root).parents: |
Marc Herbert | 1c8632c | 2019-04-15 17:58:45 -0700 | [diff] [blame] | 1497 | # This is in ZEPHYR_BASE, so include path in name for uniqueness |
Anas Nashif | f18ad9d | 2018-11-20 09:03:17 -0500 | [diff] [blame] | 1498 | # FIXME: We should not depend on path of test for unique names. |
Marc Herbert | 1c8632c | 2019-04-15 17:58:45 -0700 | [diff] [blame] | 1499 | relative_tc_root = os.path.relpath(canonical_testcase_root, |
| 1500 | start=canonical_zephyr_base) |
Anas Nashif | f18ad9d | 2018-11-20 09:03:17 -0500 | [diff] [blame] | 1501 | else: |
Marc Herbert | 1c8632c | 2019-04-15 17:58:45 -0700 | [diff] [blame] | 1502 | relative_tc_root = "" |
Anas Nashif | f18ad9d | 2018-11-20 09:03:17 -0500 | [diff] [blame] | 1503 | |
Marc Herbert | 1c8632c | 2019-04-15 17:58:45 -0700 | [diff] [blame] | 1504 | # workdir can be "." |
| 1505 | unique = os.path.normpath(os.path.join(relative_tc_root, workdir, name)) |
Anas Nashif | f18ad9d | 2018-11-20 09:03:17 -0500 | [diff] [blame] | 1506 | return unique |
| 1507 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1508 | @staticmethod |
| 1509 | def scan_file(inf_name): |
Anas Nashif | aae71d7 | 2018-04-21 22:26:48 -0500 | [diff] [blame] | 1510 | suite_regex = re.compile( |
Inaky Perez-Gonzalez | 75e2d90 | 2018-04-26 00:17:01 -0700 | [diff] [blame] | 1511 | # do not match until end-of-line, otherwise we won't allow |
| 1512 | # stc_regex below to catch the ones that are declared in the same |
| 1513 | # line--as we only search starting the end of this match |
| 1514 | br"^\s*ztest_test_suite\(\s*(?P<suite_name>[a-zA-Z0-9_]+)\s*,", |
Anas Nashif | aae71d7 | 2018-04-21 22:26:48 -0500 | [diff] [blame] | 1515 | re.MULTILINE) |
| 1516 | stc_regex = re.compile( |
Inaky Perez-Gonzalez | 75e2d90 | 2018-04-26 00:17:01 -0700 | [diff] [blame] | 1517 | br"^\s*" # empy space at the beginning is ok |
| 1518 | # catch the case where it is declared in the same sentence, e.g: |
| 1519 | # |
| 1520 | # ztest_test_suite(mutex_complex, ztest_user_unit_test(TESTNAME)); |
| 1521 | br"(?:ztest_test_suite\([a-zA-Z0-9_]+,\s*)?" |
| 1522 | # Catch ztest[_user]_unit_test-[_setup_teardown](TESTNAME) |
| 1523 | br"ztest_(?:user_)?unit_test(?:_setup_teardown)?" |
| 1524 | # Consume the argument that becomes the extra testcse |
| 1525 | br"\(\s*" |
| 1526 | br"(?P<stc_name>[a-zA-Z0-9_]+)" |
| 1527 | # _setup_teardown() variant has two extra arguments that we ignore |
| 1528 | br"(?:\s*,\s*[a-zA-Z0-9_]+\s*,\s*[a-zA-Z0-9_]+)?" |
| 1529 | br"\s*\)", |
| 1530 | # We don't check how it finishes; we don't care |
Anas Nashif | aae71d7 | 2018-04-21 22:26:48 -0500 | [diff] [blame] | 1531 | re.MULTILINE) |
| 1532 | suite_run_regex = re.compile( |
| 1533 | br"^\s*ztest_run_test_suite\((?P<suite_name>[a-zA-Z0-9_]+)\)", |
| 1534 | re.MULTILINE) |
| 1535 | achtung_regex = re.compile( |
| 1536 | br"(#ifdef|#endif)", |
| 1537 | re.MULTILINE) |
| 1538 | warnings = None |
| 1539 | |
| 1540 | with open(inf_name) as inf: |
Anas Nashif | 19d67e4 | 2019-11-21 11:33:12 -0500 | [diff] [blame] | 1541 | if os.name == 'nt': |
| 1542 | mmap_args = {'fileno':inf.fileno(), 'length':0, 'access':mmap.ACCESS_READ} |
| 1543 | else: |
| 1544 | mmap_args = {'fileno':inf.fileno(), 'length':0, 'flags':mmap.MAP_PRIVATE, 'prot':mmap.PROT_READ, 'offset':0} |
| 1545 | |
| 1546 | with contextlib.closing(mmap.mmap(**mmap_args)) as main_c: |
Ulf Magnusson | 5c2e814 | 2019-10-29 11:50:26 +0100 | [diff] [blame] | 1547 | # contextlib makes pylint think main_c isn't subscriptable |
| 1548 | # pylint: disable=unsubscriptable-object |
| 1549 | |
Anas Nashif | aae71d7 | 2018-04-21 22:26:48 -0500 | [diff] [blame] | 1550 | suite_regex_match = suite_regex.search(main_c) |
| 1551 | if not suite_regex_match: |
| 1552 | # can't find ztest_test_suite, maybe a client, because |
| 1553 | # it includes ztest.h |
| 1554 | return None, None |
| 1555 | |
| 1556 | suite_run_match = suite_run_regex.search(main_c) |
| 1557 | if not suite_run_match: |
| 1558 | raise ValueError("can't find ztest_run_test_suite") |
| 1559 | |
| 1560 | achtung_matches = re.findall( |
| 1561 | achtung_regex, |
| 1562 | main_c[suite_regex_match.end():suite_run_match.start()]) |
| 1563 | if achtung_matches: |
| 1564 | warnings = "found invalid %s in ztest_test_suite()" \ |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1565 | % ", ".join({match.decode() for match in achtung_matches}) |
Inaky Perez-Gonzalez | 75e2d90 | 2018-04-26 00:17:01 -0700 | [diff] [blame] | 1566 | _matches = re.findall( |
Anas Nashif | aae71d7 | 2018-04-21 22:26:48 -0500 | [diff] [blame] | 1567 | stc_regex, |
| 1568 | main_c[suite_regex_match.end():suite_run_match.start()]) |
Inaky Perez-Gonzalez | 75e2d90 | 2018-04-26 00:17:01 -0700 | [diff] [blame] | 1569 | matches = [ match.decode().replace("test_", "") for match in _matches ] |
Anas Nashif | aae71d7 | 2018-04-21 22:26:48 -0500 | [diff] [blame] | 1570 | return matches, warnings |
| 1571 | |
| 1572 | def scan_path(self, path): |
| 1573 | subcases = [] |
| 1574 | for filename in glob.glob(os.path.join(path, "src", "*.c")): |
| 1575 | try: |
| 1576 | _subcases, warnings = self.scan_file(filename) |
| 1577 | if warnings: |
Inaky Perez-Gonzalez | 75e2d90 | 2018-04-26 00:17:01 -0700 | [diff] [blame] | 1578 | error("%s: %s" % (filename, warnings)) |
Anas Nashif | aae71d7 | 2018-04-21 22:26:48 -0500 | [diff] [blame] | 1579 | if _subcases: |
| 1580 | subcases += _subcases |
| 1581 | except ValueError as e: |
Flavio Ceolin | bf878ce | 2019-04-19 22:24:09 -0700 | [diff] [blame] | 1582 | error("%s: can't find: %s" % (filename, e)) |
Anas Nashif | aae71d7 | 2018-04-21 22:26:48 -0500 | [diff] [blame] | 1583 | return subcases |
| 1584 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1585 | def parse_subcases(self, test_path): |
| 1586 | results = self.scan_path(os.path.dirname(test_path)) |
Anas Nashif | aae71d7 | 2018-04-21 22:26:48 -0500 | [diff] [blame] | 1587 | for sub in results: |
Inaky Perez-Gonzalez | 75e2d90 | 2018-04-26 00:17:01 -0700 | [diff] [blame] | 1588 | name = "{}.{}".format(self.id, sub) |
Anas Nashif | aae71d7 | 2018-04-21 22:26:48 -0500 | [diff] [blame] | 1589 | self.cases.append(name) |
| 1590 | |
Anas Nashif | f16e92c | 2019-03-31 16:58:12 -0400 | [diff] [blame] | 1591 | if not results: |
| 1592 | self.cases.append(self.id) |
| 1593 | |
Anas Nashif | aae71d7 | 2018-04-21 22:26:48 -0500 | [diff] [blame] | 1594 | |
Anas Nashif | 75547e2 | 2018-02-24 08:32:14 -0600 | [diff] [blame] | 1595 | def __str__(self): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1596 | return self.name |
| 1597 | |
| 1598 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1599 | class TestInstance: |
| 1600 | """Class representing the execution of a particular TestCase on a platform |
| 1601 | |
| 1602 | @param test The TestCase object we want to build/execute |
| 1603 | @param platform Platform object that we want to build and run against |
| 1604 | @param base_outdir Base directory for all test results. The actual |
| 1605 | out directory used is <outdir>/<platform>/<test case name> |
| 1606 | """ |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 1607 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1608 | def __init__(self, testcase, platform, base_outdir): |
Anas Nashif | 7dd19ea | 2018-11-14 08:46:49 -0500 | [diff] [blame] | 1609 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1610 | self.testcase = testcase |
| 1611 | self.platform = platform |
| 1612 | |
| 1613 | self.status = None |
| 1614 | self.reason = "N/A" |
| 1615 | self.metrics = dict() |
| 1616 | self.handler = None |
| 1617 | |
| 1618 | |
| 1619 | self.name = os.path.join(platform.name, testcase.name) |
| 1620 | self.build_dir = os.path.join(base_outdir, platform.name, testcase.name) |
| 1621 | |
| 1622 | self.build_only = self.check_build_or_run() |
| 1623 | self.run = not self.build_only |
| 1624 | |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 1625 | self.results = {} |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1626 | |
Marc Herbert | 0f7255c | 2019-04-05 14:14:21 -0700 | [diff] [blame] | 1627 | def __lt__(self, other): |
| 1628 | return self.name < other.name |
| 1629 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1630 | def check_build_or_run(self): |
Anas Nashif | 19d67e4 | 2019-11-21 11:33:12 -0500 | [diff] [blame] | 1631 | # right now we only support building on windows. running is still work |
| 1632 | # in progress. |
| 1633 | |
| 1634 | if os.name == 'nt': |
| 1635 | return True |
Anas Nashif | 7dd19ea | 2018-11-14 08:46:49 -0500 | [diff] [blame] | 1636 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1637 | build_only = True |
| 1638 | |
| 1639 | # we asked for build-only on the command line |
| 1640 | if options.build_only: |
| 1641 | return True |
| 1642 | |
| 1643 | # The testcase is designed to be build only. |
| 1644 | if self.testcase.build_only: |
| 1645 | return True |
| 1646 | |
| 1647 | # Do not run slow tests: |
| 1648 | skip_slow = self.testcase.slow and not options.enable_slow |
| 1649 | if skip_slow: |
| 1650 | return True |
| 1651 | |
| 1652 | runnable =bool(self.testcase.type == "unit" or \ |
| 1653 | self.platform.type == "native" or \ |
| 1654 | self.platform.simulation in ["nsim", "renode", "qemu"] or \ |
| 1655 | options.device_testing) |
| 1656 | |
| 1657 | if self.platform.simulation == "nsim": |
| 1658 | if not find_executable("nsimdrv"): |
| 1659 | runnable = False |
| 1660 | |
| 1661 | if self.platform.simulation == "renode": |
| 1662 | if not find_executable("renode"): |
| 1663 | runnable = False |
| 1664 | |
| 1665 | # console harness allows us to run the test and capture data. |
| 1666 | if self.testcase.harness == 'console': |
| 1667 | |
| 1668 | # if we have a fixture that is also being supplied on the |
| 1669 | # command-line, then we need to run the test, not just build it. |
| 1670 | if "fixture" in self.testcase.harness_config: |
| 1671 | fixture = self.testcase.harness_config['fixture'] |
| 1672 | if fixture in options.fixture: |
| 1673 | build_only = False |
| 1674 | else: |
| 1675 | build_only = True |
| 1676 | else: |
| 1677 | build_only = False |
| 1678 | elif self.testcase.harness: |
| 1679 | build_only = True |
| 1680 | else: |
| 1681 | build_only = False |
| 1682 | |
| 1683 | return not (not build_only and runnable) |
Anas Nashif | 7dd19ea | 2018-11-14 08:46:49 -0500 | [diff] [blame] | 1684 | |
Anas Nashif | dbd7649 | 2018-11-23 20:24:19 -0500 | [diff] [blame] | 1685 | def create_overlay(self, platform): |
Marc Herbert | c7633de | 2019-07-06 15:52:31 -0700 | [diff] [blame] | 1686 | # Create this in a "sanitycheck/" subdirectory otherwise this |
| 1687 | # will pass this overlay to kconfig.py *twice* and kconfig.cmake |
| 1688 | # will silently give that second time precedence over any |
| 1689 | # --extra-args=CONFIG_* |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1690 | subdir = os.path.join(self.build_dir, "sanitycheck") |
Marc Herbert | c7633de | 2019-07-06 15:52:31 -0700 | [diff] [blame] | 1691 | os.makedirs(subdir, exist_ok=True) |
| 1692 | file = os.path.join(subdir, "testcase_extra.conf") |
Anas Nashif | f4d8ecc | 2019-03-02 15:43:23 -0500 | [diff] [blame] | 1693 | with open(file, "w") as f: |
| 1694 | content = "" |
Anas Nashif | 3cbffef | 2018-11-07 23:50:54 -0500 | [diff] [blame] | 1695 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1696 | if self.testcase.extra_configs: |
| 1697 | content = "\n".join(self.testcase.extra_configs) |
Anas Nashif | 3cbffef | 2018-11-07 23:50:54 -0500 | [diff] [blame] | 1698 | |
Anas Nashif | f4d8ecc | 2019-03-02 15:43:23 -0500 | [diff] [blame] | 1699 | if options.enable_coverage: |
Jan Van Winkel | 21212f3 | 2019-09-12 00:03:35 +0200 | [diff] [blame] | 1700 | if platform.name in options.coverage_platform: |
Anas Nashif | f4d8ecc | 2019-03-02 15:43:23 -0500 | [diff] [blame] | 1701 | content = content + "\nCONFIG_COVERAGE=y" |
| 1702 | |
Jan Van Winkel | 21212f3 | 2019-09-12 00:03:35 +0200 | [diff] [blame] | 1703 | if options.enable_asan: |
| 1704 | if platform.type == "native": |
| 1705 | content = content + "\nCONFIG_ASAN=y" |
| 1706 | |
Anas Nashif | f4d8ecc | 2019-03-02 15:43:23 -0500 | [diff] [blame] | 1707 | f.write(content) |
Anas Nashif | fa695d2 | 2017-10-04 16:14:27 -0400 | [diff] [blame] | 1708 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1709 | def calculate_sizes(self): |
| 1710 | """Get the RAM/ROM sizes of a test case. |
| 1711 | |
| 1712 | This can only be run after the instance has been executed by |
| 1713 | MakeGenerator, otherwise there won't be any binaries to measure. |
| 1714 | |
| 1715 | @return A SizeCalculator object |
| 1716 | """ |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1717 | fns = glob.glob(os.path.join(self.build_dir, "zephyr", "*.elf")) |
| 1718 | fns.extend(glob.glob(os.path.join(self.build_dir, "zephyr", "*.exe"))) |
Anas Nashif | f8dcad4 | 2016-10-27 18:10:08 -0400 | [diff] [blame] | 1719 | fns = [x for x in fns if not x.endswith('_prebuilt.elf')] |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1720 | if len(fns) != 1: |
Andrew Boie | 5d4eb78 | 2015-10-02 10:04:56 -0700 | [diff] [blame] | 1721 | raise BuildError("Missing/multiple output ELF binary") |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1722 | |
| 1723 | return SizeCalculator(fns[0], self.testcase.extra_sections) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1724 | |
| 1725 | def __repr__(self): |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1726 | return "<TestCase %s on %s>" % (self.testcase.name, self.platform.name) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 1727 | |
| 1728 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1729 | class CMake(): |
Andrew Boie | 4ef16c5 | 2015-08-28 12:36:03 -0700 | [diff] [blame] | 1730 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1731 | config_re = re.compile('(CONFIG_[A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$') |
| 1732 | dt_re = re.compile('([A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$') |
| 1733 | |
| 1734 | def __init__(self, testcase, platform, source_dir, build_dir): |
| 1735 | |
| 1736 | self.cwd = None |
| 1737 | self.capture_output = True |
| 1738 | |
| 1739 | self.defconfig = {} |
| 1740 | self.cmake_cache = {} |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1741 | |
| 1742 | self.instance = None |
| 1743 | self.testcase = testcase |
| 1744 | self.platform = platform |
| 1745 | self.source_dir = source_dir |
| 1746 | self.build_dir = build_dir |
| 1747 | self.log = "build.log" |
| 1748 | |
| 1749 | def parse_generated(self): |
| 1750 | self.defconfig = {} |
| 1751 | return {} |
| 1752 | |
| 1753 | def run_build(self, args=[]): |
| 1754 | |
| 1755 | verbose("Building %s for %s" % (self.source_dir, self.platform.name)) |
| 1756 | |
| 1757 | cmake_args = [] |
| 1758 | cmake_args.extend(args) |
| 1759 | cmake = shutil.which('cmake') |
| 1760 | cmd = [cmake] + cmake_args |
| 1761 | kwargs = dict() |
| 1762 | |
| 1763 | if self.capture_output: |
| 1764 | kwargs['stdout'] = subprocess.PIPE |
| 1765 | # CMake sends the output of message() to stderr unless it's STATUS |
| 1766 | kwargs['stderr'] = subprocess.STDOUT |
| 1767 | |
| 1768 | if self.cwd: |
| 1769 | kwargs['cwd'] = self.cwd |
| 1770 | |
| 1771 | p = subprocess.Popen(cmd, **kwargs) |
| 1772 | out, _ = p.communicate() |
| 1773 | |
| 1774 | results = {} |
| 1775 | if p.returncode == 0: |
| 1776 | msg = "Finished building %s for %s" %(self.source_dir, self.platform.name) |
| 1777 | |
| 1778 | self.instance.status = "passed" |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1779 | results = {'msg': msg, "returncode": p.returncode, "instance": self.instance} |
| 1780 | |
| 1781 | if out: |
| 1782 | log_msg = out.decode(sys.getdefaultencoding()) |
| 1783 | with open(os.path.join(self.build_dir, self.log), "a") as log: |
| 1784 | log.write(log_msg) |
| 1785 | |
| 1786 | else: |
| 1787 | return None |
| 1788 | else: |
| 1789 | # A real error occurred, raise an exception |
| 1790 | if out: |
| 1791 | log_msg = out.decode(sys.getdefaultencoding()) |
| 1792 | with open(os.path.join(self.build_dir, self.log), "a") as log: |
| 1793 | log.write(log_msg) |
| 1794 | |
| 1795 | overflow_flash = "region `FLASH' overflowed by" |
| 1796 | overflow_ram = "region `RAM' overflowed by" |
| 1797 | |
| 1798 | if log_msg: |
| 1799 | if log_msg.find(overflow_flash) > 0 or log_msg.find(overflow_ram) > 0: |
| 1800 | verbose("RAM/ROM Overflow") |
| 1801 | self.instance.status = "skipped" |
| 1802 | self.instance.reason = "overflow" |
| 1803 | else: |
| 1804 | self.instance.status = "failed" |
| 1805 | self.instance.reason = "Build failure" |
| 1806 | |
| 1807 | results = { |
| 1808 | "returncode": p.returncode, |
| 1809 | "instance": self.instance, |
| 1810 | } |
| 1811 | |
| 1812 | return results |
| 1813 | |
| 1814 | def run_cmake(self, args=[]): |
| 1815 | |
| 1816 | verbose("Running cmake on %s for %s" %(self.source_dir, self.platform.name)) |
| 1817 | |
Anas Nashif | a5984ab | 2019-10-22 07:36:24 -0700 | [diff] [blame] | 1818 | ldflags="-Wl,--fatal-warnings" |
| 1819 | |
| 1820 | #fixme: add additional cflags based on options |
| 1821 | cmake_args = [ |
| 1822 | '-B{}'.format(self.build_dir), |
| 1823 | '-S{}'.format(self.source_dir), |
| 1824 | '-DEXTRA_CFLAGS="-Werror ', |
| 1825 | '-DEXTRA_AFLAGS=-Wa,--fatal-warnings', |
| 1826 | '-DEXTRA_LDFLAGS="{}'.format(ldflags), |
| 1827 | '-G{}'.format(get_generator()[1]) |
| 1828 | ] |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1829 | |
| 1830 | args = ["-D{}".format(a.replace('"', '')) for a in args] |
| 1831 | cmake_args.extend(args) |
| 1832 | |
| 1833 | cmake_opts = ['-DBOARD={}'.format(self.platform.name)] |
| 1834 | cmake_args.extend(cmake_opts) |
| 1835 | |
| 1836 | cmake = shutil.which('cmake') |
| 1837 | cmd = [cmake] + cmake_args |
| 1838 | kwargs = dict() |
| 1839 | |
| 1840 | if self.capture_output: |
| 1841 | kwargs['stdout'] = subprocess.PIPE |
| 1842 | # CMake sends the output of message() to stderr unless it's STATUS |
| 1843 | kwargs['stderr'] = subprocess.STDOUT |
| 1844 | |
| 1845 | if self.cwd: |
| 1846 | kwargs['cwd'] = self.cwd |
| 1847 | |
| 1848 | p = subprocess.Popen(cmd, **kwargs) |
| 1849 | out, _ = p.communicate() |
| 1850 | |
| 1851 | if p.returncode == 0: |
| 1852 | filter_results = self.parse_generated() |
| 1853 | msg = "Finished building %s for %s" %(self.source_dir, self.platform.name) |
| 1854 | |
| 1855 | results = {'msg': msg, 'filter': filter_results} |
| 1856 | |
| 1857 | else: |
| 1858 | self.instance.status = "failed" |
| 1859 | self.instance.reason = "Cmake build failure" |
| 1860 | results = {"returncode": p.returncode} |
| 1861 | |
| 1862 | |
| 1863 | if out: |
| 1864 | with open(os.path.join(self.build_dir, self.log), "a") as log: |
| 1865 | log_msg = out.decode(sys.getdefaultencoding()) |
| 1866 | log.write(log_msg) |
| 1867 | |
| 1868 | return results |
| 1869 | |
| 1870 | |
| 1871 | class FilterBuilder(CMake): |
| 1872 | |
| 1873 | def __init__(self, testcase, platform, source_dir, build_dir): |
| 1874 | super().__init__(testcase, platform, source_dir, build_dir) |
| 1875 | |
| 1876 | self.log = "config-sanitycheck.log" |
| 1877 | |
| 1878 | def parse_generated(self): |
| 1879 | |
| 1880 | if self.platform.name == "unit_testing": |
| 1881 | return {} |
| 1882 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1883 | cmake_cache_path = os.path.join(self.build_dir, "CMakeCache.txt") |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1884 | defconfig_path = os.path.join(self.build_dir, "zephyr", ".config") |
| 1885 | |
| 1886 | with open(defconfig_path, "r") as fp: |
| 1887 | defconfig = {} |
| 1888 | for line in fp.readlines(): |
| 1889 | m = self.config_re.match(line) |
| 1890 | if not m: |
| 1891 | if line.strip() and not line.startswith("#"): |
| 1892 | sys.stderr.write("Unrecognized line %s\n" % line) |
| 1893 | continue |
| 1894 | defconfig[m.group(1)] = m.group(2).strip() |
| 1895 | |
| 1896 | self.defconfig = defconfig |
| 1897 | |
| 1898 | cmake_conf = {} |
| 1899 | try: |
| 1900 | cache = CMakeCache.from_file(cmake_cache_path) |
| 1901 | except FileNotFoundError: |
| 1902 | cache = {} |
| 1903 | |
| 1904 | for k in iter(cache): |
| 1905 | cmake_conf[k.name] = k.value |
| 1906 | |
| 1907 | self.cmake_cache = cmake_conf |
| 1908 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1909 | filter_data = { |
| 1910 | "ARCH": self.platform.arch, |
| 1911 | "PLATFORM": self.platform.name |
| 1912 | } |
| 1913 | filter_data.update(os.environ) |
| 1914 | filter_data.update(self.defconfig) |
| 1915 | filter_data.update(self.cmake_cache) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1916 | |
Anas Nashif | 556f3cb | 2019-11-05 15:36:15 -0800 | [diff] [blame] | 1917 | dts_path = os.path.join(self.build_dir, "zephyr", self.platform.name + ".dts.pre.tmp") |
Kumar Gala | ae88e44 | 2019-11-06 04:55:24 -0600 | [diff] [blame] | 1918 | if self.testcase and self.testcase.tc_filter: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1919 | try: |
Kumar Gala | ae88e44 | 2019-11-06 04:55:24 -0600 | [diff] [blame] | 1920 | if os.path.exists(dts_path): |
| 1921 | edt = edtlib.EDT(dts_path, [os.path.join(ZEPHYR_BASE, "dts", "bindings")]) |
| 1922 | else: |
| 1923 | edt = None |
Kumar Gala | 7733b94 | 2019-09-12 17:08:43 -0500 | [diff] [blame] | 1924 | res = expr_parser.parse(self.testcase.tc_filter, filter_data, edt) |
| 1925 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 1926 | except (ValueError, SyntaxError) as se: |
| 1927 | sys.stderr.write( |
| 1928 | "Failed processing %s\n" % self.testcase.yamlfile) |
| 1929 | raise se |
| 1930 | |
| 1931 | if not res: |
| 1932 | return {os.path.join(self.platform.name, self.testcase.name): True} |
| 1933 | else: |
| 1934 | return {os.path.join(self.platform.name, self.testcase.name): False} |
| 1935 | else: |
| 1936 | self.platform.filter_data = filter_data |
| 1937 | return filter_data |
| 1938 | |
| 1939 | |
| 1940 | class ProjectBuilder(FilterBuilder): |
| 1941 | |
| 1942 | def __init__(self, suite, instance): |
| 1943 | super().__init__(instance.testcase, instance.platform, instance.testcase.source_dir, instance.build_dir) |
| 1944 | |
| 1945 | self.log = "build.log" |
| 1946 | self.instance = instance |
| 1947 | self.suite = suite |
| 1948 | |
| 1949 | def setup_handler(self): |
| 1950 | |
| 1951 | instance = self.instance |
| 1952 | args = [] |
| 1953 | |
| 1954 | # FIXME: Needs simplification |
| 1955 | if instance.platform.simulation == "qemu": |
| 1956 | instance.handler = QEMUHandler(instance, "qemu") |
| 1957 | args.append("QEMU_PIPE=%s" % instance.handler.get_fifo()) |
| 1958 | instance.handler.call_make_run = True |
| 1959 | elif instance.testcase.type == "unit": |
| 1960 | instance.handler = BinaryHandler(instance, "unit") |
| 1961 | instance.handler.binary = os.path.join(instance.build_dir, "testbinary") |
| 1962 | elif instance.platform.type == "native": |
| 1963 | instance.handler = BinaryHandler(instance, "native") |
| 1964 | instance.handler.binary = os.path.join(instance.build_dir, "zephyr", "zephyr.exe") |
| 1965 | elif instance.platform.simulation == "nsim": |
| 1966 | if find_executable("nsimdrv"): |
| 1967 | instance.handler = BinaryHandler(instance, "nsim") |
| 1968 | instance.handler.call_make_run = True |
| 1969 | elif instance.platform.simulation == "renode": |
| 1970 | if find_executable("renode"): |
| 1971 | instance.handler = BinaryHandler(instance, "renode") |
| 1972 | instance.handler.pid_fn = os.path.join(instance.build_dir, "renode.pid") |
| 1973 | instance.handler.call_make_run = True |
| 1974 | elif options.device_testing: |
| 1975 | instance.handler = DeviceHandler(instance, "device") |
| 1976 | |
| 1977 | if instance.handler: |
| 1978 | instance.handler.args = args |
| 1979 | |
| 1980 | def process(self, message): |
| 1981 | op = message.get('op') |
| 1982 | |
| 1983 | if not self.instance.handler: |
| 1984 | self.setup_handler() |
| 1985 | |
| 1986 | # The build process, call cmake and build with configured generator |
| 1987 | if op == "cmake": |
| 1988 | results = self.cmake() |
| 1989 | if self.instance.status == "failed": |
| 1990 | pipeline.put({"op": "report", "test": self.instance}) |
| 1991 | elif options.cmake_only: |
| 1992 | pipeline.put({"op": "report", "test": self.instance}) |
| 1993 | else: |
| 1994 | if self.instance.name in results['filter'] and results['filter'][self.instance.name]: |
| 1995 | verbose("filtering %s" % self.instance.name) |
| 1996 | self.instance.status = "skipped" |
| 1997 | self.instance.reason = "filter" |
| 1998 | pipeline.put({"op": "report", "test": self.instance}) |
| 1999 | else: |
| 2000 | pipeline.put({"op": "build", "test": self.instance}) |
| 2001 | |
| 2002 | |
| 2003 | elif op == "build": |
| 2004 | verbose("build test: %s" %self.instance.name) |
| 2005 | results = self.build() |
| 2006 | |
| 2007 | if results.get('returncode', 1) > 0: |
| 2008 | pipeline.put({"op": "report", "test": self.instance}) |
| 2009 | else: |
| 2010 | if self.instance.run: |
| 2011 | pipeline.put({"op": "run", "test": self.instance}) |
| 2012 | else: |
| 2013 | pipeline.put({"op": "report", "test": self.instance}) |
| 2014 | # Run the generated binary using one of the supported handlers |
| 2015 | elif op == "run": |
| 2016 | verbose("run test: %s" %self.instance.name) |
| 2017 | self.run() |
| 2018 | self.instance.status, _ = self.instance.handler.get_state() |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2019 | pipeline.put({ |
| 2020 | "op": "report", |
| 2021 | "test": self.instance, |
| 2022 | "state": "executed", |
| 2023 | "status": self.instance.status, |
Stephanos Ioannidis | 9388900 | 2019-11-11 20:31:03 +0900 | [diff] [blame] | 2024 | "reason": self.instance.reason} |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2025 | ) |
| 2026 | |
| 2027 | # Report results and output progress to screen |
| 2028 | elif op == "report": |
Anas Nashif | c1ea452 | 2019-10-11 07:32:45 -0700 | [diff] [blame] | 2029 | with report_lock: |
| 2030 | self.report_out() |
| 2031 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2032 | def report_out(self): |
| 2033 | total_tests_width = len(str(self.suite.total_tests)) |
| 2034 | self.suite.total_done += 1 |
| 2035 | instance = self.instance |
| 2036 | |
| 2037 | if instance.status in ["failed", "timeout"]: |
| 2038 | self.suite.total_failed += 1 |
| 2039 | if VERBOSE or not TERMINAL: |
| 2040 | status = COLOR_RED + "FAILED " + COLOR_NORMAL + instance.reason |
| 2041 | else: |
| 2042 | info( |
Anas Nashif | c1ea452 | 2019-10-11 07:32:45 -0700 | [diff] [blame] | 2043 | "\n{:<25} {:<50} {}FAILED{}: {}".format( |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2044 | instance.platform.name, |
| 2045 | instance.testcase.name, |
| 2046 | COLOR_RED, |
| 2047 | COLOR_NORMAL, |
| 2048 | instance.reason), False) |
Anas Nashif | c1ea452 | 2019-10-11 07:32:45 -0700 | [diff] [blame] | 2049 | if not VERBOSE: |
Ulf Magnusson | e73d286 | 2019-10-29 11:54:02 +0100 | [diff] [blame] | 2050 | log_info_file(instance) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2051 | elif instance.status == "skipped": |
| 2052 | self.suite.total_skipped += 1 |
| 2053 | status = COLOR_YELLOW + "SKIPPED" + COLOR_NORMAL |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2054 | else: |
| 2055 | status = COLOR_GREEN + "PASSED" + COLOR_NORMAL |
| 2056 | |
| 2057 | if VERBOSE or not TERMINAL: |
| 2058 | if options.cmake_only: |
| 2059 | more_info = "cmake" |
| 2060 | elif instance.status == "skipped": |
| 2061 | more_info = instance.reason |
| 2062 | else: |
| 2063 | if instance.handler and instance.run: |
| 2064 | more_info = instance.handler.type_str |
| 2065 | htime = instance.handler.duration |
| 2066 | if htime: |
| 2067 | more_info += " {:.3f}s".format(htime) |
| 2068 | else: |
| 2069 | more_info = "build" |
| 2070 | |
| 2071 | info("{:>{}}/{} {:<25} {:<50} {} ({})".format( |
| 2072 | self.suite.total_done, total_tests_width, self.suite.total_tests, instance.platform.name, |
| 2073 | instance.testcase.name, status, more_info)) |
| 2074 | |
| 2075 | if instance.status in ["failed", "timeout"]: |
Ulf Magnusson | e73d286 | 2019-10-29 11:54:02 +0100 | [diff] [blame] | 2076 | log_info_file(instance) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2077 | else: |
| 2078 | sys.stdout.write("\rtotal complete: %s%4d/%4d%s %2d%% skipped: %s%4d%s, failed: %s%4d%s" % ( |
| 2079 | COLOR_GREEN, |
| 2080 | self.suite.total_done, |
| 2081 | self.suite.total_tests, |
| 2082 | COLOR_NORMAL, |
| 2083 | int((float(self.suite.total_done) / self.suite.total_tests) * 100), |
| 2084 | COLOR_YELLOW if self.suite.total_skipped > 0 else COLOR_NORMAL, |
| 2085 | self.suite.total_skipped, |
| 2086 | COLOR_NORMAL, |
| 2087 | COLOR_RED if self.suite.total_failed > 0 else COLOR_NORMAL, |
| 2088 | self.suite.total_failed, |
| 2089 | COLOR_NORMAL |
| 2090 | ) |
| 2091 | ) |
| 2092 | sys.stdout.flush() |
| 2093 | |
| 2094 | def cmake(self): |
| 2095 | |
| 2096 | instance = self.instance |
| 2097 | args = self.testcase.extra_args[:] |
| 2098 | |
| 2099 | if options.extra_args: |
| 2100 | args += options.extra_args |
| 2101 | |
| 2102 | if instance.handler: |
| 2103 | args += instance.handler.args |
| 2104 | |
| 2105 | # merge overlay files into one variable |
| 2106 | overlays = "" |
| 2107 | idx = 0 |
| 2108 | for arg in args: |
| 2109 | match = re.search('OVERLAY_CONFIG="(.*)"', arg) |
| 2110 | if match: |
| 2111 | overlays += match.group(1) |
| 2112 | del args[idx] |
| 2113 | idx += 1 |
| 2114 | |
Jan Van Winkel | 21212f3 | 2019-09-12 00:03:35 +0200 | [diff] [blame] | 2115 | if (self.testcase.extra_configs or options.coverage or |
| 2116 | options.enable_asan): |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2117 | args.append("OVERLAY_CONFIG=\"%s %s\"" %(overlays, |
| 2118 | os.path.join(instance.build_dir, |
| 2119 | "sanitycheck", "testcase_extra.conf"))) |
| 2120 | |
| 2121 | results = self.run_cmake(args) |
| 2122 | return results |
| 2123 | |
| 2124 | def build(self): |
| 2125 | results = self.run_build(['--build', self.build_dir]) |
| 2126 | return results |
| 2127 | |
| 2128 | def run(self): |
| 2129 | |
| 2130 | instance = self.instance |
| 2131 | |
| 2132 | if instance.handler.type_str == "device": |
| 2133 | instance.handler.suite = self.suite |
| 2134 | |
| 2135 | instance.handler.handle() |
| 2136 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2137 | sys.stdout.flush() |
| 2138 | |
| 2139 | |
| 2140 | pipeline = queue.LifoQueue() |
| 2141 | |
| 2142 | class BoundedExecutor(concurrent.futures.ThreadPoolExecutor): |
| 2143 | """BoundedExecutor behaves as a ThreadPoolExecutor which will block on |
| 2144 | calls to submit() once the limit given as "bound" work items are queued for |
| 2145 | execution. |
| 2146 | :param bound: Integer - the maximum number of items in the work queue |
| 2147 | :param max_workers: Integer - the size of the thread pool |
| 2148 | """ |
| 2149 | def __init__(self, bound, max_workers, **kwargs): |
| 2150 | super().__init__(max_workers) |
| 2151 | #self.executor = ThreadPoolExecutor(max_workers=max_workers) |
| 2152 | self.semaphore = BoundedSemaphore(bound + max_workers) |
| 2153 | |
| 2154 | def submit(self, fn, *args, **kwargs): |
| 2155 | self.semaphore.acquire() |
| 2156 | try: |
| 2157 | future = super().submit(fn, *args, **kwargs) |
| 2158 | except: |
| 2159 | self.semaphore.release() |
| 2160 | raise |
| 2161 | else: |
| 2162 | future.add_done_callback(lambda x: self.semaphore.release()) |
| 2163 | return future |
| 2164 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2165 | |
| 2166 | class TestSuite: |
Andrew Boie | 60823c2 | 2017-02-10 09:43:07 -0800 | [diff] [blame] | 2167 | config_re = re.compile('(CONFIG_[A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$') |
Anas Nashif | 1c65b6b | 2018-12-02 19:12:21 -0500 | [diff] [blame] | 2168 | dt_re = re.compile('([A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$') |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2169 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2170 | tc_schema = scl.yaml_load( |
Oleg Zhurakivskyy | 4282208 | 2018-08-17 14:54:41 +0300 | [diff] [blame] | 2171 | os.path.join(ZEPHYR_BASE, |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2172 | "scripts", "sanity_chk", "testcase-schema.yaml")) |
Inaky Perez-Gonzalez | 662dde6 | 2017-07-24 10:24:35 -0700 | [diff] [blame] | 2173 | |
Anas Nashif | 37f9dc5 | 2018-02-23 08:53:46 -0600 | [diff] [blame] | 2174 | def __init__(self, board_root_list, testcase_roots, outdir): |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2175 | |
| 2176 | self.roots = testcase_roots |
| 2177 | if not isinstance(board_root_list, list): |
| 2178 | self.board_roots= [board_root_list] |
| 2179 | else: |
| 2180 | self.board_roots = board_root_list |
| 2181 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2182 | # Keep track of which test cases we've filtered out and why |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2183 | self.testcases = {} |
| 2184 | self.platforms = [] |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2185 | self.default_platforms = [] |
Andrew Boie | b391e66 | 2015-08-31 15:25:45 -0700 | [diff] [blame] | 2186 | self.outdir = os.path.abspath(outdir) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2187 | self.discards = None |
Kumar Gala | c84235e | 2018-04-10 13:32:51 -0500 | [diff] [blame] | 2188 | self.load_errors = 0 |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2189 | self.instances = dict() |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2190 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2191 | self.total_tests = 0 # number of test instances |
| 2192 | self.total_cases = 0 # number of test cases |
| 2193 | self.total_done = 0 # tests completed |
| 2194 | self.total_failed = 0 |
| 2195 | self.total_skipped = 0 |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2196 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2197 | self.total_platforms = 0 |
| 2198 | self.start_time = 0 |
| 2199 | self.duration = 0 |
| 2200 | self.warnings = 0 |
| 2201 | self.cv = threading.Condition() |
Anas Nashif | 61e2163 | 2018-04-08 13:30:16 -0500 | [diff] [blame] | 2202 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2203 | # hardcoded for now |
| 2204 | self.connected_hardware = [] |
Andrew Boie | 3d34871 | 2016-04-08 11:52:13 -0700 | [diff] [blame] | 2205 | |
Inaky Perez-Gonzalez | 662dde6 | 2017-07-24 10:24:35 -0700 | [diff] [blame] | 2206 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2207 | if options.jobs: |
| 2208 | self.jobs = options.jobs |
| 2209 | elif options.build_only: |
| 2210 | self.jobs = multiprocessing.cpu_count() * 2 |
Andy Ross | 9c9162d | 2019-01-03 10:50:53 -0800 | [diff] [blame] | 2211 | else: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2212 | self.jobs = multiprocessing.cpu_count() |
Daniel Leung | 6b17007 | 2016-04-07 12:10:25 -0700 | [diff] [blame] | 2213 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2214 | info("JOBS: %d" % self.jobs) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2215 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2216 | def update(self): |
| 2217 | self.total_tests = len(self.instances) |
| 2218 | self.total_cases = len(self.testcases) |
Anas Nashif | bd166f4 | 2017-09-02 12:32:08 -0400 | [diff] [blame] | 2219 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2220 | |
| 2221 | def compare_metrics(self, filename): |
| 2222 | # name, datatype, lower results better |
| 2223 | interesting_metrics = [("ram_size", int, True), |
| 2224 | ("rom_size", int, True)] |
| 2225 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2226 | |
| 2227 | if not os.path.exists(filename): |
| 2228 | info("Cannot compare metrics, %s not found" % filename) |
| 2229 | return [] |
| 2230 | |
| 2231 | results = [] |
| 2232 | saved_metrics = {} |
| 2233 | with open(filename) as fp: |
| 2234 | cr = csv.DictReader(fp) |
| 2235 | for row in cr: |
| 2236 | d = {} |
| 2237 | for m, _, _ in interesting_metrics: |
| 2238 | d[m] = row[m] |
| 2239 | saved_metrics[(row["test"], row["platform"])] = d |
| 2240 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2241 | for instance in self.instances.values(): |
| 2242 | mkey = (instance.testcase.name, instance.platform.name) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2243 | if mkey not in saved_metrics: |
| 2244 | continue |
| 2245 | sm = saved_metrics[mkey] |
| 2246 | for metric, mtype, lower_better in interesting_metrics: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2247 | if metric not in instance.metrics: |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2248 | continue |
| 2249 | if sm[metric] == "": |
| 2250 | continue |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2251 | delta = instance.metrics.get(metric, 0) - mtype(sm[metric]) |
Andrew Boie | ea7928f | 2015-08-14 14:27:38 -0700 | [diff] [blame] | 2252 | if delta == 0: |
| 2253 | continue |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2254 | results.append((instance, metric, instance.metrics.get(metric, 0 ), delta, |
Andrew Boie | ea7928f | 2015-08-14 14:27:38 -0700 | [diff] [blame] | 2255 | lower_better)) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2256 | return results |
| 2257 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2258 | def misc_reports(self, report, show_footprint, all_deltas, |
| 2259 | footprint_threshold, last_metrics): |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 2260 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2261 | if not report: |
| 2262 | return |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 2263 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2264 | deltas = self.compare_metrics(report) |
| 2265 | warnings = 0 |
| 2266 | if deltas and show_footprint: |
| 2267 | for i, metric, value, delta, lower_better in deltas: |
| 2268 | if not all_deltas and ((delta < 0 and lower_better) or |
| 2269 | (delta > 0 and not lower_better)): |
| 2270 | continue |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 2271 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2272 | percentage = (float(delta) / float(value - delta)) |
| 2273 | if not all_deltas and (percentage < |
| 2274 | (footprint_threshold / 100.0)): |
| 2275 | continue |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 2276 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2277 | info("{:<25} {:<60} {}{}{}: {} {:<+4}, is now {:6} {:+.2%}".format( |
| 2278 | i.platform.name, i.testcase.name, COLOR_YELLOW, |
| 2279 | "INFO" if all_deltas else "WARNING", COLOR_NORMAL, |
| 2280 | metric, delta, value, percentage)) |
| 2281 | warnings += 1 |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 2282 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2283 | if warnings: |
| 2284 | info("Deltas based on metrics from last %s" % |
| 2285 | ("release" if not last_metrics else "run")) |
Anas Nashif | 61e2163 | 2018-04-08 13:30:16 -0500 | [diff] [blame] | 2286 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2287 | def summary(self, unrecognized_sections): |
| 2288 | failed = 0 |
| 2289 | for instance in self.instances.values(): |
| 2290 | if instance.status == "failed": |
| 2291 | failed += 1 |
| 2292 | elif instance.metrics.get("unrecognized") and not unrecognized_sections: |
| 2293 | info("%sFAILED%s: %s has unrecognized binary sections: %s" % |
| 2294 | (COLOR_RED, COLOR_NORMAL, instance.name, |
| 2295 | str(instance.metrics.get("unrecognized", [])))) |
| 2296 | failed += 1 |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 2297 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2298 | if self.total_tests and self.total_tests != self.total_skipped: |
| 2299 | pass_rate = (float(self.total_tests - self.total_failed - self.total_skipped)/ float(self.total_tests - self.total_skipped)) |
| 2300 | else: |
| 2301 | pass_rate = 0 |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 2302 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2303 | info("{}{} of {}{} tests passed ({:.2%}), {}{}{} failed, {} skipped with {}{}{} warnings in {:.2f} seconds".format( |
| 2304 | COLOR_RED if failed else COLOR_GREEN, |
| 2305 | self.total_tests - self.total_failed - self.total_skipped, |
| 2306 | self.total_tests, |
| 2307 | COLOR_NORMAL, |
| 2308 | pass_rate, |
| 2309 | COLOR_RED if self.total_failed else COLOR_NORMAL, |
| 2310 | self.total_failed, |
| 2311 | COLOR_NORMAL, |
| 2312 | self.total_skipped, |
| 2313 | COLOR_YELLOW if self.warnings else COLOR_NORMAL, |
| 2314 | self.warnings, |
| 2315 | COLOR_NORMAL, |
| 2316 | self.duration)) |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 2317 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2318 | platforms = set(p.platform for p in self.instances.values()) |
| 2319 | self.total_platforms = len(self.platforms) |
| 2320 | if self.platforms: |
| 2321 | info("In total {} test cases were executed on {} out of total {} platforms ({:02.2f}%)".format( |
| 2322 | self.total_cases, |
| 2323 | len(platforms), |
| 2324 | self.total_platforms, |
| 2325 | (100 * len(platforms) / len(self.platforms)) |
| 2326 | )) |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 2327 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2328 | def save_reports(self): |
| 2329 | if not self.instances: |
| 2330 | return |
Anas Nashif | 61e2163 | 2018-04-08 13:30:16 -0500 | [diff] [blame] | 2331 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2332 | report_name = "sanitycheck" |
| 2333 | if options.report_name: |
| 2334 | report_name = options.report_name |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 2335 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2336 | if options.report_dir: |
Paul Sokolovsky | 3ea1869 | 2019-10-15 17:18:39 +0300 | [diff] [blame] | 2337 | os.makedirs(options.report_dir, exist_ok=True) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2338 | filename = os.path.join(options.report_dir, report_name) |
| 2339 | outdir = options.report_dir |
| 2340 | else: |
| 2341 | filename = os.path.join(options.outdir, report_name) |
| 2342 | outdir = options.outdir |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 2343 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2344 | if not options.no_update: |
| 2345 | self.xunit_report(filename + ".xml") |
| 2346 | self.csv_report(filename + ".csv") |
| 2347 | self.target_report(outdir) |
| 2348 | if self.discards: |
| 2349 | self.discard_report(filename + "_discard.csv") |
| 2350 | |
| 2351 | if options.release: |
| 2352 | self.csv_report(RELEASE_DATA) |
| 2353 | |
| 2354 | if log_file: |
| 2355 | log_file.close() |
| 2356 | |
| 2357 | def load_hardware_map_from_cmdline(self, serial, platform): |
| 2358 | device = { |
| 2359 | "serial": serial, |
| 2360 | "platform": platform, |
| 2361 | "counter": 0, |
| 2362 | "available": True, |
| 2363 | "connected": True |
| 2364 | } |
| 2365 | self.connected_hardware = [device] |
| 2366 | |
| 2367 | def load_hardware_map(self, map_file): |
| 2368 | with open(map_file, 'r') as stream: |
| 2369 | try: |
| 2370 | self.connected_hardware = yaml.safe_load(stream) |
| 2371 | except yaml.YAMLError as exc: |
| 2372 | print(exc) |
| 2373 | for i in self.connected_hardware: |
| 2374 | i['counter'] = 0 |
| 2375 | |
| 2376 | def add_configurations(self): |
| 2377 | |
| 2378 | for board_root in self.board_roots: |
| 2379 | board_root = os.path.abspath(board_root) |
| 2380 | |
| 2381 | debug("Reading platform configuration files under %s..." % |
| 2382 | board_root) |
| 2383 | |
| 2384 | for file in glob.glob(os.path.join(board_root, "*", "*", "*.yaml")): |
| 2385 | verbose("Found plaform configuration " + file) |
| 2386 | try: |
| 2387 | platform = Platform() |
| 2388 | platform.load(file) |
| 2389 | if platform.sanitycheck: |
| 2390 | self.platforms.append(platform) |
| 2391 | if platform.default: |
| 2392 | self.default_platforms.append(platform.name) |
| 2393 | |
| 2394 | except RuntimeError as e: |
| 2395 | error("E: %s: can't load: %s" % (file, e)) |
| 2396 | self.load_errors += 1 |
| 2397 | |
Anas Nashif | eabaa7f | 2019-11-18 07:49:17 -0800 | [diff] [blame] | 2398 | |
| 2399 | def get_all_tests(self): |
| 2400 | tests = [] |
| 2401 | for _, tc in self.testcases.items(): |
| 2402 | for case in tc.cases: |
| 2403 | tests.append(case) |
| 2404 | |
| 2405 | return tests |
| 2406 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2407 | @staticmethod |
| 2408 | def get_toolchain(): |
| 2409 | toolchain = os.environ.get("ZEPHYR_TOOLCHAIN_VARIANT", None) or \ |
| 2410 | os.environ.get("ZEPHYR_GCC_VARIANT", None) |
| 2411 | |
| 2412 | if toolchain == "gccarmemb": |
| 2413 | # Remove this translation when gccarmemb is no longer supported. |
| 2414 | toolchain = "gnuarmemb" |
| 2415 | |
Anas Nashif | b4bdd66 | 2018-08-15 17:12:28 -0500 | [diff] [blame] | 2416 | try: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2417 | if not toolchain: |
| 2418 | raise SanityRuntimeError("E: Variable ZEPHYR_TOOLCHAIN_VARIANT is not defined") |
Anas Nashif | b4bdd66 | 2018-08-15 17:12:28 -0500 | [diff] [blame] | 2419 | except Exception as e: |
| 2420 | print(str(e)) |
| 2421 | sys.exit(2) |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2422 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2423 | return toolchain |
| 2424 | |
| 2425 | |
| 2426 | def add_testcases(self): |
| 2427 | for root in self.roots: |
| 2428 | root = os.path.abspath(root) |
| 2429 | |
| 2430 | debug("Reading test case configuration files under %s..." %root) |
| 2431 | |
| 2432 | for dirpath, dirnames, filenames in os.walk(root, topdown=True): |
| 2433 | verbose("scanning %s" % dirpath) |
| 2434 | if 'sample.yaml' in filenames: |
| 2435 | filename = 'sample.yaml' |
| 2436 | elif 'testcase.yaml' in filenames: |
| 2437 | filename = 'testcase.yaml' |
| 2438 | else: |
| 2439 | continue |
| 2440 | |
| 2441 | verbose("Found possible test case in " + dirpath) |
| 2442 | |
| 2443 | dirnames[:] = [] |
| 2444 | tc_path = os.path.join(dirpath, filename) |
| 2445 | self.add_testcase(tc_path, root) |
| 2446 | |
| 2447 | def add_testcase(self, tc_data_file, root): |
| 2448 | try: |
| 2449 | parsed_data = SanityConfigParser(tc_data_file, self.tc_schema) |
| 2450 | parsed_data.load() |
| 2451 | |
| 2452 | tc_path = os.path.dirname(tc_data_file) |
| 2453 | workdir = os.path.relpath(tc_path, root) |
| 2454 | |
| 2455 | for name in parsed_data.tests.keys(): |
| 2456 | tc = TestCase() |
| 2457 | tc.name = tc.get_unique(root, workdir, name) |
| 2458 | |
| 2459 | tc_dict = parsed_data.get_test(name, testcase_valid_keys) |
| 2460 | |
| 2461 | tc.source_dir = tc_path |
| 2462 | tc.yamlfile = tc_data_file |
| 2463 | |
| 2464 | tc.id = name |
| 2465 | tc.type = tc_dict["type"] |
| 2466 | tc.tags = tc_dict["tags"] |
| 2467 | tc.extra_args = tc_dict["extra_args"] |
| 2468 | tc.extra_configs = tc_dict["extra_configs"] |
| 2469 | tc.arch_whitelist = tc_dict["arch_whitelist"] |
| 2470 | tc.arch_exclude = tc_dict["arch_exclude"] |
| 2471 | tc.skip = tc_dict["skip"] |
| 2472 | tc.platform_exclude = tc_dict["platform_exclude"] |
| 2473 | tc.platform_whitelist = tc_dict["platform_whitelist"] |
| 2474 | tc.toolchain_exclude = tc_dict["toolchain_exclude"] |
| 2475 | tc.toolchain_whitelist = tc_dict["toolchain_whitelist"] |
| 2476 | tc.tc_filter = tc_dict["filter"] |
| 2477 | tc.timeout = tc_dict["timeout"] |
| 2478 | tc.harness = tc_dict["harness"] |
| 2479 | tc.harness_config = tc_dict["harness_config"] |
| 2480 | tc.build_only = tc_dict["build_only"] |
| 2481 | tc.build_on_all = tc_dict["build_on_all"] |
| 2482 | tc.slow = tc_dict["slow"] |
| 2483 | tc.min_ram = tc_dict["min_ram"] |
| 2484 | tc.depends_on = tc_dict["depends_on"] |
| 2485 | tc.min_flash = tc_dict["min_flash"] |
| 2486 | tc.extra_sections = tc_dict["extra_sections"] |
| 2487 | |
| 2488 | tc.parse_subcases(tc_path) |
| 2489 | |
| 2490 | if tc.name: |
| 2491 | self.testcases[tc.name] = tc |
| 2492 | |
| 2493 | except Exception as e: |
| 2494 | error("E: %s: can't load (skipping): %s" % (tc_data_file, e)) |
| 2495 | self.load_errors += 1 |
| 2496 | return False |
| 2497 | |
| 2498 | return True |
| 2499 | |
| 2500 | |
| 2501 | def get_platform(self, name): |
| 2502 | selected_platform = None |
| 2503 | for platform in self.platforms: |
| 2504 | if platform.name == name: |
| 2505 | selected_platform = platform |
| 2506 | break |
| 2507 | return selected_platform |
| 2508 | |
| 2509 | def get_last_failed(self): |
| 2510 | last_run = os.path.join(options.outdir, "sanitycheck.csv") |
| 2511 | try: |
| 2512 | if not os.path.exists(last_run): |
| 2513 | raise SanityRuntimeError("Couldn't find last sanitycheck run.: %s" %last_run) |
| 2514 | except Exception as e: |
| 2515 | print(str(e)) |
| 2516 | sys.exit(2) |
| 2517 | |
| 2518 | total_tests = 0 |
| 2519 | with open(last_run, "r") as fp: |
| 2520 | cr = csv.DictReader(fp) |
| 2521 | instance_list = [] |
| 2522 | for row in cr: |
| 2523 | total_tests += 1 |
| 2524 | if row["passed"] == "True": |
| 2525 | continue |
| 2526 | test = row["test"] |
| 2527 | platform = self.get_platform(row["platform"]) |
| 2528 | instance = TestInstance(self.testcases[test], platform, self.outdir) |
Jan Van Winkel | 21212f3 | 2019-09-12 00:03:35 +0200 | [diff] [blame] | 2529 | instance.create_overlay(platform) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2530 | instance_list.append(instance) |
| 2531 | self.add_instances(instance_list) |
| 2532 | |
| 2533 | tests_to_run = len(self.instances) |
| 2534 | info("%d tests passed already, retyring %d tests" %(total_tests - tests_to_run, tests_to_run)) |
| 2535 | |
| 2536 | def load_from_file(self, file): |
| 2537 | try: |
| 2538 | if not os.path.exists(file): |
| 2539 | raise SanityRuntimeError( |
| 2540 | "Couldn't find input file with list of tests.") |
| 2541 | except Exception as e: |
| 2542 | print(str(e)) |
| 2543 | sys.exit(2) |
| 2544 | |
| 2545 | with open(file, "r") as fp: |
| 2546 | cr = csv.DictReader(fp) |
| 2547 | instance_list = [] |
| 2548 | for row in cr: |
| 2549 | if row["arch"] == "arch": |
| 2550 | continue |
| 2551 | test = row["test"] |
| 2552 | platform = self.get_platform(row["platform"]) |
| 2553 | instance = TestInstance(self.testcases[test], platform, self.outdir) |
Jan Van Winkel | 21212f3 | 2019-09-12 00:03:35 +0200 | [diff] [blame] | 2554 | instance.create_overlay(platform) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2555 | instance_list.append(instance) |
| 2556 | self.add_instances(instance_list) |
| 2557 | |
| 2558 | |
| 2559 | def apply_filters(self): |
| 2560 | |
| 2561 | toolchain = self.get_toolchain() |
| 2562 | |
| 2563 | discards = {} |
| 2564 | platform_filter = options.platform |
| 2565 | testcase_filter = run_individual_tests |
| 2566 | arch_filter = options.arch |
| 2567 | tag_filter = options.tag |
| 2568 | exclude_tag = options.exclude_tag |
| 2569 | |
| 2570 | verbose("platform filter: " + str(platform_filter)) |
| 2571 | verbose(" arch_filter: " + str(arch_filter)) |
| 2572 | verbose(" tag_filter: " + str(tag_filter)) |
| 2573 | verbose(" exclude_tag: " + str(exclude_tag)) |
| 2574 | |
| 2575 | default_platforms = False |
| 2576 | |
| 2577 | if platform_filter: |
| 2578 | platforms = list(filter(lambda p: p.name in platform_filter, self.platforms)) |
| 2579 | else: |
| 2580 | platforms = self.platforms |
| 2581 | |
| 2582 | if options.all: |
| 2583 | info("Selecting all possible platforms per test case") |
| 2584 | # When --all used, any --platform arguments ignored |
| 2585 | platform_filter = [] |
| 2586 | elif not platform_filter: |
| 2587 | info("Selecting default platforms per test case") |
| 2588 | default_platforms = True |
| 2589 | |
| 2590 | info("Building initial testcase list...") |
| 2591 | |
| 2592 | for tc_name, tc in self.testcases.items(): |
| 2593 | # list of instances per testcase, aka configurations. |
| 2594 | instance_list = [] |
| 2595 | for plat in platforms: |
| 2596 | instance = TestInstance(tc, plat, self.outdir) |
| 2597 | |
| 2598 | if (plat.arch == "unit") != (tc.type == "unit"): |
| 2599 | # Discard silently |
| 2600 | continue |
| 2601 | |
| 2602 | if options.device_testing and instance.build_only: |
| 2603 | discards[instance] = "Not runnable on device" |
| 2604 | continue |
| 2605 | |
| 2606 | if tc.skip: |
| 2607 | discards[instance] = "Skip filter" |
| 2608 | continue |
| 2609 | |
| 2610 | if tc.build_on_all and not platform_filter: |
| 2611 | platform_filter = [] |
| 2612 | |
| 2613 | if tag_filter and not tc.tags.intersection(tag_filter): |
| 2614 | discards[instance] = "Command line testcase tag filter" |
| 2615 | continue |
| 2616 | |
| 2617 | if exclude_tag and tc.tags.intersection(exclude_tag): |
| 2618 | discards[instance] = "Command line testcase exclude filter" |
| 2619 | continue |
| 2620 | |
| 2621 | if testcase_filter and tc_name not in testcase_filter: |
| 2622 | discards[instance] = "Testcase name filter" |
| 2623 | continue |
| 2624 | |
| 2625 | if arch_filter and plat.arch not in arch_filter: |
| 2626 | discards[instance] = "Command line testcase arch filter" |
| 2627 | continue |
| 2628 | |
| 2629 | if tc.arch_whitelist and plat.arch not in tc.arch_whitelist: |
| 2630 | discards[instance] = "Not in test case arch whitelist" |
| 2631 | continue |
| 2632 | |
| 2633 | if tc.arch_exclude and plat.arch in tc.arch_exclude: |
| 2634 | discards[instance] = "In test case arch exclude" |
| 2635 | continue |
| 2636 | |
| 2637 | if tc.platform_exclude and plat.name in tc.platform_exclude: |
| 2638 | discards[instance] = "In test case platform exclude" |
| 2639 | continue |
| 2640 | |
| 2641 | if tc.toolchain_exclude and toolchain in tc.toolchain_exclude: |
| 2642 | discards[instance] = "In test case toolchain exclude" |
| 2643 | continue |
| 2644 | |
| 2645 | if platform_filter and plat.name not in platform_filter: |
| 2646 | discards[instance] = "Command line platform filter" |
| 2647 | continue |
| 2648 | |
| 2649 | if tc.platform_whitelist and plat.name not in tc.platform_whitelist: |
| 2650 | discards[instance] = "Not in testcase platform whitelist" |
| 2651 | continue |
| 2652 | |
| 2653 | if tc.toolchain_whitelist and toolchain not in tc.toolchain_whitelist: |
| 2654 | discards[instance] = "Not in testcase toolchain whitelist" |
| 2655 | continue |
| 2656 | |
| 2657 | if not plat.env_satisfied: |
| 2658 | discards[instance] = "Environment ({}) not satisfied".format(", ".join(plat.env)) |
| 2659 | continue |
| 2660 | |
| 2661 | if not options.force_toolchain \ |
| 2662 | and toolchain and (toolchain not in plat.supported_toolchains) \ |
| 2663 | and tc.type != 'unit': |
| 2664 | discards[instance] = "Not supported by the toolchain" |
| 2665 | continue |
| 2666 | |
| 2667 | if plat.ram < tc.min_ram: |
| 2668 | discards[instance] = "Not enough RAM" |
| 2669 | continue |
| 2670 | |
| 2671 | if tc.depends_on: |
| 2672 | dep_intersection = tc.depends_on.intersection(set(plat.supported)) |
| 2673 | if dep_intersection != set(tc.depends_on): |
| 2674 | discards[instance] = "No hardware support" |
| 2675 | continue |
| 2676 | |
| 2677 | if plat.flash < tc.min_flash: |
| 2678 | discards[instance] = "Not enough FLASH" |
| 2679 | continue |
| 2680 | |
| 2681 | if set(plat.ignore_tags) & tc.tags: |
| 2682 | discards[instance] = "Excluded tags per platform" |
| 2683 | continue |
| 2684 | |
| 2685 | # if nothing stopped us until now, it means this configuration |
| 2686 | # needs to be added. |
| 2687 | instance_list.append(instance) |
| 2688 | |
| 2689 | # no configurations, so jump to next testcase |
| 2690 | if not instance_list: |
| 2691 | continue |
| 2692 | |
| 2693 | # if sanitycheck was launched with no platform options at all, we |
| 2694 | # take all default platforms |
| 2695 | if default_platforms and not tc.build_on_all: |
| 2696 | if tc.platform_whitelist: |
| 2697 | a = set(self.default_platforms) |
| 2698 | b = set(tc.platform_whitelist) |
| 2699 | c = a.intersection(b) |
| 2700 | if c: |
| 2701 | aa = list( filter( lambda tc: tc.platform.name in c, instance_list)) |
| 2702 | self.add_instances(aa) |
| 2703 | else: |
| 2704 | self.add_instances(instance_list[:1]) |
| 2705 | else: |
| 2706 | instances = list( filter( lambda tc: tc.platform.default, instance_list)) |
| 2707 | self.add_instances(instances) |
| 2708 | |
| 2709 | for instance in list(filter(lambda tc: not tc.platform.default, instance_list)): |
| 2710 | discards[instance] = "Not a default test platform" |
| 2711 | |
| 2712 | else: |
| 2713 | self.add_instances(instance_list) |
| 2714 | |
| 2715 | for _, case in self.instances.items(): |
Jan Van Winkel | 21212f3 | 2019-09-12 00:03:35 +0200 | [diff] [blame] | 2716 | case.create_overlay(case.platform) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2717 | |
| 2718 | self.discards = discards |
| 2719 | |
| 2720 | return discards |
| 2721 | |
| 2722 | def add_instances(self, instance_list): |
| 2723 | for instance in instance_list: |
| 2724 | self.instances[instance.name] = instance |
| 2725 | |
| 2726 | def add_tasks_to_queue(self): |
| 2727 | for instance in self.instances.values(): |
| 2728 | if options.test_only: |
| 2729 | if instance.run: |
| 2730 | pipeline.put({"op": "run", "test": instance, "status": "built"}) |
| 2731 | else: |
| 2732 | if instance.status not in ['passed', 'skipped']: |
| 2733 | instance.status = None |
| 2734 | pipeline.put({"op": "cmake", "test": instance}) |
| 2735 | |
| 2736 | return "DONE FEEDING" |
| 2737 | |
| 2738 | def execute(self): |
| 2739 | def calc_one_elf_size(instance): |
| 2740 | if instance.status not in ["failed", "skipped"]: |
| 2741 | if instance.platform.type != "native": |
| 2742 | size_calc = instance.calculate_sizes() |
| 2743 | instance.metrics["ram_size"] = size_calc.get_ram_size() |
| 2744 | instance.metrics["rom_size"] = size_calc.get_rom_size() |
| 2745 | instance.metrics["unrecognized"] = size_calc.unrecognized_sections() |
| 2746 | else: |
| 2747 | instance.metrics["ram_size"] = 0 |
| 2748 | instance.metrics["rom_size"] = 0 |
| 2749 | instance.metrics["unrecognized"] = [] |
| 2750 | |
| 2751 | instance.metrics["handler_time"] = instance.handler.duration if instance.handler else 0 |
| 2752 | |
| 2753 | info("Adding tasks to the queue...") |
| 2754 | # We can use a with statement to ensure threads are cleaned up promptly |
| 2755 | with BoundedExecutor(bound=self.jobs, max_workers=self.jobs) as executor: |
| 2756 | |
| 2757 | # start a future for a thread which sends work in through the queue |
| 2758 | future_to_test = { |
| 2759 | executor.submit(self.add_tasks_to_queue): 'FEEDER DONE'} |
| 2760 | |
| 2761 | while future_to_test: |
| 2762 | # check for status of the futures which are currently working |
| 2763 | done, _ = concurrent.futures.wait( |
| 2764 | future_to_test, timeout=0.25, |
| 2765 | return_when=concurrent.futures.FIRST_COMPLETED) |
| 2766 | |
| 2767 | # if there is incoming work, start a new future |
| 2768 | while not pipeline.empty(): |
| 2769 | # fetch a url from the queue |
| 2770 | message = pipeline.get() |
| 2771 | test = message['test'] |
| 2772 | |
| 2773 | # Start the load operation and mark the future with its URL |
| 2774 | pb = ProjectBuilder(self, test) |
| 2775 | future_to_test[executor.submit(pb.process, message)] = test.name |
| 2776 | |
| 2777 | # process any completed futures |
| 2778 | for future in done: |
| 2779 | test = future_to_test[future] |
| 2780 | try: |
| 2781 | data = future.result() |
| 2782 | except Exception as exc: |
Anas Nashif | 4f04386 | 2019-11-05 06:01:49 -0800 | [diff] [blame] | 2783 | sys.exit('%r generated an exception: %s' % (test, exc)) |
| 2784 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2785 | else: |
| 2786 | if data: |
| 2787 | verbose(data) |
| 2788 | |
| 2789 | # remove the now completed future |
| 2790 | del future_to_test[future] |
| 2791 | |
| 2792 | if options.enable_size_report and not options.cmake_only: |
| 2793 | # Parallelize size calculation |
| 2794 | executor = concurrent.futures.ThreadPoolExecutor(self.jobs) |
| 2795 | futures = [executor.submit(calc_one_elf_size, instance) |
| 2796 | for instance in self.instances.values()] |
| 2797 | concurrent.futures.wait(futures) |
| 2798 | else: |
| 2799 | for instance in self.instances.values(): |
| 2800 | instance.metrics["ram_size"] = 0 |
| 2801 | instance.metrics["rom_size"] = 0 |
| 2802 | instance.metrics["handler_time"] = instance.handler.duration if instance.handler else 0 |
| 2803 | instance.metrics["unrecognized"] = [] |
| 2804 | |
| 2805 | |
| 2806 | def discard_report(self, filename): |
| 2807 | |
| 2808 | try: |
| 2809 | if self.discards is None: |
| 2810 | raise SanityRuntimeError("apply_filters() hasn't been run!") |
| 2811 | except Exception as e: |
| 2812 | error(str(e)) |
| 2813 | sys.exit(2) |
| 2814 | |
| 2815 | with open(filename, "wt") as csvfile: |
| 2816 | fieldnames = ["test", "arch", "platform", "reason"] |
| 2817 | cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep) |
| 2818 | cw.writeheader() |
| 2819 | for instance, reason in sorted(self.discards.items()): |
| 2820 | rowdict = {"test": instance.testcase.name, |
| 2821 | "arch": instance.platform.arch, |
| 2822 | "platform": instance.platform.name, |
| 2823 | "reason": reason} |
| 2824 | cw.writerow(rowdict) |
| 2825 | |
| 2826 | |
| 2827 | def target_report(self, outdir): |
| 2828 | run = "Sanitycheck" |
| 2829 | eleTestsuite = None |
| 2830 | |
| 2831 | platforms = {inst.platform.name for _,inst in self.instances.items()} |
| 2832 | for platform in platforms: |
| 2833 | errors = 0 |
| 2834 | passes = 0 |
| 2835 | fails = 0 |
| 2836 | duration = 0 |
| 2837 | skips = 0 |
| 2838 | for _, instance in self.instances.items(): |
| 2839 | if instance.platform.name != platform: |
| 2840 | continue |
| 2841 | |
| 2842 | handler_time = instance.metrics.get('handler_time', 0) |
| 2843 | duration += handler_time |
| 2844 | for k in instance.results.keys(): |
| 2845 | if instance.results[k] == 'PASS': |
| 2846 | passes += 1 |
| 2847 | elif instance.results[k] == 'BLOCK': |
| 2848 | errors += 1 |
| 2849 | elif instance.results[k] == 'SKIP': |
| 2850 | skips += 1 |
| 2851 | else: |
| 2852 | fails += 1 |
| 2853 | |
| 2854 | eleTestsuites = ET.Element('testsuites') |
| 2855 | eleTestsuite = ET.SubElement(eleTestsuites, 'testsuite', |
| 2856 | name=run, time="%f" % duration, |
| 2857 | tests="%d" % (errors + passes + fails), |
| 2858 | failures="%d" % fails, |
| 2859 | errors="%d" % errors, skipped="%d" %skips) |
| 2860 | |
| 2861 | handler_time = 0 |
| 2862 | |
| 2863 | # print out test results |
| 2864 | for _, instance in self.instances.items(): |
| 2865 | if instance.platform.name != platform: |
| 2866 | continue |
| 2867 | handler_time = instance.metrics.get('handler_time', 0) |
| 2868 | for k in instance.results.keys(): |
| 2869 | eleTestcase = ET.SubElement( |
| 2870 | eleTestsuite, 'testcase', classname="%s:%s" %(instance.platform.name, os.path.basename(instance.testcase.name)), |
| 2871 | name="%s" % (k), time="%f" %handler_time) |
| 2872 | if instance.results[k] in ['FAIL', 'BLOCK']: |
| 2873 | el = None |
| 2874 | |
| 2875 | if instance.results[k] == 'FAIL': |
| 2876 | el = ET.SubElement( |
| 2877 | eleTestcase, |
| 2878 | 'failure', |
| 2879 | type="failure", |
| 2880 | message="failed") |
| 2881 | elif instance.results[k] == 'BLOCK': |
| 2882 | el = ET.SubElement( |
| 2883 | eleTestcase, |
| 2884 | 'error', |
| 2885 | type="failure", |
| 2886 | message="failed") |
| 2887 | p = os.path.join(options.outdir, instance.platform.name, instance.testcase.name) |
| 2888 | log_file = os.path.join(p, "handler.log") |
| 2889 | |
| 2890 | if os.path.exists(log_file): |
| 2891 | with open(log_file, "rb") as f: |
| 2892 | log = f.read().decode("utf-8") |
| 2893 | filtered_string = ''.join(filter(lambda x: x in string.printable, log)) |
| 2894 | el.text = filtered_string |
| 2895 | |
| 2896 | elif instance.results[k] == 'SKIP': |
| 2897 | el = ET.SubElement( |
| 2898 | eleTestcase, |
| 2899 | 'skipped', |
| 2900 | type="skipped", |
| 2901 | message="Skipped") |
| 2902 | |
| 2903 | |
| 2904 | result = ET.tostring(eleTestsuites) |
| 2905 | with open(os.path.join(outdir, platform + ".xml"), 'wb') as f: |
| 2906 | f.write(result) |
| 2907 | |
| 2908 | |
| 2909 | def xunit_report(self, filename): |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2910 | fails = 0 |
| 2911 | passes = 0 |
| 2912 | errors = 0 |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2913 | skips = 0 |
| 2914 | duration = 0 |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2915 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2916 | for instance in self.instances.values(): |
| 2917 | handler_time = instance.metrics.get('handler_time', 0) |
| 2918 | duration += handler_time |
| 2919 | if instance.status == "failed": |
| 2920 | if instance.reason in ['build_error', 'handler_crash']: |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2921 | errors += 1 |
| 2922 | else: |
| 2923 | fails += 1 |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2924 | elif instance.status == 'skipped': |
| 2925 | skips += 1 |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2926 | else: |
| 2927 | passes += 1 |
| 2928 | |
| 2929 | run = "Sanitycheck" |
| 2930 | eleTestsuite = None |
Anas Nashif | 4f02888 | 2017-12-30 11:48:43 -0500 | [diff] [blame] | 2931 | append = options.only_failed |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2932 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2933 | # When we re-run the tests, we re-use the results and update only with |
| 2934 | # the newly run tests. |
Anas Nashif | 0605fa3 | 2017-05-07 08:51:02 -0400 | [diff] [blame] | 2935 | if os.path.exists(filename) and append: |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2936 | tree = ET.parse(filename) |
| 2937 | eleTestsuites = tree.getroot() |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 2938 | eleTestsuite = tree.findall('testsuite')[0] |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2939 | else: |
| 2940 | eleTestsuites = ET.Element('testsuites') |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 2941 | eleTestsuite = ET.SubElement(eleTestsuites, 'testsuite', |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2942 | name=run, time="%f" % duration, |
| 2943 | tests="%d" % (errors + passes + fails + skips), |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 2944 | failures="%d" % fails, |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2945 | errors="%d" %(errors), skip="%s" %(skips)) |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2946 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2947 | for instance in self.instances.values(): |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2948 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2949 | # remove testcases that are a re-run |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2950 | if append: |
| 2951 | for tc in eleTestsuite.findall('testcase'): |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 2952 | if tc.get('classname') == "%s:%s" % ( |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2953 | instance.platform.name, instance.testcase.name): |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2954 | eleTestsuite.remove(tc) |
| 2955 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2956 | handler_time = 0 |
| 2957 | if instance.status != "failed" and instance.handler: |
| 2958 | handler_time = instance.metrics.get("handler_time", 0) |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2959 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 2960 | eleTestcase = ET.SubElement( |
| 2961 | eleTestsuite, 'testcase', classname="%s:%s" % |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2962 | (instance.platform.name, instance.testcase.name), name="%s" % |
| 2963 | (instance.testcase.name), time="%f" %handler_time) |
| 2964 | |
| 2965 | if instance.status == "failed": |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 2966 | failure = ET.SubElement( |
| 2967 | eleTestcase, |
| 2968 | 'failure', |
| 2969 | type="failure", |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2970 | message=instance.reason) |
| 2971 | p = ("%s/%s/%s" % (options.outdir, instance.platform.name, instance.testcase.name)) |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2972 | bl = os.path.join(p, "build.log") |
Anas Nashif | c96c90a | 2019-02-05 07:38:32 -0500 | [diff] [blame] | 2973 | hl = os.path.join(p, "handler.log") |
| 2974 | log_file = bl |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2975 | if instance.reason != 'Build error': |
Anas Nashif | c96c90a | 2019-02-05 07:38:32 -0500 | [diff] [blame] | 2976 | if os.path.exists(hl): |
| 2977 | log_file = hl |
| 2978 | else: |
| 2979 | log_file = bl |
Anas Nashif | accc8eb | 2017-05-01 16:33:43 -0400 | [diff] [blame] | 2980 | |
Anas Nashif | c96c90a | 2019-02-05 07:38:32 -0500 | [diff] [blame] | 2981 | if os.path.exists(log_file): |
| 2982 | with open(log_file, "rb") as f: |
Anas Nashif | 712d345 | 2017-12-29 22:09:03 -0500 | [diff] [blame] | 2983 | log = f.read().decode("utf-8") |
Anas Nashif | a4c368e | 2018-10-15 09:45:59 -0400 | [diff] [blame] | 2984 | filtered_string = ''.join(filter(lambda x: x in string.printable, log)) |
| 2985 | failure.text = filtered_string |
Anas Nashif | ba4643b | 2018-09-23 09:41:59 -0500 | [diff] [blame] | 2986 | f.close() |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2987 | elif instance.status == "skipped": |
| 2988 | ET.SubElement( eleTestcase, 'skipped', type="skipped", message="Skipped") |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2989 | |
| 2990 | result = ET.tostring(eleTestsuites) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2991 | with open(filename, 'wb') as report: |
| 2992 | report.write(result) |
Anas Nashif | b3311ed | 2017-04-13 14:44:48 -0400 | [diff] [blame] | 2993 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2994 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2995 | def csv_report(self, filename): |
Andrew Boie | 08ce5a5 | 2016-02-22 13:28:10 -0800 | [diff] [blame] | 2996 | with open(filename, "wt") as csvfile: |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2997 | fieldnames = ["test", "arch", "platform", "passed", "status", |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 2998 | "extra_args", "handler", "handler_time", "ram_size", |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 2999 | "rom_size"] |
| 3000 | cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep) |
| 3001 | cw.writeheader() |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3002 | for instance in sorted(self.instances.values()): |
| 3003 | rowdict = {"test": instance.testcase.name, |
| 3004 | "arch": instance.platform.arch, |
| 3005 | "platform": instance.platform.name, |
| 3006 | "extra_args": " ".join(instance.testcase.extra_args), |
| 3007 | "handler": instance.platform.simulation} |
| 3008 | |
| 3009 | if instance.status in ["failed", "timeout"]: |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3010 | rowdict["passed"] = False |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3011 | rowdict["status"] = instance.reason |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3012 | else: |
| 3013 | rowdict["passed"] = True |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3014 | if instance.handler: |
| 3015 | rowdict["handler_time"] = instance.metrics.get("handler_time", 0) |
| 3016 | ram_size = instance.metrics.get("ram_size", 0) |
| 3017 | rom_size = instance.metrics.get("rom_size", 0) |
| 3018 | rowdict["ram_size"] = ram_size |
| 3019 | rowdict["rom_size"] = rom_size |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3020 | cw.writerow(rowdict) |
| 3021 | |
| 3022 | |
Anas Nashif | 19ca783 | 2019-11-18 08:16:21 -0800 | [diff] [blame] | 3023 | def get_testcase(self, identifier): |
| 3024 | results = [] |
| 3025 | for _, tc in self.testcases.items(): |
| 3026 | for case in tc.cases: |
| 3027 | if case == identifier: |
| 3028 | results.append(tc) |
| 3029 | return results |
| 3030 | |
| 3031 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3032 | def parse_arguments(): |
| 3033 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3034 | parser = argparse.ArgumentParser( |
| 3035 | description=__doc__, |
| 3036 | formatter_class=argparse.RawDescriptionHelpFormatter) |
Genaro Saucedo Tejada | 28bba92 | 2016-10-24 18:00:58 -0500 | [diff] [blame] | 3037 | parser.fromfile_prefix_chars = "+" |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3038 | |
Marc Herbert | 932a33a | 2019-03-12 11:37:53 -0700 | [diff] [blame] | 3039 | case_select = parser.add_argument_group("Test case selection", |
| 3040 | """ |
| 3041 | Artificially long but functional example: |
| 3042 | $ ./scripts/sanitycheck -v \\ |
Marc Herbert | e5cedca | 2019-04-08 14:02:34 -0700 | [diff] [blame] | 3043 | --testcase-root tests/ztest/base \\ |
| 3044 | --testcase-root tests/kernel \\ |
Marc Herbert | 932a33a | 2019-03-12 11:37:53 -0700 | [diff] [blame] | 3045 | --test tests/ztest/base/testing.ztest.verbose_0 \\ |
| 3046 | --test tests/kernel/fifo/fifo_api/kernel.fifo.poll |
| 3047 | |
| 3048 | "kernel.fifo.poll" is one of the test section names in |
| 3049 | __/fifo_api/testcase.yaml |
| 3050 | """) |
Marc Herbert | edf1759 | 2019-03-08 12:39:11 -0800 | [diff] [blame] | 3051 | |
Anas Nashif | 07d54c0 | 2018-07-21 19:29:08 -0500 | [diff] [blame] | 3052 | parser.add_argument("--force-toolchain", action="store_true", |
| 3053 | help="Do not filter based on toolchain, use the set " |
| 3054 | " toolchain unconditionally") |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3055 | parser.add_argument( |
| 3056 | "-p", "--platform", action="append", |
| 3057 | help="Platform filter for testing. This option may be used multiple " |
| 3058 | "times. Testcases will only be built/run on the platforms " |
| 3059 | "specified. If this option is not used, then platforms marked " |
| 3060 | "as default in the platform metadata file will be chosen " |
| 3061 | "to build and test. ") |
| 3062 | parser.add_argument( |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3063 | "-a", "--arch", action="append", |
| 3064 | help="Arch filter for testing. Takes precedence over --platform. " |
| 3065 | "If unspecified, test all arches. Multiple invocations " |
| 3066 | "are treated as a logical 'or' relationship") |
| 3067 | parser.add_argument( |
| 3068 | "-t", "--tag", action="append", |
| 3069 | help="Specify tags to restrict which tests to run by tag value. " |
| 3070 | "Default is to not do any tag filtering. Multiple invocations " |
| 3071 | "are treated as a logical 'or' relationship") |
Anas Nashif | dfa86e2 | 2016-10-24 17:08:56 -0400 | [diff] [blame] | 3072 | parser.add_argument("-e", "--exclude-tag", action="append", |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3073 | help="Specify tags of tests that should not run. " |
| 3074 | "Default is to run all tests with all tags.") |
Marc Herbert | edf1759 | 2019-03-08 12:39:11 -0800 | [diff] [blame] | 3075 | case_select.add_argument( |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3076 | "-f", |
| 3077 | "--only-failed", |
| 3078 | action="store_true", |
| 3079 | help="Run only those tests that failed the previous sanity check " |
| 3080 | "invocation.") |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3081 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3082 | parser.add_argument( |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3083 | "--retry-failed", type=int, default=0, |
| 3084 | help="Retry failing tests again, up to the number of times specified.") |
Anas Nashif | 1ea5d7b | 2018-07-12 09:25:22 -0500 | [diff] [blame] | 3085 | |
Marc Herbert | 0c465bb | 2019-03-11 17:28:36 -0700 | [diff] [blame] | 3086 | test_xor_subtest = case_select.add_mutually_exclusive_group() |
| 3087 | |
| 3088 | test_xor_subtest.add_argument( |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3089 | "-s", "--test", action="append", |
| 3090 | help="Run only the specified test cases. These are named by " |
Marc Herbert | e5cedca | 2019-04-08 14:02:34 -0700 | [diff] [blame] | 3091 | "<path/relative/to/Zephyr/base/section.name.in.testcase.yaml>") |
Anas Nashif | 1ea5d7b | 2018-07-12 09:25:22 -0500 | [diff] [blame] | 3092 | |
Marc Herbert | 0c465bb | 2019-03-11 17:28:36 -0700 | [diff] [blame] | 3093 | test_xor_subtest.add_argument( |
Anas Nashif | 1ea5d7b | 2018-07-12 09:25:22 -0500 | [diff] [blame] | 3094 | "--sub-test", action="append", |
Marc Herbert | 932a33a | 2019-03-12 11:37:53 -0700 | [diff] [blame] | 3095 | help="""Recursively find sub-test functions and run the entire |
| 3096 | test section where they were found, including all sibling test |
| 3097 | functions. Sub-tests are named by: |
| 3098 | section.name.in.testcase.yaml.function_name_without_test_prefix |
| 3099 | Example: kernel.fifo.poll.fifo_loop |
| 3100 | """) |
Anas Nashif | 1ea5d7b | 2018-07-12 09:25:22 -0500 | [diff] [blame] | 3101 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3102 | parser.add_argument( |
| 3103 | "-l", "--all", action="store_true", |
| 3104 | help="Build/test on all platforms. Any --platform arguments " |
| 3105 | "ignored.") |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3106 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3107 | parser.add_argument( |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3108 | "-o", "--report-dir", |
| 3109 | help="""Output reports containing results of the test run into the |
| 3110 | specified directory. |
| 3111 | The output will be both in CSV and JUNIT format |
| 3112 | (sanitycheck.csv and sanitycheck.xml). |
| 3113 | """) |
| 3114 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3115 | parser.add_argument( |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3116 | "--report-name", |
| 3117 | help="""Create a report with a custom name. |
| 3118 | """) |
| 3119 | |
| 3120 | parser.add_argument("--detailed-report", |
| 3121 | action="store", |
| 3122 | metavar="FILENAME", |
| 3123 | help="""Generate a junit report with detailed testcase results. |
| 3124 | Unlike the CSV file produced by --testcase-report, this XML |
| 3125 | report includes only tests which have run and none which were |
| 3126 | merely built. If an image with multiple tests crashes early then |
| 3127 | later tests are not accounted for either.""") |
| 3128 | |
| 3129 | parser.add_argument("--report-excluded", |
| 3130 | action="store_true", |
| 3131 | help="""List all tests that are never run based on current scope and |
| 3132 | coverage. If you are looking for accurate results, run this with |
| 3133 | --all, but this will take a while...""") |
| 3134 | |
Daniel Leung | 7f85010 | 2016-04-08 11:07:32 -0700 | [diff] [blame] | 3135 | parser.add_argument("--compare-report", |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3136 | help="Use this report file for size comparison") |
Daniel Leung | 7f85010 | 2016-04-08 11:07:32 -0700 | [diff] [blame] | 3137 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3138 | parser.add_argument( |
| 3139 | "-B", "--subset", |
| 3140 | help="Only run a subset of the tests, 1/4 for running the first 25%%, " |
| 3141 | "3/5 means run the 3rd fifth of the total. " |
| 3142 | "This option is useful when running a large number of tests on " |
| 3143 | "different hosts to speed up execution time.") |
Anas Nashif | a8a1388 | 2017-12-30 13:01:06 -0500 | [diff] [blame] | 3144 | |
| 3145 | parser.add_argument( |
| 3146 | "-N", "--ninja", action="store_true", |
Anas Nashif | 25f6ab6 | 2018-03-06 07:15:11 -0600 | [diff] [blame] | 3147 | help="Use the Ninja generator with CMake") |
Anas Nashif | a8a1388 | 2017-12-30 13:01:06 -0500 | [diff] [blame] | 3148 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3149 | parser.add_argument( |
| 3150 | "-y", "--dry-run", action="store_true", |
Anas Nashif | 12d8cce | 2019-11-20 03:47:27 -0800 | [diff] [blame] | 3151 | help="""Create the filtered list of test cases, but don't actually |
| 3152 | run them. Useful if you're just interested in the discard report |
| 3153 | generated for every run and saved in the specified output |
| 3154 | directory (sanitycheck_discard.csv). |
| 3155 | """) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3156 | |
Anas Nashif | 75547e2 | 2018-02-24 08:32:14 -0600 | [diff] [blame] | 3157 | parser.add_argument("--list-tags", action="store_true", |
| 3158 | help="list all tags in selected tests") |
| 3159 | |
Marc Herbert | edf1759 | 2019-03-08 12:39:11 -0800 | [diff] [blame] | 3160 | case_select.add_argument("--list-tests", action="store_true", |
Marc Herbert | 932a33a | 2019-03-12 11:37:53 -0700 | [diff] [blame] | 3161 | help="""List of all sub-test functions recursively found in |
| 3162 | all --testcase-root arguments. Note different sub-tests can share |
| 3163 | the same section name and come from different directories. |
| 3164 | The output is flattened and reports --sub-test names only, |
| 3165 | not their directories. For instance net.socket.getaddrinfo_ok |
| 3166 | and net.socket.fd_set belong to different directories. |
| 3167 | """) |
Anas Nashif | c0149cc | 2018-04-14 23:12:58 -0500 | [diff] [blame] | 3168 | |
Anas Nashif | 19ca783 | 2019-11-18 08:16:21 -0800 | [diff] [blame] | 3169 | case_select.add_argument("--list-test-duplicates", action="store_true", |
| 3170 | help="""List tests with duplicate identifiers. |
| 3171 | """) |
| 3172 | |
Anas Nashif | 5d6e7eb | 2018-06-01 09:51:08 -0500 | [diff] [blame] | 3173 | parser.add_argument("--export-tests", action="store", |
| 3174 | metavar="FILENAME", |
| 3175 | help="Export tests case meta-data to a file in CSV format.") |
| 3176 | |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 3177 | |
Anas Nashif | 654ec598 | 2019-04-11 08:38:21 -0400 | [diff] [blame] | 3178 | parser.add_argument("--timestamps", |
| 3179 | action="store_true", |
| 3180 | help="Print all messages with time stamps") |
| 3181 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3182 | parser.add_argument( |
| 3183 | "-r", "--release", action="store_true", |
| 3184 | help="Update the benchmark database with the results of this test " |
| 3185 | "run. Intended to be run by CI when tagging an official " |
| 3186 | "release. This database is used as a basis for comparison " |
| 3187 | "when looking for deltas in metrics such as footprint") |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3188 | parser.add_argument("-w", "--warnings-as-errors", action="store_true", |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3189 | help="Treat warning conditions as errors") |
| 3190 | parser.add_argument( |
| 3191 | "-v", |
| 3192 | "--verbose", |
| 3193 | action="count", |
| 3194 | default=0, |
| 3195 | help="Emit debugging information, call multiple times to increase " |
| 3196 | "verbosity") |
| 3197 | parser.add_argument( |
| 3198 | "-i", "--inline-logs", action="store_true", |
| 3199 | help="Upon test failure, print relevant log data to stdout " |
| 3200 | "instead of just a path to it") |
Inaky Perez-Gonzalez | 9a36cb6 | 2016-11-29 10:43:40 -0800 | [diff] [blame] | 3201 | parser.add_argument("--log-file", metavar="FILENAME", action="store", |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3202 | help="log also to file") |
| 3203 | parser.add_argument( |
| 3204 | "-m", "--last-metrics", action="store_true", |
| 3205 | help="Instead of comparing metrics from the last --release, " |
| 3206 | "compare with the results of the previous sanity check " |
| 3207 | "invocation") |
| 3208 | parser.add_argument( |
| 3209 | "-u", |
| 3210 | "--no-update", |
| 3211 | action="store_true", |
| 3212 | help="do not update the results of the last run of the sanity " |
| 3213 | "checks") |
Marc Herbert | edf1759 | 2019-03-08 12:39:11 -0800 | [diff] [blame] | 3214 | case_select.add_argument( |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3215 | "-F", |
| 3216 | "--load-tests", |
| 3217 | metavar="FILENAME", |
| 3218 | action="store", |
Marc Herbert | 932a33a | 2019-03-12 11:37:53 -0700 | [diff] [blame] | 3219 | help="Load list of tests and platforms to be run from file.") |
Anas Nashif | bd166f4 | 2017-09-02 12:32:08 -0400 | [diff] [blame] | 3220 | |
Marc Herbert | edf1759 | 2019-03-08 12:39:11 -0800 | [diff] [blame] | 3221 | case_select.add_argument( |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3222 | "-E", |
| 3223 | "--save-tests", |
| 3224 | metavar="FILENAME", |
| 3225 | action="store", |
Marc Herbert | 682961a | 2019-03-20 16:48:49 -0700 | [diff] [blame] | 3226 | help="Append list of tests and platforms to be run to file.") |
Anas Nashif | bd166f4 | 2017-09-02 12:32:08 -0400 | [diff] [blame] | 3227 | |
Andy Doan | cbecadd | 2019-02-08 10:19:10 -0600 | [diff] [blame] | 3228 | test_or_build = parser.add_mutually_exclusive_group() |
| 3229 | test_or_build.add_argument( |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3230 | "-b", "--build-only", action="store_true", |
| 3231 | help="Only build the code, do not execute any of it in QEMU") |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3232 | |
Andy Doan | cbecadd | 2019-02-08 10:19:10 -0600 | [diff] [blame] | 3233 | test_or_build.add_argument( |
| 3234 | "--test-only", action="store_true", |
| 3235 | help="""Only run device tests with current artifacts, do not build |
| 3236 | the code""") |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3237 | parser.add_argument( |
Kumar Gala | 84cf9dc | 2019-06-15 09:36:06 -0500 | [diff] [blame] | 3238 | "--cmake-only", action="store_true", |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3239 | help="Only run cmake, do not build or run.") |
Kumar Gala | 84cf9dc | 2019-06-15 09:36:06 -0500 | [diff] [blame] | 3240 | |
| 3241 | parser.add_argument( |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3242 | "-j", "--jobs", type=int, |
Marc Herbert | 9e57338 | 2019-07-03 12:49:42 -0700 | [diff] [blame] | 3243 | help="Number of jobs for building, defaults to number of CPU threads, " |
| 3244 | "overcommited by factor 2 when --build-only") |
Anas Nashif | 73440ea | 2018-02-19 10:57:03 -0600 | [diff] [blame] | 3245 | |
| 3246 | parser.add_argument( |
Anas Nashif | 424a3db | 2018-02-20 08:37:24 -0600 | [diff] [blame] | 3247 | "--show-footprint", action="store_true", |
| 3248 | help="Show footprint statistics and deltas since last release." |
| 3249 | ) |
| 3250 | parser.add_argument( |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3251 | "-H", "--footprint-threshold", type=float, default=5, |
| 3252 | help="When checking test case footprint sizes, warn the user if " |
| 3253 | "the new app size is greater then the specified percentage " |
| 3254 | "from the last release. Default is 5. 0 to warn on any " |
| 3255 | "increase on app size") |
| 3256 | parser.add_argument( |
| 3257 | "-D", "--all-deltas", action="store_true", |
| 3258 | help="Show all footprint deltas, positive or negative. Implies " |
| 3259 | "--footprint-threshold=0") |
Håkon Øye Amundsen | de223cc | 2018-04-25 06:24:25 +0000 | [diff] [blame] | 3260 | parser.add_argument( |
| 3261 | "-O", "--outdir", |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3262 | default=os.path.join(os.getcwd(),"sanity-out"), |
Håkon Øye Amundsen | de223cc | 2018-04-25 06:24:25 +0000 | [diff] [blame] | 3263 | help="Output directory for logs and binaries. " |
Anas Nashif | f114a13 | 2018-11-20 11:51:34 -0500 | [diff] [blame] | 3264 | "Default is 'sanity-out' in the current directory. " |
Håkon Øye Amundsen | de223cc | 2018-04-25 06:24:25 +0000 | [diff] [blame] | 3265 | "This directory will be deleted unless '--no-clean' is set.") |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3266 | parser.add_argument( |
| 3267 | "-n", "--no-clean", action="store_true", |
| 3268 | help="Do not delete the outdir before building. Will result in " |
| 3269 | "faster compilation since builds will be incremental") |
Marc Herbert | edf1759 | 2019-03-08 12:39:11 -0800 | [diff] [blame] | 3270 | case_select.add_argument( |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3271 | "-T", "--testcase-root", action="append", default=[], |
| 3272 | help="Base directory to recursively search for test cases. All " |
| 3273 | "testcase.yaml files under here will be processed. May be " |
Marc Herbert | 932a33a | 2019-03-12 11:37:53 -0700 | [diff] [blame] | 3274 | "called multiple times. Defaults to the 'samples/' and " |
| 3275 | "'tests/' directories at the base of the Zephyr tree.") |
Anas Nashif | 7dd19ea | 2018-11-14 08:46:49 -0500 | [diff] [blame] | 3276 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3277 | board_root_list = ["%s/boards" % ZEPHYR_BASE, |
| 3278 | "%s/scripts/sanity_chk/boards" % ZEPHYR_BASE] |
Anas Nashif | 7dd19ea | 2018-11-14 08:46:49 -0500 | [diff] [blame] | 3279 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3280 | parser.add_argument( |
Thomas Stilwell | 5310510 | 2019-10-23 14:25:09 +0200 | [diff] [blame] | 3281 | "-A", "--board-root", action="append", default=board_root_list, |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3282 | help="""Directory to search for board configuration files. All .yaml |
| 3283 | files in the directory will be processed. The directory should have the same |
| 3284 | structure in the main Zephyr tree: boards/<arch>/<board_name>/""") |
| 3285 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3286 | parser.add_argument( |
| 3287 | "-z", "--size", action="append", |
| 3288 | help="Don't run sanity checks. Instead, produce a report to " |
| 3289 | "stdout detailing RAM/ROM sizes on the specified filenames. " |
| 3290 | "All other command line arguments ignored.") |
| 3291 | parser.add_argument( |
| 3292 | "-S", "--enable-slow", action="store_true", |
| 3293 | help="Execute time-consuming test cases that have been marked " |
| 3294 | "as 'slow' in testcase.yaml. Normally these are only built.") |
Sebastian Bøe | 03ed095 | 2018-11-13 13:36:19 +0100 | [diff] [blame] | 3295 | parser.add_argument( |
| 3296 | "--disable-unrecognized-section-test", action="store_true", |
| 3297 | default=False, |
| 3298 | help="Skip the 'unrecognized section' test.") |
Andrew Boie | 5512105 | 2016-07-20 11:52:04 -0700 | [diff] [blame] | 3299 | parser.add_argument("-R", "--enable-asserts", action="store_true", |
Kumar Gala | 0d48cbe | 2018-01-26 10:22:20 -0600 | [diff] [blame] | 3300 | default=True, |
Andrew Boie | 29599f6 | 2018-05-24 13:33:09 -0700 | [diff] [blame] | 3301 | help="deprecated, left for compatibility") |
Kumar Gala | 0d48cbe | 2018-01-26 10:22:20 -0600 | [diff] [blame] | 3302 | parser.add_argument("--disable-asserts", action="store_false", |
| 3303 | dest="enable_asserts", |
Andrew Boie | 29599f6 | 2018-05-24 13:33:09 -0700 | [diff] [blame] | 3304 | help="deprecated, left for compatibility") |
Anas Nashif | e3febe9 | 2016-11-30 14:25:44 -0500 | [diff] [blame] | 3305 | parser.add_argument("-Q", "--error-on-deprecations", action="store_false", |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3306 | help="Error on deprecation warnings.") |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3307 | parser.add_argument("--enable-size-report", action="store_true", |
| 3308 | help="Enable expensive computation of RAM/ROM segment sizes.") |
Sebastian Bøe | c218261 | 2017-11-09 12:25:02 +0100 | [diff] [blame] | 3309 | |
| 3310 | parser.add_argument( |
| 3311 | "-x", "--extra-args", action="append", default=[], |
Alberto Escolar Piedras | 25e0636 | 2018-02-10 17:40:33 +0100 | [diff] [blame] | 3312 | help="""Extra CMake cache entries to define when building test cases. |
| 3313 | May be called multiple times. The key-value entries will be |
Sebastian Bøe | c218261 | 2017-11-09 12:25:02 +0100 | [diff] [blame] | 3314 | prefixed with -D before being passed to CMake. |
| 3315 | |
| 3316 | E.g |
| 3317 | "sanitycheck -x=USE_CCACHE=0" |
| 3318 | will translate to |
| 3319 | "cmake -DUSE_CCACHE=0" |
| 3320 | |
| 3321 | which will ultimately disable ccache. |
| 3322 | """ |
| 3323 | ) |
Michael Scott | 421ce46 | 2019-06-18 09:37:46 -0700 | [diff] [blame] | 3324 | |
Andy Doan | 79c4884 | 2019-02-08 10:09:04 -0600 | [diff] [blame] | 3325 | parser.add_argument( |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3326 | "--device-testing", action="store_true", |
| 3327 | help="Test on device directly. Specify the serial device to " |
| 3328 | "use with the --device-serial option.") |
| 3329 | |
| 3330 | parser.add_argument( |
| 3331 | "-X", "--fixture", action="append", default=[], |
| 3332 | help="Specify a fixture that a board might support") |
| 3333 | parser.add_argument( |
| 3334 | "--device-serial", |
| 3335 | help="Serial device for accessing the board (e.g., /dev/ttyACM0)") |
| 3336 | |
| 3337 | parser.add_argument("--generate-hardware-map", |
| 3338 | help="""Probe serial devices connected to this platform |
| 3339 | and create a hardware map file to be used with |
| 3340 | --device-testing |
| 3341 | """) |
| 3342 | |
| 3343 | parser.add_argument("--hardware-map", |
| 3344 | help="""Load hardware map from a file. This will be used |
| 3345 | for testing on hardware that is listed in the file. |
| 3346 | """) |
| 3347 | |
| 3348 | parser.add_argument( |
Andy Doan | 79c4884 | 2019-02-08 10:09:04 -0600 | [diff] [blame] | 3349 | "--west-flash", nargs='?', const=[], |
| 3350 | help="""Uses west instead of ninja or make to flash when running with |
Jorgen Kvalvaag | d50fb31 | 2019-09-02 15:25:20 +0200 | [diff] [blame] | 3351 | --device-testing. Supports comma-separated argument list. |
Andy Doan | 79c4884 | 2019-02-08 10:09:04 -0600 | [diff] [blame] | 3352 | |
Michael Scott | 4ca5439 | 2019-07-09 14:21:30 -0700 | [diff] [blame] | 3353 | E.g "sanitycheck --device-testing --device-serial /dev/ttyACM0 |
Jorgen Kvalvaag | d50fb31 | 2019-09-02 15:25:20 +0200 | [diff] [blame] | 3354 | --west-flash="--board-id=foobar,--erase" |
| 3355 | will translate to "west flash -- --board-id=foobar --erase" |
Michael Scott | 4ca5439 | 2019-07-09 14:21:30 -0700 | [diff] [blame] | 3356 | |
| 3357 | NOTE: device-testing must be enabled to use this option. |
Andy Doan | 79c4884 | 2019-02-08 10:09:04 -0600 | [diff] [blame] | 3358 | """ |
| 3359 | ) |
Michael Scott | 421ce46 | 2019-06-18 09:37:46 -0700 | [diff] [blame] | 3360 | parser.add_argument( |
| 3361 | "--west-runner", |
| 3362 | help="""Uses the specified west runner instead of default when running |
| 3363 | with --west-flash. |
| 3364 | |
| 3365 | E.g "sanitycheck --device-testing --device-serial /dev/ttyACM0 |
| 3366 | --west-flash --west-runner=pyocd" |
| 3367 | will translate to "west flash --runner pyocd" |
| 3368 | |
| 3369 | NOTE: west-flash must be enabled to use this option. |
| 3370 | """ |
| 3371 | ) |
Jan Van Winkel | 21212f3 | 2019-09-12 00:03:35 +0200 | [diff] [blame] | 3372 | |
| 3373 | valgrind_asan_group = parser.add_mutually_exclusive_group() |
| 3374 | |
| 3375 | valgrind_asan_group.add_argument( |
Anas Nashif | c1ea452 | 2019-10-11 07:32:45 -0700 | [diff] [blame] | 3376 | "--enable-valgrind", action="store_true", |
| 3377 | help="""Run binary through valgrind and check for several memory access |
Jan Van Winkel | 21212f3 | 2019-09-12 00:03:35 +0200 | [diff] [blame] | 3378 | errors. Valgrind needs to be installed on the host. This option only |
Anas Nashif | c1ea452 | 2019-10-11 07:32:45 -0700 | [diff] [blame] | 3379 | works with host binaries such as those generated for the native_posix |
Jan Van Winkel | 21212f3 | 2019-09-12 00:03:35 +0200 | [diff] [blame] | 3380 | configuration and is mutual exclusive with --enable-asan. |
| 3381 | """) |
| 3382 | |
| 3383 | valgrind_asan_group.add_argument( |
| 3384 | "--enable-asan", action="store_true", |
| 3385 | help="""Enable address sanitizer to check for several memory access |
| 3386 | errors. Libasan needs to be installed on the host. This option only |
| 3387 | works with host binaries such as those generated for the native_posix |
| 3388 | configuration and is mutual exclusive with --enable-valgrind. |
| 3389 | """) |
| 3390 | |
| 3391 | parser.add_argument( |
| 3392 | "--enable-lsan", action="store_true", |
| 3393 | help="""Enable leak sanitizer to check for heap memory leaks. |
| 3394 | Libasan needs to be installed on the host. This option only |
| 3395 | works with host binaries such as those generated for the native_posix |
| 3396 | configuration and when --enable-asan is given. |
Anas Nashif | c1ea452 | 2019-10-11 07:32:45 -0700 | [diff] [blame] | 3397 | """) |
Andrew Boie | 8047a6f | 2019-07-02 15:43:29 -0700 | [diff] [blame] | 3398 | |
Alberto Escolar Piedras | c026c2e | 2018-06-21 09:30:20 +0200 | [diff] [blame] | 3399 | parser.add_argument("--enable-coverage", action="store_true", |
Anas Nashif | 8d72bb9 | 2018-11-07 23:05:42 -0500 | [diff] [blame] | 3400 | help="Enable code coverage using gcov.") |
Alberto Escolar Piedras | c026c2e | 2018-06-21 09:30:20 +0200 | [diff] [blame] | 3401 | |
Jaakko Hannikainen | 54c90bc | 2016-08-31 14:17:03 +0300 | [diff] [blame] | 3402 | parser.add_argument("-C", "--coverage", action="store_true", |
Andrew Boie | 8047a6f | 2019-07-02 15:43:29 -0700 | [diff] [blame] | 3403 | help="Generate coverage reports. Implies " |
| 3404 | "--enable_coverage and --enable-slow") |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3405 | |
Andrew Boie | 8047a6f | 2019-07-02 15:43:29 -0700 | [diff] [blame] | 3406 | parser.add_argument("--coverage-platform", action="append", default=[], |
Anas Nashif | dbd7649 | 2018-11-23 20:24:19 -0500 | [diff] [blame] | 3407 | help="Plarforms to run coverage reports on. " |
Andrew Boie | 8047a6f | 2019-07-02 15:43:29 -0700 | [diff] [blame] | 3408 | "This option may be used multiple times. " |
| 3409 | "Default to what was selected with --platform.") |
Anas Nashif | dbd7649 | 2018-11-23 20:24:19 -0500 | [diff] [blame] | 3410 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3411 | parser.add_argument("--gcov-tool", default=None, |
| 3412 | help="Path to the gcov tool to use for code coverage " |
| 3413 | "reports") |
| 3414 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3415 | return parser.parse_args() |
| 3416 | |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3417 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3418 | def log_info(filename): |
Mazen NEIFER | 8f1dd3a | 2017-02-04 14:32:04 +0100 | [diff] [blame] | 3419 | filename = os.path.relpath(os.path.realpath(filename)) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3420 | if options.inline_logs: |
Andrew Boie | 08ce5a5 | 2016-02-22 13:28:10 -0800 | [diff] [blame] | 3421 | info("{:-^100}".format(filename)) |
Andrew Boie | d01535c | 2017-01-10 13:15:02 -0800 | [diff] [blame] | 3422 | |
| 3423 | try: |
| 3424 | with open(filename) as fp: |
| 3425 | data = fp.read() |
| 3426 | except Exception as e: |
| 3427 | data = "Unable to read log data (%s)\n" % (str(e)) |
| 3428 | |
| 3429 | sys.stdout.write(data) |
| 3430 | if log_file: |
| 3431 | log_file.write(data) |
Andrew Boie | 08ce5a5 | 2016-02-22 13:28:10 -0800 | [diff] [blame] | 3432 | info("{:-^100}".format(filename)) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3433 | else: |
Anas Nashif | c1ea452 | 2019-10-11 07:32:45 -0700 | [diff] [blame] | 3434 | info("\n\tsee: " + COLOR_YELLOW + filename + COLOR_NORMAL) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3435 | |
Ulf Magnusson | e73d286 | 2019-10-29 11:54:02 +0100 | [diff] [blame] | 3436 | |
| 3437 | def log_info_file(instance): |
| 3438 | build_dir = instance.build_dir |
| 3439 | h_log = "{}/handler.log".format(build_dir) |
| 3440 | b_log = "{}/build.log".format(build_dir) |
| 3441 | v_log = "{}/valgrind.log".format(build_dir) |
| 3442 | |
| 3443 | if os.path.exists(v_log) and "Valgrind" in instance.reason: |
| 3444 | log_info("{}".format(v_log)) |
| 3445 | elif os.path.exists(h_log): |
| 3446 | log_info("{}".format(h_log)) |
| 3447 | else: |
| 3448 | log_info("{}".format(b_log)) |
| 3449 | |
| 3450 | |
Andrew Boie | bbd670c | 2015-08-17 13:16:11 -0700 | [diff] [blame] | 3451 | def size_report(sc): |
| 3452 | info(sc.filename) |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 3453 | info("SECTION NAME VMA LMA SIZE HEX SZ TYPE") |
Andrew Boie | 9882dcd | 2015-10-07 14:25:51 -0700 | [diff] [blame] | 3454 | for i in range(len(sc.sections)): |
| 3455 | v = sc.sections[i] |
| 3456 | |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 3457 | info("%-17s 0x%08x 0x%08x %8d 0x%05x %-7s" % |
| 3458 | (v["name"], v["virt_addr"], v["load_addr"], v["size"], v["size"], |
| 3459 | v["type"])) |
Andrew Boie | 9882dcd | 2015-10-07 14:25:51 -0700 | [diff] [blame] | 3460 | |
Andrew Boie | 73b4ee6 | 2015-10-07 11:33:22 -0700 | [diff] [blame] | 3461 | info("Totals: %d bytes (ROM), %d bytes (RAM)" % |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3462 | (sc.rom_size, sc.ram_size)) |
Andrew Boie | bbd670c | 2015-08-17 13:16:11 -0700 | [diff] [blame] | 3463 | info("") |
| 3464 | |
Anas Nashif | f29087e | 2019-01-25 09:37:38 -0500 | [diff] [blame] | 3465 | def retrieve_gcov_data(intput_file): |
Anas Nashif | db9592a | 2018-10-08 10:19:41 -0400 | [diff] [blame] | 3466 | if VERBOSE: |
| 3467 | print("Working on %s" %intput_file) |
| 3468 | extracted_coverage_info = {} |
| 3469 | capture_data = False |
Anas Nashif | f29087e | 2019-01-25 09:37:38 -0500 | [diff] [blame] | 3470 | capture_complete = False |
Anas Nashif | db9592a | 2018-10-08 10:19:41 -0400 | [diff] [blame] | 3471 | with open(intput_file, 'r') as fp: |
| 3472 | for line in fp.readlines(): |
| 3473 | if re.search("GCOV_COVERAGE_DUMP_START", line): |
| 3474 | capture_data = True |
| 3475 | continue |
| 3476 | if re.search("GCOV_COVERAGE_DUMP_END", line): |
Anas Nashif | f29087e | 2019-01-25 09:37:38 -0500 | [diff] [blame] | 3477 | capture_complete = True |
Anas Nashif | db9592a | 2018-10-08 10:19:41 -0400 | [diff] [blame] | 3478 | break |
| 3479 | # Loop until the coverage data is found. |
| 3480 | if not capture_data: |
| 3481 | continue |
| 3482 | if line.startswith("*"): |
| 3483 | sp = line.split("<") |
| 3484 | if len(sp) > 1: |
| 3485 | # Remove the leading delimiter "*" |
| 3486 | file_name = sp[0][1:] |
| 3487 | # Remove the trailing new line char |
| 3488 | hex_dump = sp[1][:-1] |
| 3489 | else: |
| 3490 | continue |
| 3491 | else: |
| 3492 | continue |
| 3493 | extracted_coverage_info.update({file_name:hex_dump}) |
Anas Nashif | f29087e | 2019-01-25 09:37:38 -0500 | [diff] [blame] | 3494 | if not capture_data: |
| 3495 | capture_complete = True |
| 3496 | return {'complete': capture_complete, 'data': extracted_coverage_info} |
Anas Nashif | db9592a | 2018-10-08 10:19:41 -0400 | [diff] [blame] | 3497 | |
| 3498 | def create_gcda_files(extracted_coverage_info): |
| 3499 | if VERBOSE: |
| 3500 | print("Generating gcda files") |
| 3501 | for filename, hexdump_val in extracted_coverage_info.items(): |
| 3502 | # if kobject_hash is given for coverage gcovr fails |
| 3503 | # hence skipping it problem only in gcovr v4.1 |
| 3504 | if "kobject_hash" in filename: |
| 3505 | filename = (filename[:-4]) +"gcno" |
| 3506 | try: |
| 3507 | os.remove(filename) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3508 | except Exception: |
Anas Nashif | db9592a | 2018-10-08 10:19:41 -0400 | [diff] [blame] | 3509 | pass |
| 3510 | continue |
| 3511 | |
| 3512 | with open(filename, 'wb') as fp: |
| 3513 | fp.write(bytes.fromhex(hexdump_val)) |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3514 | |
Jaakko Hannikainen | 54c90bc | 2016-08-31 14:17:03 +0300 | [diff] [blame] | 3515 | def generate_coverage(outdir, ignores): |
Anas Nashif | db9592a | 2018-10-08 10:19:41 -0400 | [diff] [blame] | 3516 | |
| 3517 | for filename in glob.glob("%s/**/handler.log" %outdir, recursive=True): |
Anas Nashif | f29087e | 2019-01-25 09:37:38 -0500 | [diff] [blame] | 3518 | gcov_data = retrieve_gcov_data(filename) |
| 3519 | capture_complete = gcov_data['complete'] |
| 3520 | extracted_coverage_info = gcov_data['data'] |
| 3521 | if capture_complete: |
| 3522 | create_gcda_files(extracted_coverage_info) |
| 3523 | verbose("Gcov data captured: {}".format(filename)) |
| 3524 | else: |
| 3525 | error("Gcov data capture incomplete: {}".format(filename)) |
Anas Nashif | db9592a | 2018-10-08 10:19:41 -0400 | [diff] [blame] | 3526 | |
Anas Nashif | 47cf4bf | 2019-01-24 21:50:59 -0500 | [diff] [blame] | 3527 | gcov_tool = options.gcov_tool |
| 3528 | |
Jaakko Hannikainen | 54c90bc | 2016-08-31 14:17:03 +0300 | [diff] [blame] | 3529 | with open(os.path.join(outdir, "coverage.log"), "a") as coveragelog: |
| 3530 | coveragefile = os.path.join(outdir, "coverage.info") |
| 3531 | ztestfile = os.path.join(outdir, "ztest.info") |
Anas Nashif | 47cf4bf | 2019-01-24 21:50:59 -0500 | [diff] [blame] | 3532 | subprocess.call(["lcov", "--gcov-tool", gcov_tool, |
| 3533 | "--capture", "--directory", outdir, |
| 3534 | "--rc", "lcov_branch_coverage=1", |
| 3535 | "--output-file", coveragefile], stdout=coveragelog) |
Jaakko Hannikainen | 54c90bc | 2016-08-31 14:17:03 +0300 | [diff] [blame] | 3536 | # We want to remove tests/* and tests/ztest/test/* but save tests/ztest |
Anas Nashif | 47cf4bf | 2019-01-24 21:50:59 -0500 | [diff] [blame] | 3537 | subprocess.call(["lcov", "--gcov-tool", gcov_tool, "--extract", coveragefile, |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3538 | os.path.join(ZEPHYR_BASE, "tests", "ztest", "*"), |
Alberto Escolar Piedras | bc101c5 | 2018-02-11 09:33:55 +0100 | [diff] [blame] | 3539 | "--output-file", ztestfile, |
| 3540 | "--rc", "lcov_branch_coverage=1"], stdout=coveragelog) |
| 3541 | |
Anas Nashif | 3cbffef | 2018-11-07 23:50:54 -0500 | [diff] [blame] | 3542 | if os.path.exists(ztestfile) and os.path.getsize(ztestfile) > 0: |
Anas Nashif | 47cf4bf | 2019-01-24 21:50:59 -0500 | [diff] [blame] | 3543 | subprocess.call(["lcov", "--gcov-tool", gcov_tool, "--remove", ztestfile, |
Alberto Escolar Piedras | bc101c5 | 2018-02-11 09:33:55 +0100 | [diff] [blame] | 3544 | os.path.join(ZEPHYR_BASE, "tests/ztest/test/*"), |
| 3545 | "--output-file", ztestfile, |
| 3546 | "--rc", "lcov_branch_coverage=1"], |
| 3547 | stdout=coveragelog) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3548 | files = [coveragefile, ztestfile] |
Alberto Escolar Piedras | bc101c5 | 2018-02-11 09:33:55 +0100 | [diff] [blame] | 3549 | else: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3550 | files = [coveragefile] |
Alberto Escolar Piedras | bc101c5 | 2018-02-11 09:33:55 +0100 | [diff] [blame] | 3551 | |
Jaakko Hannikainen | 54c90bc | 2016-08-31 14:17:03 +0300 | [diff] [blame] | 3552 | for i in ignores: |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3553 | subprocess.call( |
Anas Nashif | 47cf4bf | 2019-01-24 21:50:59 -0500 | [diff] [blame] | 3554 | ["lcov", "--gcov-tool", gcov_tool, "--remove", |
| 3555 | coveragefile, i, "--output-file", |
| 3556 | coveragefile, "--rc", "lcov_branch_coverage=1"], |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3557 | stdout=coveragelog) |
Alberto Escolar Piedras | bc101c5 | 2018-02-11 09:33:55 +0100 | [diff] [blame] | 3558 | |
Alberto Escolar Piedras | 834b86f | 2019-02-03 15:48:07 +0100 | [diff] [blame] | 3559 | #The --ignore-errors source option is added to avoid it exiting due to |
| 3560 | #samples/application_development/external_lib/ |
Alberto Escolar Piedras | bc101c5 | 2018-02-11 09:33:55 +0100 | [diff] [blame] | 3561 | ret = subprocess.call(["genhtml", "--legend", "--branch-coverage", |
Alberto Escolar Piedras | 834b86f | 2019-02-03 15:48:07 +0100 | [diff] [blame] | 3562 | "--ignore-errors", "source", |
Alberto Escolar Piedras | bc101c5 | 2018-02-11 09:33:55 +0100 | [diff] [blame] | 3563 | "-output-directory", |
| 3564 | os.path.join(outdir, "coverage")] + files, |
| 3565 | stdout=coveragelog) |
| 3566 | if ret==0: |
| 3567 | info("HTML report generated: %s"% |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3568 | os.path.join(outdir, "coverage","index.html")) |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3569 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3570 | def get_generator(): |
| 3571 | if options.ninja: |
| 3572 | generator_cmd = "ninja" |
| 3573 | generator = "Ninja" |
| 3574 | else: |
| 3575 | generator_cmd = "make" |
| 3576 | generator = "Unix Makefiles" |
| 3577 | return generator_cmd, generator |
| 3578 | |
| 3579 | |
| 3580 | def export_tests(filename, tests): |
| 3581 | with open(filename, "wt") as csvfile: |
| 3582 | fieldnames = ['section', 'subsection', 'title', 'reference'] |
| 3583 | cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep) |
| 3584 | for test in tests: |
| 3585 | data = test.split(".") |
Anas Nashif | 19ca783 | 2019-11-18 08:16:21 -0800 | [diff] [blame] | 3586 | if len(data) > 1: |
| 3587 | subsec = " ".join(data[1].split("_")).title() |
| 3588 | rowdict = { |
| 3589 | "section": data[0].capitalize(), |
| 3590 | "subsection": subsec, |
| 3591 | "title": test, |
| 3592 | "reference": test |
| 3593 | } |
| 3594 | cw.writerow(rowdict) |
| 3595 | else: |
| 3596 | info("{} can't be exported".format(test)) |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3597 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3598 | |
| 3599 | def native_and_unit_first(a, b): |
| 3600 | if a[0].startswith('unit_testing'): |
| 3601 | return -1 |
| 3602 | if b[0].startswith('unit_testing'): |
| 3603 | return 1 |
| 3604 | if a[0].startswith('native_posix'): |
| 3605 | return -1 |
| 3606 | if b[0].startswith('native_posix'): |
| 3607 | return 1 |
| 3608 | if a[0].split("/",1)[0].endswith("_bsim"): |
| 3609 | return -1 |
| 3610 | if b[0].split("/",1)[0].endswith("_bsim"): |
| 3611 | return 1 |
| 3612 | |
| 3613 | return (a > b) - (a < b) |
| 3614 | |
| 3615 | |
| 3616 | run_individual_tests = None |
| 3617 | options = None |
Andrew Boie | bbd670c | 2015-08-17 13:16:11 -0700 | [diff] [blame] | 3618 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3619 | def main(): |
Andrew Boie | 4b18247 | 2015-07-31 12:25:22 -0700 | [diff] [blame] | 3620 | start_time = time.time() |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3621 | global VERBOSE, log_file |
Anas Nashif | e10b651 | 2017-12-30 13:01:45 -0500 | [diff] [blame] | 3622 | global options |
Anas Nashif | 1ea5d7b | 2018-07-12 09:25:22 -0500 | [diff] [blame] | 3623 | global run_individual_tests |
Andrew Boie | 1578ef7 | 2019-07-03 10:19:29 -0700 | [diff] [blame] | 3624 | |
Anas Nashif | e10b651 | 2017-12-30 13:01:45 -0500 | [diff] [blame] | 3625 | options = parse_arguments() |
Andrew Boie | bbd670c | 2015-08-17 13:16:11 -0700 | [diff] [blame] | 3626 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3627 | |
| 3628 | if options.generate_hardware_map: |
| 3629 | from serial.tools import list_ports |
| 3630 | serial_devices = list_ports.comports() |
| 3631 | filtered = [] |
| 3632 | for d in serial_devices: |
Kumar Gala | a4b2b5b | 2019-11-13 11:11:32 -0600 | [diff] [blame] | 3633 | if d.manufacturer in ['ARM', 'SEGGER', 'MBED', 'STMicroelectronics', |
Peter Bigot | 52356eb | 2019-11-20 13:59:56 -0600 | [diff] [blame] | 3634 | 'Atmel Corp.', 'Texas Instruments', |
| 3635 | 'Silicon Labs', 'NXP Semiconductors']: |
Kumar Gala | a4b2b5b | 2019-11-13 11:11:32 -0600 | [diff] [blame] | 3636 | # TI XDS110 can have multiple serial devices for a single board |
| 3637 | # assume endpoint 0 is the serial, skip all others |
| 3638 | if d.manufacturer == 'Texas Instruments' and not d.location.endswith('0'): |
| 3639 | continue |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3640 | s_dev = {} |
| 3641 | s_dev['platform'] = "unknown" |
| 3642 | s_dev['id'] = d.serial_number |
| 3643 | s_dev['serial'] = d.device |
| 3644 | s_dev['product'] = d.product |
| 3645 | if s_dev['product'] in ['DAPLink CMSIS-DAP', 'MBED CMSIS-DAP']: |
| 3646 | s_dev['runner'] = "pyocd" |
Peter Bigot | 52356eb | 2019-11-20 13:59:56 -0600 | [diff] [blame] | 3647 | elif s_dev['product'] in ['J-Link', 'J-Link OB']: |
Kumar Gala | 1ad3543 | 2019-11-13 07:21:12 -0600 | [diff] [blame] | 3648 | s_dev['runner'] = "jlink" |
| 3649 | elif s_dev['product'] in ['STM32 STLink']: |
| 3650 | s_dev['runner'] = "openocd" |
Kumar Gala | a4b2b5b | 2019-11-13 11:11:32 -0600 | [diff] [blame] | 3651 | elif s_dev['product'].startswith('XDS110'): |
| 3652 | s_dev['runner'] = "openocd" |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3653 | else: |
| 3654 | s_dev['runner'] = "unknown" |
| 3655 | s_dev['available'] = True |
| 3656 | s_dev['connected'] = True |
| 3657 | filtered.append(s_dev) |
| 3658 | else: |
| 3659 | print("Unsupported device (%s): %s" %(d.manufacturer, d)) |
| 3660 | |
| 3661 | if os.path.exists(options.generate_hardware_map): |
| 3662 | # use existing map |
| 3663 | |
| 3664 | with open(options.generate_hardware_map, 'r') as yaml_file: |
| 3665 | hwm = yaml.load(yaml_file, Loader=yaml.FullLoader) |
| 3666 | # disconnect everything |
| 3667 | for h in hwm: |
| 3668 | h['connected'] = False |
Peter Bigot | 57c6355 | 2019-11-21 12:14:02 -0600 | [diff] [blame] | 3669 | h['serial'] = None |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3670 | |
| 3671 | for d in filtered: |
| 3672 | for h in hwm: |
| 3673 | if d['id'] == h['id'] and d['product'] == h['product']: |
| 3674 | print("Already in map: %s (%s)" %(d['product'], d['id'])) |
| 3675 | h['connected'] = True |
| 3676 | h['serial'] = d['serial'] |
| 3677 | d['match'] = True |
| 3678 | |
| 3679 | new = list(filter(lambda n: not n.get('match', False), filtered)) |
| 3680 | hwm = hwm + new |
| 3681 | |
| 3682 | #import pprint |
| 3683 | #pprint.pprint(hwm) |
| 3684 | with open(options.generate_hardware_map, 'w') as yaml_file: |
| 3685 | yaml.dump(hwm, yaml_file, default_flow_style=False) |
| 3686 | |
| 3687 | |
| 3688 | else: |
| 3689 | # create new file |
| 3690 | with open(options.generate_hardware_map, 'w') as yaml_file: |
| 3691 | yaml.dump(filtered, yaml_file, default_flow_style=False) |
| 3692 | |
| 3693 | return |
| 3694 | |
| 3695 | |
Michael Scott | 421ce46 | 2019-06-18 09:37:46 -0700 | [diff] [blame] | 3696 | if options.west_runner and not options.west_flash: |
| 3697 | error("west-runner requires west-flash to be enabled") |
| 3698 | sys.exit(1) |
| 3699 | |
Michael Scott | 4ca5439 | 2019-07-09 14:21:30 -0700 | [diff] [blame] | 3700 | if options.west_flash and not options.device_testing: |
| 3701 | error("west-flash requires device-testing to be enabled") |
| 3702 | sys.exit(1) |
| 3703 | |
Alberto Escolar Piedras | c026c2e | 2018-06-21 09:30:20 +0200 | [diff] [blame] | 3704 | if options.coverage: |
| 3705 | options.enable_coverage = True |
Andrew Boie | 8047a6f | 2019-07-02 15:43:29 -0700 | [diff] [blame] | 3706 | options.enable_slow = True |
Christian Taedcke | c415a4e | 2019-11-23 23:25:36 +0100 | [diff] [blame] | 3707 | |
| 3708 | if not options.coverage_platform: |
| 3709 | options.coverage_platform = options.platform |
Alberto Escolar Piedras | c026c2e | 2018-06-21 09:30:20 +0200 | [diff] [blame] | 3710 | |
Anas Nashif | e10b651 | 2017-12-30 13:01:45 -0500 | [diff] [blame] | 3711 | if options.size: |
| 3712 | for fn in options.size: |
Andrew Boie | 52fef67 | 2016-11-29 12:21:59 -0800 | [diff] [blame] | 3713 | size_report(SizeCalculator(fn, [])) |
Andrew Boie | bbd670c | 2015-08-17 13:16:11 -0700 | [diff] [blame] | 3714 | sys.exit(0) |
| 3715 | |
Anas Nashif | e10b651 | 2017-12-30 13:01:45 -0500 | [diff] [blame] | 3716 | VERBOSE += options.verbose |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3717 | |
Anas Nashif | e10b651 | 2017-12-30 13:01:45 -0500 | [diff] [blame] | 3718 | if options.log_file: |
| 3719 | log_file = open(options.log_file, "w") |
Oleg Zhurakivskyy | c97054c | 2018-08-17 14:56:05 +0300 | [diff] [blame] | 3720 | |
Anas Nashif | e10b651 | 2017-12-30 13:01:45 -0500 | [diff] [blame] | 3721 | if options.subset: |
| 3722 | subset, sets = options.subset.split("/") |
Anas Nashif | 035799f | 2017-05-13 21:31:53 -0400 | [diff] [blame] | 3723 | if int(subset) > 0 and int(sets) >= int(subset): |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3724 | info("Running only a subset: %s/%s" % (subset, sets)) |
Anas Nashif | 035799f | 2017-05-13 21:31:53 -0400 | [diff] [blame] | 3725 | else: |
Anas Nashif | e10b651 | 2017-12-30 13:01:45 -0500 | [diff] [blame] | 3726 | error("You have provided a wrong subset value: %s." % options.subset) |
Anas Nashif | 035799f | 2017-05-13 21:31:53 -0400 | [diff] [blame] | 3727 | return |
| 3728 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3729 | # Cleanup |
| 3730 | |
| 3731 | if options.no_clean or options.only_failed or options.test_only: |
| 3732 | if os.path.exists(options.outdir): |
| 3733 | info("Keeping artifacts untouched") |
| 3734 | elif os.path.exists(options.outdir): |
| 3735 | for i in range(1,100): |
| 3736 | new_out = options.outdir + ".{}".format(i) |
| 3737 | if not os.path.exists(new_out): |
| 3738 | info("Renaming output directory to {}".format(new_out)) |
| 3739 | shutil.move(options.outdir, new_out) |
| 3740 | break |
| 3741 | #shutil.rmtree("%s.old" %options.outdir) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3742 | |
Anas Nashif | e10b651 | 2017-12-30 13:01:45 -0500 | [diff] [blame] | 3743 | if not options.testcase_root: |
| 3744 | options.testcase_root = [os.path.join(ZEPHYR_BASE, "tests"), |
Andrew Boie | 3d34871 | 2016-04-08 11:52:13 -0700 | [diff] [blame] | 3745 | os.path.join(ZEPHYR_BASE, "samples")] |
| 3746 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3747 | suite = TestSuite(options.board_root, options.testcase_root, options.outdir) |
| 3748 | suite.add_testcases() |
| 3749 | suite.add_configurations() |
Anas Nashif | bd166f4 | 2017-09-02 12:32:08 -0400 | [diff] [blame] | 3750 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3751 | if options.device_testing: |
| 3752 | if options.hardware_map: |
| 3753 | suite.load_hardware_map(options.hardware_map) |
| 3754 | if not options.platform: |
| 3755 | options.platform = [] |
| 3756 | for platform in suite.connected_hardware: |
| 3757 | if platform['connected']: |
| 3758 | options.platform.append(platform['platform']) |
| 3759 | |
| 3760 | elif options.device_serial: #back-ward compatibility |
| 3761 | if options.platform and len(options.platform) == 1: |
| 3762 | suite.load_hardware_map_from_cmdline(options.device_serial, |
| 3763 | options.platform[0]) |
| 3764 | else: |
| 3765 | error("""When --device-testing is used with --device-serial, only one |
| 3766 | platform is allowed""") |
| 3767 | |
| 3768 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3769 | if suite.load_errors: |
Kumar Gala | c84235e | 2018-04-10 13:32:51 -0500 | [diff] [blame] | 3770 | sys.exit(1) |
| 3771 | |
Anas Nashif | 75547e2 | 2018-02-24 08:32:14 -0600 | [diff] [blame] | 3772 | if options.list_tags: |
| 3773 | tags = set() |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3774 | for _, tc in suite.testcases.items(): |
Anas Nashif | 75547e2 | 2018-02-24 08:32:14 -0600 | [diff] [blame] | 3775 | tags = tags.union(tc.tags) |
| 3776 | |
| 3777 | for t in tags: |
| 3778 | print("- {}".format(t)) |
| 3779 | |
| 3780 | return |
| 3781 | |
Anas Nashif | 5d6e7eb | 2018-06-01 09:51:08 -0500 | [diff] [blame] | 3782 | if options.export_tests: |
| 3783 | cnt = 0 |
Anas Nashif | eabaa7f | 2019-11-18 07:49:17 -0800 | [diff] [blame] | 3784 | tests = suite.get_all_tests() |
Anas Nashif | 5d6e7eb | 2018-06-01 09:51:08 -0500 | [diff] [blame] | 3785 | export_tests(options.export_tests, tests) |
| 3786 | return |
| 3787 | |
Anas Nashif | 1ea5d7b | 2018-07-12 09:25:22 -0500 | [diff] [blame] | 3788 | run_individual_tests = [] |
Anas Nashif | 5d6e7eb | 2018-06-01 09:51:08 -0500 | [diff] [blame] | 3789 | |
Anas Nashif | 1ea5d7b | 2018-07-12 09:25:22 -0500 | [diff] [blame] | 3790 | if options.test: |
| 3791 | run_individual_tests = options.test |
| 3792 | |
Anas Nashif | 19ca783 | 2019-11-18 08:16:21 -0800 | [diff] [blame] | 3793 | if options.list_tests or options.list_test_duplicates or options.sub_test: |
Anas Nashif | 49b22d4 | 2019-06-14 13:45:34 -0400 | [diff] [blame] | 3794 | cnt = 0 |
Anas Nashif | eabaa7f | 2019-11-18 07:49:17 -0800 | [diff] [blame] | 3795 | all_tests = suite.get_all_tests() |
Anas Nashif | 49b22d4 | 2019-06-14 13:45:34 -0400 | [diff] [blame] | 3796 | |
Anas Nashif | 19ca783 | 2019-11-18 08:16:21 -0800 | [diff] [blame] | 3797 | if options.list_test_duplicates: |
| 3798 | import collections |
| 3799 | dupes = [item for item, count in collections.Counter(all_tests).items() if count > 1] |
| 3800 | if dupes: |
| 3801 | print("Tests with duplicate identifiers:") |
| 3802 | for dupe in dupes: |
| 3803 | print("- {}".format(dupe)) |
| 3804 | for dc in suite.get_testcase(dupe): |
| 3805 | print(" - {}".format(dc)) |
| 3806 | else: |
| 3807 | print("No duplicates found.") |
| 3808 | return |
| 3809 | |
Anas Nashif | 1ea5d7b | 2018-07-12 09:25:22 -0500 | [diff] [blame] | 3810 | if options.sub_test: |
Anas Nashif | d11fd78 | 2019-11-18 10:22:56 -0800 | [diff] [blame] | 3811 | for st in options.sub_test: |
| 3812 | subtests = suite.get_testcase(st) |
| 3813 | for sti in subtests: |
| 3814 | run_individual_tests.append(sti.name) |
| 3815 | |
Anas Nashif | 1ea5d7b | 2018-07-12 09:25:22 -0500 | [diff] [blame] | 3816 | if run_individual_tests: |
| 3817 | info("Running the following tests:") |
Anas Nashif | eabaa7f | 2019-11-18 07:49:17 -0800 | [diff] [blame] | 3818 | for test in run_individual_tests: |
| 3819 | print(" - {}".format(test)) |
Anas Nashif | 1ea5d7b | 2018-07-12 09:25:22 -0500 | [diff] [blame] | 3820 | else: |
| 3821 | info("Tests not found") |
| 3822 | return |
| 3823 | |
| 3824 | elif options.list_tests: |
Anas Nashif | eabaa7f | 2019-11-18 07:49:17 -0800 | [diff] [blame] | 3825 | for test in all_tests: |
Anas Nashif | 1ea5d7b | 2018-07-12 09:25:22 -0500 | [diff] [blame] | 3826 | cnt = cnt + 1 |
Anas Nashif | eabaa7f | 2019-11-18 07:49:17 -0800 | [diff] [blame] | 3827 | print(" - {}".format(test)) |
Anas Nashif | 1ea5d7b | 2018-07-12 09:25:22 -0500 | [diff] [blame] | 3828 | print("{} total.".format(cnt)) |
| 3829 | return |
Anas Nashif | c0149cc | 2018-04-14 23:12:58 -0500 | [diff] [blame] | 3830 | |
Anas Nashif | bd166f4 | 2017-09-02 12:32:08 -0400 | [diff] [blame] | 3831 | discards = [] |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3832 | |
| 3833 | if options.only_failed: |
| 3834 | suite.get_last_failed() |
| 3835 | elif options.load_tests: |
| 3836 | suite.load_from_file(options.load_tests) |
| 3837 | elif options.test_only: |
| 3838 | last_run = os.path.join(options.outdir, "sanitycheck.csv") |
| 3839 | suite.load_from_file(last_run) |
Anas Nashif | bd166f4 | 2017-09-02 12:32:08 -0400 | [diff] [blame] | 3840 | else: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3841 | discards = suite.apply_filters() |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3842 | |
Anas Nashif | 30551f4 | 2018-01-12 21:56:59 -0500 | [diff] [blame] | 3843 | if VERBOSE > 1 and discards: |
Anas Nashif | f18e2ab | 2018-01-13 07:57:42 -0500 | [diff] [blame] | 3844 | # if we are using command line platform filter, no need to list every |
| 3845 | # other platform as excluded, we know that already. |
| 3846 | # Show only the discards that apply to the selected platforms on the |
| 3847 | # command line |
| 3848 | |
Andrew Boie | 08ce5a5 | 2016-02-22 13:28:10 -0800 | [diff] [blame] | 3849 | for i, reason in discards.items(): |
Anas Nashif | f18e2ab | 2018-01-13 07:57:42 -0500 | [diff] [blame] | 3850 | if options.platform and i.platform.name not in options.platform: |
| 3851 | continue |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3852 | debug( |
| 3853 | "{:<25} {:<50} {}SKIPPED{}: {}".format( |
| 3854 | i.platform.name, |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3855 | i.testcase.name, |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3856 | COLOR_YELLOW, |
| 3857 | COLOR_NORMAL, |
| 3858 | reason)) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3859 | |
Anas Nashif | 49b22d4 | 2019-06-14 13:45:34 -0400 | [diff] [blame] | 3860 | if options.report_excluded: |
Anas Nashif | eabaa7f | 2019-11-18 07:49:17 -0800 | [diff] [blame] | 3861 | all_tests = suite.get_all_tests() |
Anas Nashif | 49b22d4 | 2019-06-14 13:45:34 -0400 | [diff] [blame] | 3862 | to_be_run = set() |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3863 | for i,p in suite.instances.items(): |
| 3864 | to_be_run.update(p.testcase.cases) |
Anas Nashif | 49b22d4 | 2019-06-14 13:45:34 -0400 | [diff] [blame] | 3865 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3866 | if all_tests - to_be_run: |
Anas Nashif | 49b22d4 | 2019-06-14 13:45:34 -0400 | [diff] [blame] | 3867 | print("Tests that never build or run:") |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3868 | for not_run in all_tests - to_be_run: |
Anas Nashif | 49b22d4 | 2019-06-14 13:45:34 -0400 | [diff] [blame] | 3869 | print("- {}".format(not_run)) |
| 3870 | |
| 3871 | return |
| 3872 | |
Anas Nashif | e10b651 | 2017-12-30 13:01:45 -0500 | [diff] [blame] | 3873 | if options.subset: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3874 | #suite.instances = OrderedDict(sorted(suite.instances.items(), |
| 3875 | # key=cmp_to_key(native_and_unit_first))) |
Anas Nashif | e10b651 | 2017-12-30 13:01:45 -0500 | [diff] [blame] | 3876 | subset, sets = options.subset.split("/") |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3877 | total = len(suite.instances) |
Anas Nashif | 035799f | 2017-05-13 21:31:53 -0400 | [diff] [blame] | 3878 | per_set = round(total / int(sets)) |
Anas Nashif | 3ba1d43 | 2017-12-05 15:28:44 -0500 | [diff] [blame] | 3879 | start = (int(subset) - 1) * per_set |
Anas Nashif | 035799f | 2017-05-13 21:31:53 -0400 | [diff] [blame] | 3880 | if subset == sets: |
| 3881 | end = total |
| 3882 | else: |
| 3883 | end = start + per_set |
| 3884 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3885 | sliced_instances = islice(suite.instances.items(), start, end) |
| 3886 | suite.instances = OrderedDict(sliced_instances) |
Anas Nashif | 035799f | 2017-05-13 21:31:53 -0400 | [diff] [blame] | 3887 | |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3888 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3889 | if options.save_tests: |
| 3890 | suite.csv_report(options.save_tests) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3891 | return |
| 3892 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3893 | info("%d test configurations selected, %d configurations discarded due to filters." % |
| 3894 | (len(suite.instances), len(discards))) |
| 3895 | |
Peter Bigot | 3d46ea5 | 2019-11-21 12:00:18 -0600 | [diff] [blame] | 3896 | if options.device_testing: |
| 3897 | print("\nDevice testing on:") |
| 3898 | for p in suite.connected_hardware: |
| 3899 | if p['connected']: |
| 3900 | print("%s (%s) on %s" %(p['platform'], p.get('id', None), p['serial'])) |
| 3901 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3902 | if options.dry_run: |
| 3903 | duration = time.time() - start_time |
| 3904 | info("Completed in %d seconds" % (duration)) |
| 3905 | return |
| 3906 | |
| 3907 | retries = options.retry_failed + 1 |
| 3908 | completed = 0 |
| 3909 | |
| 3910 | suite.update() |
| 3911 | suite.start_time = start_time |
| 3912 | |
| 3913 | while True: |
| 3914 | completed += 1 |
| 3915 | |
| 3916 | if completed > 1: |
| 3917 | info("%d Iteration:" %(completed )) |
| 3918 | time.sleep(60) # waiting for the system to settle down |
| 3919 | suite.total_done = suite.total_tests - suite.total_failed |
| 3920 | suite.total_failed = 0 |
| 3921 | |
| 3922 | suite.execute() |
Anas Nashif | 654ec598 | 2019-04-11 08:38:21 -0400 | [diff] [blame] | 3923 | info("", False) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3924 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3925 | retries = retries - 1 |
| 3926 | if retries == 0 or suite.total_failed == 0: |
| 3927 | break |
Anas Nashif | e0a6a0b | 2018-02-15 20:07:24 -0600 | [diff] [blame] | 3928 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3929 | suite.misc_reports(options.compare_report, options.show_footprint, |
| 3930 | options.all_deltas, options.footprint_threshold, options.last_metrics) |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3931 | |
Anas Nashif | 83a98e5 | 2019-11-24 07:42:06 -0500 | [diff] [blame^] | 3932 | suite.duration = time.time() - start_time |
| 3933 | suite.summary(options.disable_unrecognized_section_test) |
| 3934 | |
Anas Nashif | e10b651 | 2017-12-30 13:01:45 -0500 | [diff] [blame] | 3935 | if options.coverage: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3936 | if options.gcov_tool is None: |
Alberto Escolar Piedras | af30f2b | 2019-09-12 14:44:08 +0200 | [diff] [blame] | 3937 | use_system_gcov = False |
Andrew Boie | 49cf486 | 2019-07-08 12:02:13 -0700 | [diff] [blame] | 3938 | |
| 3939 | for plat in options.coverage_platform: |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3940 | ts_plat = suite.get_platform(plat) |
Alberto Escolar Piedras | af30f2b | 2019-09-12 14:44:08 +0200 | [diff] [blame] | 3941 | if ts_plat and (ts_plat.type in {"native", "unit"}): |
| 3942 | use_system_gcov = True |
Andrew Boie | 49cf486 | 2019-07-08 12:02:13 -0700 | [diff] [blame] | 3943 | |
Alberto Escolar Piedras | af30f2b | 2019-09-12 14:44:08 +0200 | [diff] [blame] | 3944 | if use_system_gcov or "ZEPHYR_SDK_INSTALL_DIR" not in os.environ: |
Andrew Boie | 49cf486 | 2019-07-08 12:02:13 -0700 | [diff] [blame] | 3945 | options.gcov_tool = "gcov" |
| 3946 | else: |
| 3947 | options.gcov_tool = os.path.join(os.environ["ZEPHYR_SDK_INSTALL_DIR"], |
| 3948 | "i586-zephyr-elf/bin/i586-zephyr-elf-gcov") |
| 3949 | |
Jaakko Hannikainen | 54c90bc | 2016-08-31 14:17:03 +0300 | [diff] [blame] | 3950 | info("Generating coverage files...") |
Anas Nashif | 4df6c56 | 2018-11-07 19:29:04 -0500 | [diff] [blame] | 3951 | generate_coverage(options.outdir, ["*generated*", "tests/*", "samples/*"]) |
Jaakko Hannikainen | 54c90bc | 2016-08-31 14:17:03 +0300 | [diff] [blame] | 3952 | |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3953 | if options.device_testing: |
| 3954 | print("\nHardware distribution summary:\n") |
| 3955 | for p in suite.connected_hardware: |
| 3956 | if p['connected']: |
| 3957 | print("%s (%s): %d" %(p['platform'], p.get('id', None), p['counter'])) |
| 3958 | |
Flavio Ceolin | ca1feea | 2019-10-22 14:03:48 -0700 | [diff] [blame] | 3959 | suite.save_reports() |
Anas Nashif | 83fc06a | 2019-06-22 11:04:10 -0400 | [diff] [blame] | 3960 | if suite.total_failed or (suite.warnings and options.warnings_as_errors): |
Andrew Boie | 6acbe63 | 2015-07-17 12:03:52 -0700 | [diff] [blame] | 3961 | sys.exit(1) |
| 3962 | |
| 3963 | if __name__ == "__main__": |
| 3964 | main() |