blob: d989e0b18eaabe316b095059c8614f7f73fa3d37 [file] [log] [blame]
Andrew Boie08ce5a52016-02-22 13:28:10 -08001#!/usr/bin/env python3
Anas Nashifa792a3d2017-04-04 18:47:49 -04002# vim: set syntax=python ts=4 :
Anas Nashif3ae52622019-04-06 09:08:09 -04003# SPDX-License-Identifier: Apache-2.0
Andrew Boie6acbe632015-07-17 12:03:52 -07004"""Zephyr Sanity Tests
5
Marc Herberte5cedca2019-04-08 14:02:34 -07006Also check the "User and Developer Guides" at https://docs.zephyrproject.org/
7
Andrew Boie6acbe632015-07-17 12:03:52 -07008This script scans for the set of unit test applications in the git
9repository and attempts to execute them. By default, it tries to
10build each test case on one platform per architecture, using a precedence
Paul Sokolovskyff70add2017-06-16 01:31:54 +030011list defined in an architecture configuration file, and if possible
Anas Nashif83fc06a2019-06-22 11:04:10 -040012run the tests in any available emulators or simulators on the system.
Andrew Boie6acbe632015-07-17 12:03:52 -070013
Anas Nashifa792a3d2017-04-04 18:47:49 -040014Test cases are detected by the presence of a 'testcase.yaml' or a sample.yaml
15files in the application's project directory. This file may contain one or more
16blocks, each identifying a test scenario. The title of the block is a name for
17the test case, which only needs to be unique for the test cases specified in
18that testcase meta-data. The full canonical name for each test case is <path to
19test case>/<block>.
Andrew Boie6acbe632015-07-17 12:03:52 -070020
Anas Nashif3ba1d432017-12-05 15:28:44 -050021Each test block in the testcase meta data can define the following key/value
22pairs:
Andrew Boie6acbe632015-07-17 12:03:52 -070023
Anas Nashiffa695d22017-10-04 16:14:27 -040024 tags: <list of tags> (required)
Andrew Boie6acbe632015-07-17 12:03:52 -070025 A set of string tags for the testcase. Usually pertains to
26 functional domains but can be anything. Command line invocations
27 of this script can filter the set of tests to run based on tag.
28
Anas Nashifa792a3d2017-04-04 18:47:49 -040029 skip: <True|False> (default False)
Anas Nashif2bd99bc2015-10-12 13:10:57 -040030 skip testcase unconditionally. This can be used for broken tests.
31
Anas Nashifa792a3d2017-04-04 18:47:49 -040032 slow: <True|False> (default False)
Andy Rossdc4151f2019-01-03 14:17:43 -080033 Don't build or run this test case unless --enable-slow was passed
34 in on the command line. Intended for time-consuming test cases
35 that are only run under certain circumstances, like daily
36 builds.
Andrew Boie6bb087c2016-02-10 13:39:00 -080037
Anas Nashifa792a3d2017-04-04 18:47:49 -040038 extra_args: <list of extra arguments>
Sebastian Bøec2182612017-11-09 12:25:02 +010039 Extra cache entries to pass to CMake when building or running the
Andrew Boie6acbe632015-07-17 12:03:52 -070040 test case.
41
Anas Nashifebc329d2017-10-17 09:00:33 -040042 extra_configs: <list of extra configurations>
43 Extra configuration options to be merged with a master prj.conf
44 when building or running the test case.
45
Anas Nashifa792a3d2017-04-04 18:47:49 -040046 build_only: <True|False> (default False)
Marc Herberte5cedca2019-04-08 14:02:34 -070047 If true, don't try to run the test even if the selected platform
48 supports it.
Andrew Boie6acbe632015-07-17 12:03:52 -070049
Anas Nashifa792a3d2017-04-04 18:47:49 -040050 build_on_all: <True|False> (default False)
51 If true, attempt to build test on all available platforms.
52
53 depends_on: <list of features>
54 A board or platform can announce what features it supports, this option
55 will enable the test only those platforms that provide this feature.
56
57 min_ram: <integer>
58 minimum amount of RAM needed for this test to build and run. This is
59 compared with information provided by the board metadata.
60
61 min_flash: <integer>
62 minimum amount of ROM needed for this test to build and run. This is
63 compared with information provided by the board metadata.
64
65 timeout: <number of seconds>
Anas Nashif83fc06a2019-06-22 11:04:10 -040066 Length of time to run test in emulator before automatically killing it.
Andrew Boie6acbe632015-07-17 12:03:52 -070067 Default to 60 seconds.
68
Anas Nashifa792a3d2017-04-04 18:47:49 -040069 arch_whitelist: <list of arches, such as x86, arm, arc>
Andrew Boie6acbe632015-07-17 12:03:52 -070070 Set of architectures that this test case should only be run for.
71
Anas Nashifa792a3d2017-04-04 18:47:49 -040072 arch_exclude: <list of arches, such as x86, arm, arc>
Anas Nashif30d13872015-10-05 10:02:45 -040073 Set of architectures that this test case should not run on.
74
Anas Nashifa792a3d2017-04-04 18:47:49 -040075 platform_whitelist: <list of platforms>
Anas Nashif30d13872015-10-05 10:02:45 -040076 Set of platforms that this test case should only be run for.
77
Anas Nashifa792a3d2017-04-04 18:47:49 -040078 platform_exclude: <list of platforms>
Anas Nashif30d13872015-10-05 10:02:45 -040079 Set of platforms that this test case should not run on.
Andrew Boie6acbe632015-07-17 12:03:52 -070080
Anas Nashifa792a3d2017-04-04 18:47:49 -040081 extra_sections: <list of extra binary sections>
Andrew Boie52fef672016-11-29 12:21:59 -080082 When computing sizes, sanitycheck will report errors if it finds
83 extra, unexpected sections in the Zephyr binary unless they are named
84 here. They will not be included in the size calculation.
85
Anas Nashifa792a3d2017-04-04 18:47:49 -040086 filter: <expression>
Andrew Boie3ea78922016-03-24 14:46:00 -070087 Filter whether the testcase should be run by evaluating an expression
88 against an environment containing the following values:
89
90 { ARCH : <architecture>,
91 PLATFORM : <platform>,
Javier B Perez79414542016-08-08 12:24:59 -050092 <all CONFIG_* key/value pairs in the test's generated defconfig>,
Anas Nashif45a97862019-01-09 08:46:42 -050093 <all DT_* key/value pairs in the test's generated device tree file>,
94 <all CMake key/value pairs in the test's generated CMakeCache.txt file>,
Javier B Perez79414542016-08-08 12:24:59 -050095 *<env>: any environment variable available
Andrew Boie3ea78922016-03-24 14:46:00 -070096 }
97
98 The grammar for the expression language is as follows:
99
100 expression ::= expression "and" expression
101 | expression "or" expression
102 | "not" expression
103 | "(" expression ")"
104 | symbol "==" constant
105 | symbol "!=" constant
106 | symbol "<" number
107 | symbol ">" number
108 | symbol ">=" number
109 | symbol "<=" number
110 | symbol "in" list
Andrew Boie7a87f9f2016-06-02 12:27:54 -0700111 | symbol ":" string
Andrew Boie3ea78922016-03-24 14:46:00 -0700112 | symbol
113
114 list ::= "[" list_contents "]"
115
116 list_contents ::= constant
117 | list_contents "," constant
118
119 constant ::= number
120 | string
121
122
123 For the case where expression ::= symbol, it evaluates to true
124 if the symbol is defined to a non-empty string.
125
126 Operator precedence, starting from lowest to highest:
127
128 or (left associative)
129 and (left associative)
130 not (right associative)
131 all comparison operators (non-associative)
132
133 arch_whitelist, arch_exclude, platform_whitelist, platform_exclude
134 are all syntactic sugar for these expressions. For instance
135
136 arch_exclude = x86 arc
137
138 Is the same as:
139
140 filter = not ARCH in ["x86", "arc"]
Andrew Boie6acbe632015-07-17 12:03:52 -0700141
Andrew Boie7a87f9f2016-06-02 12:27:54 -0700142 The ':' operator compiles the string argument as a regular expression,
143 and then returns a true value only if the symbol's value in the environment
Anas Nashif578ae402019-07-12 07:54:35 -0700144 matches. For example, if CONFIG_SOC="stm32f107xc" then
Andrew Boie7a87f9f2016-06-02 12:27:54 -0700145
Anas Nashif578ae402019-07-12 07:54:35 -0700146 filter = CONFIG_SOC : "stm.*"
Andrew Boie7a87f9f2016-06-02 12:27:54 -0700147
148 Would match it.
149
Anas Nashifa792a3d2017-04-04 18:47:49 -0400150The set of test cases that actually run depends on directives in the testcase
151filed and options passed in on the command line. If there is any confusion,
Anas Nashif12d8cce2019-11-20 03:47:27 -0800152running with -v or examining the discard report (sanitycheck_discard.csv)
153can help show why particular test cases were skipped.
Andrew Boie6acbe632015-07-17 12:03:52 -0700154
155Metrics (such as pass/fail state and binary size) for the last code
156release are stored in scripts/sanity_chk/sanity_last_release.csv.
157To update this, pass the --all --release options.
158
Genaro Saucedo Tejada28bba922016-10-24 18:00:58 -0500159To load arguments from a file, write '+' before the file name, e.g.,
160+file_name. File content must be one or more valid arguments separated by
161line break instead of white spaces.
162
Andrew Boie6acbe632015-07-17 12:03:52 -0700163Most everyday users will run with no arguments.
Andy Rossdc4151f2019-01-03 14:17:43 -0800164
Andrew Boie6acbe632015-07-17 12:03:52 -0700165"""
166
Sebastian Bøe56d74712019-01-21 15:48:46 +0100167import os
Anas Nashifaae71d72018-04-21 22:26:48 -0500168import contextlib
Anas Nashifa4c368e2018-10-15 09:45:59 -0400169import string
Anas Nashifaae71d72018-04-21 22:26:48 -0500170import mmap
Andrew Boie6acbe632015-07-17 12:03:52 -0700171import argparse
Andrew Boie6acbe632015-07-17 12:03:52 -0700172import sys
Andrew Boie6acbe632015-07-17 12:03:52 -0700173import re
Andrew Boie6acbe632015-07-17 12:03:52 -0700174import subprocess
175import multiprocessing
176import select
177import shutil
Marc Herbertaf1090c2019-04-30 14:11:29 -0700178import shlex
Andrew Boie6acbe632015-07-17 12:03:52 -0700179import signal
180import threading
Anas Nashif83fc06a2019-06-22 11:04:10 -0400181import concurrent.futures
182from threading import BoundedSemaphore
183import queue
Andrew Boie6acbe632015-07-17 12:03:52 -0700184import time
Anas Nashif654ec5982019-04-11 08:38:21 -0400185import datetime
Andrew Boie6acbe632015-07-17 12:03:52 -0700186import csv
Anas Nashif83fc06a2019-06-22 11:04:10 -0400187import yaml
Andrew Boie5d4eb782015-10-02 10:04:56 -0700188import glob
Anas Nashif73440ea2018-02-19 10:57:03 -0600189import serial
Daniel Leung6b170072016-04-07 12:10:25 -0700190import concurrent
Anas Nashifb3311ed2017-04-13 14:44:48 -0400191import xml.etree.ElementTree as ET
Anas Nashif035799f2017-05-13 21:31:53 -0400192from collections import OrderedDict
193from itertools import islice
Anas Nashife24350c2018-07-11 15:09:22 -0500194from pathlib import Path
195from distutils.spawn import find_executable
Andrew Boie6acbe632015-07-17 12:03:52 -0700196
Kumar Gala7733b942019-09-12 17:08:43 -0500197ZEPHYR_BASE = os.getenv("ZEPHYR_BASE")
198if not ZEPHYR_BASE:
199 sys.exit("$ZEPHYR_BASE environment variable undefined")
200
201sys.path.insert(0, os.path.join(ZEPHYR_BASE, "scripts", "dts"))
202import edtlib
203
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -0700204import logging
Anas Nashif83fc06a2019-06-22 11:04:10 -0400205
206
207hw_map_local = threading.Lock()
Anas Nashifc1ea4522019-10-11 07:32:45 -0700208report_lock = threading.Lock()
Anas Nashif83fc06a2019-06-22 11:04:10 -0400209
Anas Nashif3ba1d432017-12-05 15:28:44 -0500210
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -0700211log_format = "%(levelname)s %(name)s::%(module)s.%(funcName)s():%(lineno)d: %(message)s"
Anas Nashif3ba1d432017-12-05 15:28:44 -0500212logging.basicConfig(format=log_format, level=30)
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -0700213
Marc Herbert1c8632c2019-04-15 17:58:45 -0700214# Use this for internal comparisons; that's what canonicalization is
215# for. Don't use it when invoking other components of the build system
216# to avoid confusing and hard to trace inconsistencies in error messages
217# and logs, generated Makefiles, etc. compared to when users invoke these
218# components directly.
219# Note "normalization" is different from canonicalization, see os.path.
220canonical_zephyr_base = os.path.realpath(ZEPHYR_BASE)
221
Andrew Boie3ea78922016-03-24 14:46:00 -0700222sys.path.insert(0, os.path.join(ZEPHYR_BASE, "scripts/"))
223
Anas Nashif83fc06a2019-06-22 11:04:10 -0400224from sanity_chk import scl
225from sanity_chk import expr_parser
226
Andrew Boie3ea78922016-03-24 14:46:00 -0700227
Andrew Boie6acbe632015-07-17 12:03:52 -0700228VERBOSE = 0
Anas Nashif83fc06a2019-06-22 11:04:10 -0400229
Andrew Boie6acbe632015-07-17 12:03:52 -0700230RELEASE_DATA = os.path.join(ZEPHYR_BASE, "scripts", "sanity_chk",
231 "sanity_last_release.csv")
Andrew Boie6acbe632015-07-17 12:03:52 -0700232
233if os.isatty(sys.stdout.fileno()):
234 TERMINAL = True
235 COLOR_NORMAL = '\033[0m'
236 COLOR_RED = '\033[91m'
237 COLOR_GREEN = '\033[92m'
238 COLOR_YELLOW = '\033[93m'
239else:
240 TERMINAL = False
241 COLOR_NORMAL = ""
242 COLOR_RED = ""
243 COLOR_GREEN = ""
244 COLOR_YELLOW = ""
245
Anas Nashif45a97862019-01-09 08:46:42 -0500246class CMakeCacheEntry:
247 '''Represents a CMake cache entry.
248
249 This class understands the type system in a CMakeCache.txt, and
250 converts the following cache types to Python types:
251
252 Cache Type Python type
253 ---------- -------------------------------------------
254 FILEPATH str
255 PATH str
256 STRING str OR list of str (if ';' is in the value)
257 BOOL bool
258 INTERNAL str OR list of str (if ';' is in the value)
259 ---------- -------------------------------------------
260 '''
261
262 # Regular expression for a cache entry.
263 #
264 # CMake variable names can include escape characters, allowing a
265 # wider set of names than is easy to match with a regular
266 # expression. To be permissive here, use a non-greedy match up to
267 # the first colon (':'). This breaks if the variable name has a
268 # colon inside, but it's good enough.
269 CACHE_ENTRY = re.compile(
270 r'''(?P<name>.*?) # name
271 :(?P<type>FILEPATH|PATH|STRING|BOOL|INTERNAL) # type
272 =(?P<value>.*) # value
273 ''', re.X)
274
275 @classmethod
276 def _to_bool(cls, val):
277 # Convert a CMake BOOL string into a Python bool.
278 #
279 # "True if the constant is 1, ON, YES, TRUE, Y, or a
280 # non-zero number. False if the constant is 0, OFF, NO,
281 # FALSE, N, IGNORE, NOTFOUND, the empty string, or ends in
282 # the suffix -NOTFOUND. Named boolean constants are
283 # case-insensitive. If the argument is not one of these
284 # constants, it is treated as a variable."
285 #
286 # https://cmake.org/cmake/help/v3.0/command/if.html
287 val = val.upper()
288 if val in ('ON', 'YES', 'TRUE', 'Y'):
289 return 1
290 elif val in ('OFF', 'NO', 'FALSE', 'N', 'IGNORE', 'NOTFOUND', ''):
291 return 0
292 elif val.endswith('-NOTFOUND'):
293 return 0
294 else:
295 try:
296 v = int(val)
297 return v != 0
298 except ValueError as exc:
299 raise ValueError('invalid bool {}'.format(val)) from exc
300
301 @classmethod
302 def from_line(cls, line, line_no):
303 # Comments can only occur at the beginning of a line.
304 # (The value of an entry could contain a comment character).
305 if line.startswith('//') or line.startswith('#'):
306 return None
307
308 # Whitespace-only lines do not contain cache entries.
309 if not line.strip():
310 return None
311
312 m = cls.CACHE_ENTRY.match(line)
313 if not m:
314 return None
315
316 name, type_, value = (m.group(g) for g in ('name', 'type', 'value'))
317 if type_ == 'BOOL':
318 try:
319 value = cls._to_bool(value)
320 except ValueError as exc:
321 args = exc.args + ('on line {}: {}'.format(line_no, line),)
322 raise ValueError(args) from exc
Anas Nashif83fc06a2019-06-22 11:04:10 -0400323 elif type_ in ['STRING','INTERNAL']:
Anas Nashif45a97862019-01-09 08:46:42 -0500324 # If the value is a CMake list (i.e. is a string which
325 # contains a ';'), convert to a Python list.
326 if ';' in value:
327 value = value.split(';')
328
329 return CMakeCacheEntry(name, value)
330
331 def __init__(self, name, value):
332 self.name = name
333 self.value = value
334
335 def __str__(self):
336 fmt = 'CMakeCacheEntry(name={}, value={})'
337 return fmt.format(self.name, self.value)
338
339
340class CMakeCache:
341 '''Parses and represents a CMake cache file.'''
342
343 @staticmethod
344 def from_file(cache_file):
345 return CMakeCache(cache_file)
346
347 def __init__(self, cache_file):
348 self.cache_file = cache_file
349 self.load(cache_file)
350
351 def load(self, cache_file):
352 entries = []
353 with open(cache_file, 'r') as cache:
354 for line_no, line in enumerate(cache):
355 entry = CMakeCacheEntry.from_line(line, line_no)
356 if entry:
357 entries.append(entry)
358 self._entries = OrderedDict((e.name, e) for e in entries)
359
360 def get(self, name, default=None):
361 entry = self._entries.get(name)
362 if entry is not None:
363 return entry.value
364 else:
365 return default
366
367 def get_list(self, name, default=None):
368 if default is None:
369 default = []
370 entry = self._entries.get(name)
371 if entry is not None:
372 value = entry.value
373 if isinstance(value, list):
374 return value
375 elif isinstance(value, str):
376 return [value] if value else []
377 else:
378 msg = 'invalid value {} type {}'
379 raise RuntimeError(msg.format(value, type(value)))
380 else:
381 return default
382
383 def __contains__(self, name):
384 return name in self._entries
385
386 def __getitem__(self, name):
387 return self._entries[name].value
388
389 def __setitem__(self, name, entry):
390 if not isinstance(entry, CMakeCacheEntry):
391 msg = 'improper type {} for value {}, expecting CMakeCacheEntry'
392 raise TypeError(msg.format(type(entry), entry))
393 self._entries[name] = entry
394
395 def __delitem__(self, name):
396 del self._entries[name]
397
398 def __iter__(self):
399 return iter(self._entries.values())
400
Andrew Boie6acbe632015-07-17 12:03:52 -0700401class SanityCheckException(Exception):
402 pass
403
Anas Nashif3ba1d432017-12-05 15:28:44 -0500404
Andrew Boie6acbe632015-07-17 12:03:52 -0700405class SanityRuntimeError(SanityCheckException):
406 pass
407
408class ConfigurationError(SanityCheckException):
409 def __init__(self, cfile, message):
Anas Nashif83fc06a2019-06-22 11:04:10 -0400410 SanityCheckException.__init__(self, cfile + ": " + message)
Andrew Boie6acbe632015-07-17 12:03:52 -0700411
Anas Nashif83fc06a2019-06-22 11:04:10 -0400412class BuildError(SanityCheckException):
Andrew Boie6acbe632015-07-17 12:03:52 -0700413 pass
414
Anas Nashif3ba1d432017-12-05 15:28:44 -0500415
Anas Nashif83fc06a2019-06-22 11:04:10 -0400416class ExecutionError(SanityCheckException):
Andrew Boie6acbe632015-07-17 12:03:52 -0700417 pass
418
Anas Nashif3ba1d432017-12-05 15:28:44 -0500419
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -0800420log_file = None
421
Andrew Boie6acbe632015-07-17 12:03:52 -0700422# Debug Functions
Anas Nashif654ec5982019-04-11 08:38:21 -0400423def info(what, show_time=True):
424 if options.timestamps and show_time:
425 date = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
426 what = "{}: {}".format(date, what)
Andrew Boie08ce5a52016-02-22 13:28:10 -0800427 sys.stdout.write(what + "\n")
Paul Sokolovsky3ba08762017-10-27 14:53:24 +0300428 sys.stdout.flush()
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -0800429 if log_file:
430 log_file.write(what + "\n")
431 log_file.flush()
Andrew Boie6acbe632015-07-17 12:03:52 -0700432
Anas Nashif3ba1d432017-12-05 15:28:44 -0500433
Andrew Boie6acbe632015-07-17 12:03:52 -0700434def error(what):
Anas Nashif654ec5982019-04-11 08:38:21 -0400435 if options.timestamps:
436 date = datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S")
437 what = "{}: {}".format(date, what)
Andrew Boie6acbe632015-07-17 12:03:52 -0700438 sys.stderr.write(COLOR_RED + what + COLOR_NORMAL + "\n")
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -0800439 if log_file:
440 log_file(what + "\n")
441 log_file.flush()
Andrew Boie6acbe632015-07-17 12:03:52 -0700442
Anas Nashif3ba1d432017-12-05 15:28:44 -0500443
Andrew Boie08ce5a52016-02-22 13:28:10 -0800444def debug(what):
445 if VERBOSE >= 1:
446 info(what)
447
Anas Nashif3ba1d432017-12-05 15:28:44 -0500448
Andrew Boie6acbe632015-07-17 12:03:52 -0700449def verbose(what):
450 if VERBOSE >= 2:
Andrew Boie08ce5a52016-02-22 13:28:10 -0800451 info(what)
Andrew Boie6acbe632015-07-17 12:03:52 -0700452
Anas Nashif576be982017-12-23 20:20:27 -0500453class HarnessImporter:
454
455 def __init__(self, name):
456 sys.path.insert(0, os.path.join(ZEPHYR_BASE, "scripts/sanity_chk"))
457 module = __import__("harness")
458 if name:
459 my_class = getattr(module, name)
460 else:
461 my_class = getattr(module, "Test")
462
463 self.instance = my_class()
Anas Nashif3ba1d432017-12-05 15:28:44 -0500464
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300465class Handler:
Anas Nashifd18ec532019-04-11 23:20:39 -0400466 def __init__(self, instance, type_str="build"):
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300467 """Constructor
468
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300469 """
470 self.lock = threading.Lock()
Anas Nashif83fc06a2019-06-22 11:04:10 -0400471
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300472 self.state = "waiting"
Anas Nashif13773752018-07-06 18:20:23 -0500473 self.run = False
Anas Nashif83fc06a2019-06-22 11:04:10 -0400474 self.duration = 0
Anas Nashifd18ec532019-04-11 23:20:39 -0400475 self.type_str = type_str
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300476
Anas Nashifdf7ee612018-07-07 06:09:01 -0500477 self.binary = None
Jan Kowalewski265895b2019-01-07 16:40:24 +0100478 self.pid_fn = None
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500479 self.call_make_run = False
Anas Nashifdf7ee612018-07-07 06:09:01 -0500480
Anas Nashifd3384fb2018-02-22 06:44:16 -0600481 self.name = instance.name
482 self.instance = instance
Anas Nashif83fc06a2019-06-22 11:04:10 -0400483 self.timeout = instance.testcase.timeout
484 self.sourcedir = instance.testcase.source_dir
485 self.build_dir = instance.build_dir
486 self.log = os.path.join(self.build_dir, "handler.log")
Anas Nashifd3384fb2018-02-22 06:44:16 -0600487 self.returncode = 0
Anas Nashif83fc06a2019-06-22 11:04:10 -0400488 self.set_state("running", self.duration)
Anas Nashifd3384fb2018-02-22 06:44:16 -0600489
Anas Nashif83fc06a2019-06-22 11:04:10 -0400490 self.args = []
491
492 def set_state(self, state, duration):
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300493 self.lock.acquire()
494 self.state = state
Anas Nashif83fc06a2019-06-22 11:04:10 -0400495 self.duration = duration
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300496 self.lock.release()
497
498 def get_state(self):
499 self.lock.acquire()
Anas Nashif83fc06a2019-06-22 11:04:10 -0400500 ret = (self.state, self.duration)
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300501 self.lock.release()
502 return ret
503
Anas Nashif83fc06a2019-06-22 11:04:10 -0400504 def record(self, harness):
505 if harness.recording:
506 filename = os.path.join(options.outdir,
507 self.instance.platform.name,
508 self.instance.testcase.name, "recording.csv")
509 with open(filename, "at") as csvfile:
510 cw = csv.writer(csvfile, harness.fieldnames, lineterminator=os.linesep)
511 cw.writerow(harness.fieldnames)
512 for instance in harness.recording:
513 cw.writerow(instance)
514
Anas Nashifdf7ee612018-07-07 06:09:01 -0500515class BinaryHandler(Handler):
Anas Nashifd18ec532019-04-11 23:20:39 -0400516 def __init__(self, instance, type_str):
Anas Nashifdf7ee612018-07-07 06:09:01 -0500517 """Constructor
518
519 @param instance Test Instance
520 """
Anas Nashifd18ec532019-04-11 23:20:39 -0400521 super().__init__(instance, type_str)
Anas Nashifdf7ee612018-07-07 06:09:01 -0500522
Alberto Escolar Piedras661bc822018-12-11 15:13:05 +0100523 self.terminated = False
Anas Nashifdf7ee612018-07-07 06:09:01 -0500524
Jan Kowalewski265895b2019-01-07 16:40:24 +0100525 def try_kill_process_by_pid(self):
Anas Nashif83fc06a2019-06-22 11:04:10 -0400526 if self.pid_fn:
Jan Kowalewski265895b2019-01-07 16:40:24 +0100527 pid = int(open(self.pid_fn).read())
528 os.unlink(self.pid_fn)
529 self.pid_fn = None # clear so we don't try to kill the binary twice
530 try:
531 os.kill(pid, signal.SIGTERM)
532 except ProcessLookupError:
533 pass
534
Anas Nashifdf7ee612018-07-07 06:09:01 -0500535 def _output_reader(self, proc, harness):
536 log_out_fp = open(self.log, "wt")
537 for line in iter(proc.stdout.readline, b''):
538 verbose("OUTPUT: {0}".format(line.decode('utf-8').rstrip()))
539 log_out_fp.write(line.decode('utf-8'))
540 log_out_fp.flush()
541 harness.handle(line.decode('utf-8').rstrip())
542 if harness.state:
Alberto Escolar Piedras661bc822018-12-11 15:13:05 +0100543 try:
544 #POSIX arch based ztests end on their own,
545 #so let's give it up to 100ms to do so
546 proc.wait(0.1)
547 except subprocess.TimeoutExpired:
548 proc.terminate()
549 self.terminated = True
Anas Nashifdf7ee612018-07-07 06:09:01 -0500550 break
551
552 log_out_fp.close()
553
554 def handle(self):
Anas Nashifdf7ee612018-07-07 06:09:01 -0500555
Anas Nashif83fc06a2019-06-22 11:04:10 -0400556 harness_name = self.instance.testcase.harness.capitalize()
Anas Nashifdf7ee612018-07-07 06:09:01 -0500557 harness_import = HarnessImporter(harness_name)
558 harness = harness_import.instance
559 harness.configure(self.instance)
560
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500561 if self.call_make_run:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400562 command = [get_generator()[0], "run"]
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500563 else:
564 command = [self.binary]
Anas Nashifdf7ee612018-07-07 06:09:01 -0500565
Anas Nashifc1ea4522019-10-11 07:32:45 -0700566 run_valgrind = False
567 if options.enable_valgrind and shutil.which("valgrind"):
Anas Nashifdf7ee612018-07-07 06:09:01 -0500568 command = ["valgrind", "--error-exitcode=2",
Alberto Escolar Piedras3980e0c2018-12-12 17:41:04 +0100569 "--leak-check=full",
570 "--suppressions="+ZEPHYR_BASE+"/scripts/valgrind.supp",
Anas Nashif83fc06a2019-06-22 11:04:10 -0400571 "--log-file="+self.build_dir+"/valgrind.log"
Alberto Escolar Piedras3980e0c2018-12-12 17:41:04 +0100572 ] + command
Anas Nashifc1ea4522019-10-11 07:32:45 -0700573 run_valgrind = True
Anas Nashifdf7ee612018-07-07 06:09:01 -0500574
Marc Herbertaf1090c2019-04-30 14:11:29 -0700575 verbose("Spawning process: " +
Jan Van Winkel6b9e1602019-01-03 19:18:35 +0100576 " ".join(shlex.quote(word) for word in command) + os.linesep +
Anas Nashif83fc06a2019-06-22 11:04:10 -0400577 "Spawning process in directory: " + self.build_dir)
Alberto Escolar Piedras649368c2019-07-02 09:59:35 +0200578
Anas Nashif83fc06a2019-06-22 11:04:10 -0400579 start_time = time.time()
Alberto Escolar Piedras649368c2019-07-02 09:59:35 +0200580
Anas Nashif89c83042019-11-05 05:55:39 -0800581 env = os.environ.copy()
Jan Van Winkel21212f32019-09-12 00:03:35 +0200582 if options.enable_asan:
Jan Van Winkel21212f32019-09-12 00:03:35 +0200583 env["ASAN_OPTIONS"] = "log_path=stdout:" + \
584 env.get("ASAN_OPTIONS", "")
585 if not options.enable_lsan:
586 env["ASAN_OPTIONS"] += "detect_leaks=0"
587 with subprocess.Popen(command, stdout=subprocess.PIPE,
588 stderr=subprocess.PIPE, cwd=self.build_dir, env=env) as proc:
Marc Herbertaf1090c2019-04-30 14:11:29 -0700589 verbose("Spawning BinaryHandler Thread for %s" % self.name)
Flavio Ceolin063ab902019-10-15 16:10:49 -0700590 t = threading.Thread(target=self._output_reader, args=(proc, harness, ), daemon=True)
Anas Nashifdf7ee612018-07-07 06:09:01 -0500591 t.start()
592 t.join(self.timeout)
593 if t.is_alive():
Jan Kowalewski265895b2019-01-07 16:40:24 +0100594 self.try_kill_process_by_pid()
Anas Nashifdf7ee612018-07-07 06:09:01 -0500595 proc.terminate()
Alberto Escolar Piedras661bc822018-12-11 15:13:05 +0100596 self.terminated = True
Anas Nashifdf7ee612018-07-07 06:09:01 -0500597 t.join()
Anas Nashifdf7ee612018-07-07 06:09:01 -0500598 proc.wait()
599 self.returncode = proc.returncode
Anas Nashifdf7ee612018-07-07 06:09:01 -0500600
Anas Nashif83fc06a2019-06-22 11:04:10 -0400601 handler_time = time.time() - start_time
Alberto Escolar Piedras649368c2019-07-02 09:59:35 +0200602
Anas Nashifdf7ee612018-07-07 06:09:01 -0500603 if options.enable_coverage:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400604 subprocess.call(["GCOV_PREFIX=" + self.build_dir,
605 "gcov", self.sourcedir, "-b", "-s", self.build_dir], shell=True)
Anas Nashifdf7ee612018-07-07 06:09:01 -0500606
Jan Kowalewski265895b2019-01-07 16:40:24 +0100607 self.try_kill_process_by_pid()
608
Anas Nashif83fc06a2019-06-22 11:04:10 -0400609 # FIXME: This is needed when killing the simulator, the console is
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500610 # garbled and needs to be reset. Did not find a better way to do that.
611
612 subprocess.call(["stty", "sane"])
Anas Nashifdf7ee612018-07-07 06:09:01 -0500613 self.instance.results = harness.tests
Anas Nashifc1ea4522019-10-11 07:32:45 -0700614
Anas Nashif83fc06a2019-06-22 11:04:10 -0400615 if not self.terminated and self.returncode != 0:
Alberto Escolar Piedras661bc822018-12-11 15:13:05 +0100616 #When a process is killed, the default handler returns 128 + SIGTERM
617 #so in that case the return code itself is not meaningful
Anas Nashiff72d1902019-10-18 17:20:44 -0400618 self.set_state("failed", handler_time)
Anas Nashifc1ea4522019-10-11 07:32:45 -0700619 self.instance.reason = "Handler Error"
620 elif run_valgrind and self.returncode == 2:
621 self.set_state("failed", handler_time)
622 self.instance.reason = "Valgrind error"
Alberto Escolar Piedras661bc822018-12-11 15:13:05 +0100623 elif harness.state:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400624 self.set_state(harness.state, handler_time)
Anas Nashifdf7ee612018-07-07 06:09:01 -0500625 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400626 self.set_state("timeout", handler_time)
627 self.instance.reason = "Handler timeout"
628
Anas Nashif83fc06a2019-06-22 11:04:10 -0400629 self.record(harness)
Anas Nashif73440ea2018-02-19 10:57:03 -0600630
631class DeviceHandler(Handler):
632
Anas Nashifd18ec532019-04-11 23:20:39 -0400633 def __init__(self, instance, type_str):
Anas Nashif73440ea2018-02-19 10:57:03 -0600634 """Constructor
635
636 @param instance Test Instance
637 """
Anas Nashifd18ec532019-04-11 23:20:39 -0400638 super().__init__(instance, type_str)
Anas Nashif73440ea2018-02-19 10:57:03 -0600639
Anas Nashif83fc06a2019-06-22 11:04:10 -0400640 self.suite = None
641
Marti Bolivar5591ca22019-02-07 15:53:39 -0700642 def monitor_serial(self, ser, halt_fileno, harness):
Anas Nashif4a9f3e62018-07-06 18:58:18 -0500643 log_out_fp = open(self.log, "wt")
Anas Nashif73440ea2018-02-19 10:57:03 -0600644
Marti Bolivar5591ca22019-02-07 15:53:39 -0700645 ser_fileno = ser.fileno()
646 readlist = [halt_fileno, ser_fileno]
647
Anas Nashif73440ea2018-02-19 10:57:03 -0600648 while ser.isOpen():
Marti Bolivar5591ca22019-02-07 15:53:39 -0700649 readable, _, _ = select.select(readlist, [], [], self.timeout)
650
651 if halt_fileno in readable:
652 verbose('halted')
653 ser.close()
654 break
655 if ser_fileno not in readable:
656 continue # Timeout.
657
658 serial_line = None
Anas Nashif61e21632018-04-08 13:30:16 -0500659 try:
660 serial_line = ser.readline()
661 except TypeError:
662 pass
Anas Nashif83fc06a2019-06-22 11:04:10 -0400663 except serial.SerialException:
Anas Nashif59237602019-03-03 10:36:35 -0500664 ser.close()
665 break
Anas Nashif61e21632018-04-08 13:30:16 -0500666
Marti Bolivar5591ca22019-02-07 15:53:39 -0700667 # Just because ser_fileno has data doesn't mean an entire line
668 # is available yet.
Anas Nashif73440ea2018-02-19 10:57:03 -0600669 if serial_line:
Anas Nashifd3384fb2018-02-22 06:44:16 -0600670 sl = serial_line.decode('utf-8', 'ignore')
671 verbose("DEVICE: {0}".format(sl.rstrip()))
672
673 log_out_fp.write(sl)
674 log_out_fp.flush()
675 harness.handle(sl.rstrip())
Marti Bolivar5591ca22019-02-07 15:53:39 -0700676
Anas Nashif73440ea2018-02-19 10:57:03 -0600677 if harness.state:
678 ser.close()
679 break
680
681 log_out_fp.close()
682
Anas Nashif83fc06a2019-06-22 11:04:10 -0400683 def device_is_available(self, device):
684 for i in self.suite.connected_hardware:
685 if i['platform'] == device and i['available'] and i['connected']:
686 return True
687
688 return False
689
690 def get_available_device(self, device):
691 for i in self.suite.connected_hardware:
692 if i['platform'] == device and i['available']:
693 i['available'] = False
694 i['counter'] += 1
695 return i
696
697 return None
698
699 def make_device_available(self, serial):
700 with hw_map_local:
701 for i in self.suite.connected_hardware:
702 if i['serial'] == serial:
703 i['available'] = True
704
Anas Nashif73440ea2018-02-19 10:57:03 -0600705 def handle(self):
706 out_state = "failed"
707
Anas Nashif83fc06a2019-06-22 11:04:10 -0400708 if options.west_flash:
709 command = ["west", "flash", "--skip-rebuild", "-d", self.build_dir]
710 if options.west_runner:
Michael Scott421ce462019-06-18 09:37:46 -0700711 command.append("--runner")
712 command.append(options.west_runner)
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +0200713 # There are three ways this option is used.
Andy Doan79c48842019-02-08 10:09:04 -0600714 # 1) bare: --west-flash
715 # This results in options.west_flash == []
716 # 2) with a value: --west-flash="--board-id=42"
717 # This results in options.west_flash == "--board-id=42"
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +0200718 # 3) Multiple values: --west-flash="--board-id=42,--erase"
719 # This results in options.west_flash == "--board-id=42 --erase"
Andy Doan79c48842019-02-08 10:09:04 -0600720 if options.west_flash != []:
721 command.append('--')
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +0200722 command.extend(options.west_flash.split(','))
Anas Nashifd3384fb2018-02-22 06:44:16 -0600723 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400724 command = [get_generator()[0], "-C", self.build_dir, "flash"]
Anas Nashifd3384fb2018-02-22 06:44:16 -0600725
Anas Nashifd3384fb2018-02-22 06:44:16 -0600726
Anas Nashif83fc06a2019-06-22 11:04:10 -0400727 while not self.device_is_available(self.instance.platform.name):
728 time.sleep(1)
729
730 hardware = self.get_available_device(self.instance.platform.name)
731
732 runner = hardware.get('runner', None)
733 if runner:
Peter Bigotda738482019-11-21 11:55:26 -0600734 board_id = hardware.get("probe_id", hardware.get("id", None))
Anas Nashif83fc06a2019-06-22 11:04:10 -0400735 product = hardware.get("product", None)
736 command = ["west", "flash", "--skip-rebuild", "-d", self.build_dir]
737 command.append("--runner")
738 command.append(hardware.get('runner', None))
739 if runner == "pyocd":
740 command.append("--board-id")
741 command.append(board_id)
742 elif runner == "nrfjprog":
743 command.append('--')
744 command.append("--snr")
745 command.append(board_id)
746 elif runner == "openocd" and product == "STM32 STLink":
747 command.append('--')
748 command.append("--cmd-pre-init")
749 command.append("hla_serial %s" %(board_id))
750 elif runner == "openocd" and product == "EDBG CMSIS-DAP":
751 command.append('--')
752 command.append("--cmd-pre-init")
753 command.append("cmsis_dap_serial %s" %(board_id))
754 elif runner == "jlink":
755 command.append("--tool-opt=-SelectEmuBySN %s" %(board_id))
756
757 serial_device = hardware['serial']
758
759 try:
760 ser = serial.Serial(
761 serial_device,
762 baudrate=115200,
763 parity=serial.PARITY_NONE,
764 stopbits=serial.STOPBITS_ONE,
765 bytesize=serial.EIGHTBITS,
766 timeout=self.timeout
767 )
768 except serial.SerialException as e:
769 self.set_state("failed", 0)
770 error("Serial device err: %s" %(str(e)))
771 self.make_device_available(serial_device)
772 return
Anas Nashif73440ea2018-02-19 10:57:03 -0600773
774 ser.flush()
775
Anas Nashif83fc06a2019-06-22 11:04:10 -0400776 harness_name = self.instance.testcase.harness.capitalize()
Anas Nashif73440ea2018-02-19 10:57:03 -0600777 harness_import = HarnessImporter(harness_name)
778 harness = harness_import.instance
779 harness.configure(self.instance)
Anas Nashif83fc06a2019-06-22 11:04:10 -0400780 read_pipe, write_pipe = os.pipe()
781 start_time = time.time()
Anas Nashif73440ea2018-02-19 10:57:03 -0600782
Marti Bolivar5591ca22019-02-07 15:53:39 -0700783 t = threading.Thread(target=self.monitor_serial, daemon=True,
Anas Nashif83fc06a2019-06-22 11:04:10 -0400784 args=(ser, read_pipe, harness))
Anas Nashif73440ea2018-02-19 10:57:03 -0600785 t.start()
786
Andy Doan79c48842019-02-08 10:09:04 -0600787 logging.debug('Flash command: %s', command)
Anas Nashif61e21632018-04-08 13:30:16 -0500788 try:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400789 if VERBOSE and not runner:
Marti Bolivar303b5222019-02-07 15:50:55 -0700790 subprocess.check_call(command)
791 else:
792 subprocess.check_output(command, stderr=subprocess.PIPE)
Anas Nashif83fc06a2019-06-22 11:04:10 -0400793
Anas Nashif61e21632018-04-08 13:30:16 -0500794 except subprocess.CalledProcessError:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400795 os.write(write_pipe, b'x') # halt the thread
Anas Nashif73440ea2018-02-19 10:57:03 -0600796
797 t.join(self.timeout)
798 if t.is_alive():
799 out_state = "timeout"
Anas Nashif73440ea2018-02-19 10:57:03 -0600800
801 if ser.isOpen():
802 ser.close()
803
Anas Nashifd3384fb2018-02-22 06:44:16 -0600804 if out_state == "timeout":
Anas Nashif83fc06a2019-06-22 11:04:10 -0400805 for c in self.instance.testcase.cases:
Anas Nashifd3384fb2018-02-22 06:44:16 -0600806 if c not in harness.tests:
807 harness.tests[c] = "BLOCK"
Anas Nashif61e21632018-04-08 13:30:16 -0500808
Anas Nashif83fc06a2019-06-22 11:04:10 -0400809 handler_time = time.time() - start_time
810
Anas Nashif61e21632018-04-08 13:30:16 -0500811 self.instance.results = harness.tests
Anas Nashif73440ea2018-02-19 10:57:03 -0600812 if harness.state:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400813 self.set_state(harness.state, handler_time)
Anas Nashif73440ea2018-02-19 10:57:03 -0600814 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400815 self.set_state(out_state, handler_time)
Anas Nashif73440ea2018-02-19 10:57:03 -0600816
Anas Nashif83fc06a2019-06-22 11:04:10 -0400817 self.make_device_available(serial_device)
Anas Nashif3ba1d432017-12-05 15:28:44 -0500818
Anas Nashif3a00d0c2019-11-12 11:07:15 -0500819 self.record(harness)
820
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300821class QEMUHandler(Handler):
Andrew Boie6acbe632015-07-17 12:03:52 -0700822 """Spawns a thread to monitor QEMU output from pipes
823
Anas Nashif3ac7b3a2017-08-03 10:03:02 -0400824 We pass QEMU_PIPE to 'make run' and monitor the pipes for output.
Andrew Boie6acbe632015-07-17 12:03:52 -0700825 We need to do this as once qemu starts, it runs forever until killed.
826 Test cases emit special messages to the console as they run, we check
827 for these to collect whether the test passed or failed.
828 """
Andrew Boie6acbe632015-07-17 12:03:52 -0700829
Anas Nashif83fc06a2019-06-22 11:04:10 -0400830
831 def __init__(self, instance, type_str):
832 """Constructor
833
834 @param instance Test instance
835 """
836
837 super().__init__(instance, type_str)
838 self.fifo_fn = os.path.join(instance.build_dir, "qemu-fifo")
839
840 self.pid_fn = os.path.join(instance.build_dir, "qemu.pid")
841
842
Andrew Boie6acbe632015-07-17 12:03:52 -0700843 @staticmethod
Anas Nashif576be982017-12-23 20:20:27 -0500844 def _thread(handler, timeout, outdir, logfile, fifo_fn, pid_fn, results, harness):
Andrew Boie6acbe632015-07-17 12:03:52 -0700845 fifo_in = fifo_fn + ".in"
846 fifo_out = fifo_fn + ".out"
847
848 # These in/out nodes are named from QEMU's perspective, not ours
849 if os.path.exists(fifo_in):
850 os.unlink(fifo_in)
851 os.mkfifo(fifo_in)
852 if os.path.exists(fifo_out):
853 os.unlink(fifo_out)
854 os.mkfifo(fifo_out)
855
856 # We don't do anything with out_fp but we need to open it for
857 # writing so that QEMU doesn't block, due to the way pipes work
858 out_fp = open(fifo_in, "wb")
859 # Disable internal buffering, we don't
860 # want read() or poll() to ever block if there is data in there
861 in_fp = open(fifo_out, "rb", buffering=0)
Andrew Boie08ce5a52016-02-22 13:28:10 -0800862 log_out_fp = open(logfile, "wt")
Andrew Boie6acbe632015-07-17 12:03:52 -0700863
864 start_time = time.time()
865 timeout_time = start_time + timeout
866 p = select.poll()
867 p.register(in_fp, select.POLLIN)
Anas Nashif39ae72b2018-08-29 01:45:38 -0400868 out_state = None
Andrew Boie6acbe632015-07-17 12:03:52 -0700869
Andrew Boie6acbe632015-07-17 12:03:52 -0700870 line = ""
Anas Nashif74dbe332019-01-17 17:11:37 -0500871 timeout_extended = False
Andrew Boie6acbe632015-07-17 12:03:52 -0700872 while True:
873 this_timeout = int((timeout_time - time.time()) * 1000)
874 if this_timeout < 0 or not p.poll(this_timeout):
Anas Nashif39ae72b2018-08-29 01:45:38 -0400875 if not out_state:
876 out_state = "timeout"
Andrew Boie6acbe632015-07-17 12:03:52 -0700877 break
878
Genaro Saucedo Tejada2968b362016-08-09 15:11:53 -0500879 try:
880 c = in_fp.read(1).decode("utf-8")
881 except UnicodeDecodeError:
882 # Test is writing something weird, fail
883 out_state = "unexpected byte"
884 break
885
Andrew Boie6acbe632015-07-17 12:03:52 -0700886 if c == "":
887 # EOF, this shouldn't happen unless QEMU crashes
888 out_state = "unexpected eof"
889 break
890 line = line + c
891 if c != "\n":
892 continue
893
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300894 # line contains a full line of data output from QEMU
Andrew Boie6acbe632015-07-17 12:03:52 -0700895 log_out_fp.write(line)
896 log_out_fp.flush()
897 line = line.strip()
898 verbose("QEMU: %s" % line)
899
Anas Nashif576be982017-12-23 20:20:27 -0500900 harness.handle(line)
901 if harness.state:
Anas Nashif39ae72b2018-08-29 01:45:38 -0400902 # if we have registered a fail make sure the state is not
903 # overridden by a false success message coming from the
904 # testsuite
905 if out_state != 'failed':
906 out_state = harness.state
907
908 # if we get some state, that means test is doing well, we reset
Marc Herbert9e573382019-07-03 12:49:42 -0700909 # the timeout and wait for 2 more seconds to catch anything
910 # printed late. We wait much longer if code
Andrew Boie095b82a2019-07-02 17:03:48 -0700911 # coverage is enabled since dumping this information can
912 # take some time.
Anas Nashiff29087e2019-01-25 09:37:38 -0500913 if not timeout_extended or harness.capture_coverage:
914 timeout_extended= True
915 if harness.capture_coverage:
Andrew Boied62e2292019-07-01 15:01:48 -0700916 timeout_time = time.time() + 30
Anas Nashiff29087e2019-01-25 09:37:38 -0500917 else:
Anas Nashif74dbe332019-01-17 17:11:37 -0500918 timeout_time = time.time() + 2
Andrew Boie6acbe632015-07-17 12:03:52 -0700919 line = ""
920
Anas Nashif83fc06a2019-06-22 11:04:10 -0400921 handler.record(harness)
922
923 handler_time = time.time() - start_time
Andrew Boie6acbe632015-07-17 12:03:52 -0700924 verbose("QEMU complete (%s) after %f seconds" %
Anas Nashif83fc06a2019-06-22 11:04:10 -0400925 (out_state, handler_time))
926 handler.set_state(out_state, handler_time)
Andrew Boie6acbe632015-07-17 12:03:52 -0700927
928 log_out_fp.close()
929 out_fp.close()
930 in_fp.close()
Anas Nashifd6476ee2019-04-11 11:40:09 -0400931 if os.path.exists(pid_fn):
932 pid = int(open(pid_fn).read())
933 os.unlink(pid_fn)
Andrew Boie6acbe632015-07-17 12:03:52 -0700934
Anas Nashifd6476ee2019-04-11 11:40:09 -0400935 try:
936 if pid:
937 os.kill(pid, signal.SIGTERM)
938 except ProcessLookupError:
939 # Oh well, as long as it's dead! User probably sent Ctrl-C
940 pass
Andrew Boie08ce5a52016-02-22 13:28:10 -0800941
Andrew Boie6acbe632015-07-17 12:03:52 -0700942 os.unlink(fifo_in)
943 os.unlink(fifo_out)
944
Anas Nashif83fc06a2019-06-22 11:04:10 -0400945 def handle(self):
Andrew Boie6acbe632015-07-17 12:03:52 -0700946 self.results = {}
Anas Nashif13773752018-07-06 18:20:23 -0500947 self.run = True
Andrew Boie6acbe632015-07-17 12:03:52 -0700948
949 # We pass this to QEMU which looks for fifos with .in and .out
950 # suffixes.
Anas Nashif83fc06a2019-06-22 11:04:10 -0400951 self.fifo_fn = os.path.join(self.instance.build_dir, "qemu-fifo")
Andrew Boie6acbe632015-07-17 12:03:52 -0700952
Anas Nashif83fc06a2019-06-22 11:04:10 -0400953 self.pid_fn = os.path.join(self.instance.build_dir, "qemu.pid")
Andrew Boie6acbe632015-07-17 12:03:52 -0700954 if os.path.exists(self.pid_fn):
955 os.unlink(self.pid_fn)
956
Anas Nashif4a9f3e62018-07-06 18:58:18 -0500957 self.log_fn = self.log
Anas Nashif576be982017-12-23 20:20:27 -0500958
Anas Nashif83fc06a2019-06-22 11:04:10 -0400959 harness_import = HarnessImporter(self.instance.testcase.harness.capitalize())
Anas Nashif576be982017-12-23 20:20:27 -0500960 harness = harness_import.instance
Anas Nashifd3384fb2018-02-22 06:44:16 -0600961 harness.configure(self.instance)
962 self.thread = threading.Thread(name=self.name, target=QEMUHandler._thread,
Anas Nashif83fc06a2019-06-22 11:04:10 -0400963 args=(self, self.timeout, self.build_dir,
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300964 self.log_fn, self.fifo_fn,
Anas Nashif576be982017-12-23 20:20:27 -0500965 self.pid_fn, self.results, harness))
Anas Nashife0a6a0b2018-02-15 20:07:24 -0600966
967 self.instance.results = harness.tests
Andrew Boie6acbe632015-07-17 12:03:52 -0700968 self.thread.daemon = True
Anas Nashif83fc06a2019-06-22 11:04:10 -0400969 verbose("Spawning QEMUHandler Thread for %s" % self.name)
Andrew Boie6acbe632015-07-17 12:03:52 -0700970 self.thread.start()
Anas Nashif83fc06a2019-06-22 11:04:10 -0400971 subprocess.call(["stty", "sane"])
Andrew Boie6acbe632015-07-17 12:03:52 -0700972
Stephanos Ioannidise1827c02019-11-12 14:11:31 +0900973 verbose("Running %s (%s)" %(self.name, self.type_str))
974 command = [get_generator()[0]]
975 command += ["-C", self.build_dir, "run"]
976
977 with subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=self.build_dir) as proc:
978 verbose("Spawning QEMUHandler Thread for %s" % self.name)
979 proc.wait()
980 self.returncode = proc.returncode
981
982 if self.returncode != 0:
983 self.set_state("failed", 0)
984 self.instance.reason = "Exited with {}".format(self.returncode)
985
Andrew Boie6acbe632015-07-17 12:03:52 -0700986 def get_fifo(self):
987 return self.fifo_fn
988
Andrew Boie6acbe632015-07-17 12:03:52 -0700989class SizeCalculator:
Andrew Boie73b4ee62015-10-07 11:33:22 -0700990
Anas Nashif83fc06a2019-06-22 11:04:10 -0400991 alloc_sections = [
992 "bss",
993 "noinit",
994 "app_bss",
995 "app_noinit",
996 "ccm_bss",
997 "ccm_noinit"
998 ]
999
1000 rw_sections = [
1001 "datas",
1002 "initlevel",
1003 "exceptions",
1004 "initshell",
1005 "_static_thread_area",
1006 "_k_timer_area",
1007 "_k_mem_slab_area",
1008 "_k_mem_pool_area",
1009 "sw_isr_table",
1010 "_k_sem_area",
1011 "_k_mutex_area",
1012 "app_shmem_regions",
1013 "_k_fifo_area",
1014 "_k_lifo_area",
1015 "_k_stack_area",
1016 "_k_msgq_area",
1017 "_k_mbox_area",
1018 "_k_pipe_area",
1019 "net_if",
1020 "net_if_dev",
1021 "net_stack",
1022 "net_l2_data",
1023 "_k_queue_area",
1024 "_net_buf_pool_area",
1025 "app_datas",
1026 "kobject_data",
1027 "mmu_tables",
1028 "app_pad",
1029 "priv_stacks",
1030 "ccm_data",
1031 "usb_descriptor",
1032 "usb_data", "usb_bos_desc",
1033 'log_backends_sections',
1034 'log_dynamic_sections',
1035 'log_const_sections',
1036 "app_smem",
1037 'shell_root_cmds_sections',
1038 'log_const_sections',
1039 "font_entry_sections",
1040 "priv_stacks_noinit",
1041 "_TEXT_SECTION_NAME_2",
1042 "_GCOV_BSS_SECTION_NAME",
1043 "gcov",
1044 "nocache"
1045 ]
Anas Nashifdb9592a2018-10-08 10:19:41 -04001046
Andrew Boie73b4ee62015-10-07 11:33:22 -07001047 # These get copied into RAM only on non-XIP
Anas Nashif83fc06a2019-06-22 11:04:10 -04001048 ro_sections = [
1049 "text",
1050 "ctors",
1051 "init_array",
1052 "reset",
1053 "object_access",
1054 "rodata",
1055 "devconfig",
1056 "net_l2",
1057 "vector",
1058 "sw_isr_table",
1059 "_settings_handlers_area",
1060 "_bt_channels_area",
1061 "_bt_br_channels_area",
1062 "_bt_services_area",
1063 "vectors",
1064 "net_socket_register",
1065 "net_ppp_proto"
1066 ]
Andrew Boie73b4ee62015-10-07 11:33:22 -07001067
Andrew Boie52fef672016-11-29 12:21:59 -08001068 def __init__(self, filename, extra_sections):
Andrew Boie6acbe632015-07-17 12:03:52 -07001069 """Constructor
1070
Andrew Boiebbd670c2015-08-17 13:16:11 -07001071 @param filename Path to the output binary
1072 The <filename> is parsed by objdump to determine section sizes
Andrew Boie6acbe632015-07-17 12:03:52 -07001073 """
Andrew Boie6acbe632015-07-17 12:03:52 -07001074 # Make sure this is an ELF binary
Andrew Boiebbd670c2015-08-17 13:16:11 -07001075 with open(filename, "rb") as f:
Andrew Boie6acbe632015-07-17 12:03:52 -07001076 magic = f.read(4)
1077
Anas Nashifb4bdd662018-08-15 17:12:28 -05001078 try:
Anas Nashif83fc06a2019-06-22 11:04:10 -04001079 if magic != b'\x7fELF':
Anas Nashifb4bdd662018-08-15 17:12:28 -05001080 raise SanityRuntimeError("%s is not an ELF binary" % filename)
1081 except Exception as e:
1082 print(str(e))
1083 sys.exit(2)
Andrew Boie6acbe632015-07-17 12:03:52 -07001084
1085 # Search for CONFIG_XIP in the ELF's list of symbols using NM and AWK.
Anas Nashif3ba1d432017-12-05 15:28:44 -05001086 # GREP can not be used as it returns an error if the symbol is not
1087 # found.
1088 is_xip_command = "nm " + filename + \
1089 " | awk '/CONFIG_XIP/ { print $3 }'"
1090 is_xip_output = subprocess.check_output(
1091 is_xip_command, shell=True, stderr=subprocess.STDOUT).decode(
1092 "utf-8").strip()
Anas Nashifb4bdd662018-08-15 17:12:28 -05001093 try:
1094 if is_xip_output.endswith("no symbols"):
1095 raise SanityRuntimeError("%s has no symbol information" % filename)
1096 except Exception as e:
1097 print(str(e))
1098 sys.exit(2)
1099
Andrew Boie6acbe632015-07-17 12:03:52 -07001100 self.is_xip = (len(is_xip_output) != 0)
1101
Andrew Boiebbd670c2015-08-17 13:16:11 -07001102 self.filename = filename
Andrew Boie73b4ee62015-10-07 11:33:22 -07001103 self.sections = []
1104 self.rom_size = 0
Andrew Boie6acbe632015-07-17 12:03:52 -07001105 self.ram_size = 0
Andrew Boie52fef672016-11-29 12:21:59 -08001106 self.extra_sections = extra_sections
Andrew Boie6acbe632015-07-17 12:03:52 -07001107
1108 self._calculate_sizes()
1109
1110 def get_ram_size(self):
1111 """Get the amount of RAM the application will use up on the device
1112
1113 @return amount of RAM, in bytes
1114 """
Andrew Boie73b4ee62015-10-07 11:33:22 -07001115 return self.ram_size
Andrew Boie6acbe632015-07-17 12:03:52 -07001116
1117 def get_rom_size(self):
1118 """Get the size of the data that this application uses on device's flash
1119
1120 @return amount of ROM, in bytes
1121 """
Andrew Boie73b4ee62015-10-07 11:33:22 -07001122 return self.rom_size
Andrew Boie6acbe632015-07-17 12:03:52 -07001123
1124 def unrecognized_sections(self):
1125 """Get a list of sections inside the binary that weren't recognized
1126
David B. Kinder29963c32017-06-16 12:32:42 -07001127 @return list of unrecognized section names
Andrew Boie6acbe632015-07-17 12:03:52 -07001128 """
1129 slist = []
Andrew Boie73b4ee62015-10-07 11:33:22 -07001130 for v in self.sections:
Andrew Boie6acbe632015-07-17 12:03:52 -07001131 if not v["recognized"]:
Andrew Boie73b4ee62015-10-07 11:33:22 -07001132 slist.append(v["name"])
Andrew Boie6acbe632015-07-17 12:03:52 -07001133 return slist
1134
1135 def _calculate_sizes(self):
1136 """ Calculate RAM and ROM usage by section """
Andrew Boiebbd670c2015-08-17 13:16:11 -07001137 objdump_command = "objdump -h " + self.filename
Anas Nashif3ba1d432017-12-05 15:28:44 -05001138 objdump_output = subprocess.check_output(
1139 objdump_command, shell=True).decode("utf-8").splitlines()
Andrew Boie6acbe632015-07-17 12:03:52 -07001140
1141 for line in objdump_output:
1142 words = line.split()
1143
Anas Nashif83fc06a2019-06-22 11:04:10 -04001144 if not words: # Skip lines that are too short
Andrew Boie6acbe632015-07-17 12:03:52 -07001145 continue
1146
1147 index = words[0]
Anas Nashif83fc06a2019-06-22 11:04:10 -04001148 if not index[0].isdigit(): # Skip lines that do not start
Andrew Boie6acbe632015-07-17 12:03:52 -07001149 continue # with a digit
1150
1151 name = words[1] # Skip lines with section names
Anas Nashif83fc06a2019-06-22 11:04:10 -04001152 if name[0] == '.': # starting with '.'
Andrew Boie6acbe632015-07-17 12:03:52 -07001153 continue
1154
Andrew Boie73b4ee62015-10-07 11:33:22 -07001155 # TODO this doesn't actually reflect the size in flash or RAM as
1156 # it doesn't include linker-imposed padding between sections.
1157 # It is close though.
Andrew Boie6acbe632015-07-17 12:03:52 -07001158 size = int(words[2], 16)
Andrew Boie9882dcd2015-10-07 14:25:51 -07001159 if size == 0:
1160 continue
1161
Andrew Boie73b4ee62015-10-07 11:33:22 -07001162 load_addr = int(words[4], 16)
1163 virt_addr = int(words[3], 16)
Andrew Boie6acbe632015-07-17 12:03:52 -07001164
1165 # Add section to memory use totals (for both non-XIP and XIP scenarios)
Andrew Boie6acbe632015-07-17 12:03:52 -07001166 # Unrecognized section names are not included in the calculations.
Andrew Boie6acbe632015-07-17 12:03:52 -07001167 recognized = True
Andrew Boie73b4ee62015-10-07 11:33:22 -07001168 if name in SizeCalculator.alloc_sections:
1169 self.ram_size += size
1170 stype = "alloc"
1171 elif name in SizeCalculator.rw_sections:
1172 self.ram_size += size
1173 self.rom_size += size
1174 stype = "rw"
1175 elif name in SizeCalculator.ro_sections:
1176 self.rom_size += size
1177 if not self.is_xip:
1178 self.ram_size += size
1179 stype = "ro"
Andrew Boie6acbe632015-07-17 12:03:52 -07001180 else:
Andrew Boie73b4ee62015-10-07 11:33:22 -07001181 stype = "unknown"
Andrew Boie52fef672016-11-29 12:21:59 -08001182 if name not in self.extra_sections:
1183 recognized = False
Andrew Boie6acbe632015-07-17 12:03:52 -07001184
Anas Nashif3ba1d432017-12-05 15:28:44 -05001185 self.sections.append({"name": name, "load_addr": load_addr,
1186 "size": size, "virt_addr": virt_addr,
1187 "type": stype, "recognized": recognized})
Andrew Boie6acbe632015-07-17 12:03:52 -07001188
1189
Andrew Boie6acbe632015-07-17 12:03:52 -07001190# "list" - List of strings
1191# "list:<type>" - List of <type>
1192# "set" - Set of unordered, unique strings
1193# "set:<type>" - Set of <type>
1194# "float" - Floating point
1195# "int" - Integer
1196# "bool" - Boolean
1197# "str" - String
1198
1199# XXX Be sure to update __doc__ if you change any of this!!
1200
Anas Nashif83fc06a2019-06-22 11:04:10 -04001201platform_valid_keys = {
Anas Nashif924a4e72018-10-18 12:25:55 -04001202 "supported_toolchains": {"type": "list", "default": []},
1203 "env": {"type": "list", "default": []}
1204 }
Andrew Boie6acbe632015-07-17 12:03:52 -07001205
Anas Nashif3ba1d432017-12-05 15:28:44 -05001206testcase_valid_keys = {"tags": {"type": "set", "required": False},
1207 "type": {"type": "str", "default": "integration"},
1208 "extra_args": {"type": "list"},
1209 "extra_configs": {"type": "list"},
1210 "build_only": {"type": "bool", "default": False},
1211 "build_on_all": {"type": "bool", "default": False},
1212 "skip": {"type": "bool", "default": False},
1213 "slow": {"type": "bool", "default": False},
1214 "timeout": {"type": "int", "default": 60},
1215 "min_ram": {"type": "int", "default": 8},
1216 "depends_on": {"type": "set"},
1217 "min_flash": {"type": "int", "default": 32},
1218 "arch_whitelist": {"type": "set"},
1219 "arch_exclude": {"type": "set"},
1220 "extra_sections": {"type": "list", "default": []},
1221 "platform_exclude": {"type": "set"},
1222 "platform_whitelist": {"type": "set"},
1223 "toolchain_exclude": {"type": "set"},
1224 "toolchain_whitelist": {"type": "set"},
Anas Nashifab940162017-12-08 10:17:57 -05001225 "filter": {"type": "str"},
Anas Nashif576be982017-12-23 20:20:27 -05001226 "harness": {"type": "str"},
Praful Swarnakarcf89e282018-09-13 02:58:28 +05301227 "harness_config": {"type": "map", "default": {}}
Anas Nashifab940162017-12-08 10:17:57 -05001228 }
Andrew Boie6acbe632015-07-17 12:03:52 -07001229
Andrew Boie6acbe632015-07-17 12:03:52 -07001230class SanityConfigParser:
Anas Nashifa792a3d2017-04-04 18:47:49 -04001231 """Class to read test case files with semantic checking
Andrew Boie6acbe632015-07-17 12:03:52 -07001232 """
Anas Nashif3ba1d432017-12-05 15:28:44 -05001233
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001234 def __init__(self, filename, schema):
Andrew Boie6acbe632015-07-17 12:03:52 -07001235 """Instantiate a new SanityConfigParser object
1236
Anas Nashifa792a3d2017-04-04 18:47:49 -04001237 @param filename Source .yaml file to read
Andrew Boie6acbe632015-07-17 12:03:52 -07001238 """
Anas Nashif83fc06a2019-06-22 11:04:10 -04001239 self.data = {}
1240 self.schema = schema
Andrew Boie6acbe632015-07-17 12:03:52 -07001241 self.filename = filename
Anas Nashif255625b2017-12-05 15:08:26 -05001242 self.tests = {}
1243 self.common = {}
Anas Nashif83fc06a2019-06-22 11:04:10 -04001244
1245 def load(self):
1246 self.data = scl.yaml_load_verify(self.filename, self.schema)
1247
Anas Nashif255625b2017-12-05 15:08:26 -05001248 if 'tests' in self.data:
1249 self.tests = self.data['tests']
1250 if 'common' in self.data:
1251 self.common = self.data['common']
Andrew Boie6acbe632015-07-17 12:03:52 -07001252
Anas Nashif83fc06a2019-06-22 11:04:10 -04001253
Andrew Boie6acbe632015-07-17 12:03:52 -07001254 def _cast_value(self, value, typestr):
Anas Nashif3ba1d432017-12-05 15:28:44 -05001255 if isinstance(value, str):
Anas Nashifa792a3d2017-04-04 18:47:49 -04001256 v = value.strip()
Andrew Boie6acbe632015-07-17 12:03:52 -07001257 if typestr == "str":
1258 return v
1259
1260 elif typestr == "float":
Anas Nashifa792a3d2017-04-04 18:47:49 -04001261 return float(value)
Andrew Boie6acbe632015-07-17 12:03:52 -07001262
1263 elif typestr == "int":
Anas Nashifa792a3d2017-04-04 18:47:49 -04001264 return int(value)
Andrew Boie6acbe632015-07-17 12:03:52 -07001265
1266 elif typestr == "bool":
Anas Nashifa792a3d2017-04-04 18:47:49 -04001267 return value
Andrew Boie6acbe632015-07-17 12:03:52 -07001268
Anas Nashif3ba1d432017-12-05 15:28:44 -05001269 elif typestr.startswith("list") and isinstance(value, list):
Anas Nashiffa695d22017-10-04 16:14:27 -04001270 return value
Anas Nashif3ba1d432017-12-05 15:28:44 -05001271 elif typestr.startswith("list") and isinstance(value, str):
Andrew Boie6acbe632015-07-17 12:03:52 -07001272 vs = v.split()
1273 if len(typestr) > 4 and typestr[4] == ":":
1274 return [self._cast_value(vsi, typestr[5:]) for vsi in vs]
1275 else:
1276 return vs
1277
1278 elif typestr.startswith("set"):
1279 vs = v.split()
1280 if len(typestr) > 3 and typestr[3] == ":":
Anas Nashif83fc06a2019-06-22 11:04:10 -04001281 return {self._cast_value(vsi, typestr[4:]) for vsi in vs}
Andrew Boie6acbe632015-07-17 12:03:52 -07001282 else:
1283 return set(vs)
1284
Anas Nashif576be982017-12-23 20:20:27 -05001285 elif typestr.startswith("map"):
1286 return value
Andrew Boie6acbe632015-07-17 12:03:52 -07001287 else:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001288 raise ConfigurationError(
1289 self.filename, "unknown type '%s'" % value)
Andrew Boie6acbe632015-07-17 12:03:52 -07001290
Anas Nashifb4754ed2017-12-05 17:27:58 -05001291 def get_test(self, name, valid_keys):
1292 """Get a dictionary representing the keys/values within a test
Andrew Boie6acbe632015-07-17 12:03:52 -07001293
Anas Nashifb4754ed2017-12-05 17:27:58 -05001294 @param name The test in the .yaml file to retrieve data from
Andrew Boie6acbe632015-07-17 12:03:52 -07001295 @param valid_keys A dictionary representing the intended semantics
Anas Nashifb4754ed2017-12-05 17:27:58 -05001296 for this test. Each key in this dictionary is a key that could
Anas Nashifa792a3d2017-04-04 18:47:49 -04001297 be specified, if a key is given in the .yaml file which isn't in
Andrew Boie6acbe632015-07-17 12:03:52 -07001298 here, it will generate an error. Each value in this dictionary
1299 is another dictionary containing metadata:
1300
1301 "default" - Default value if not given
1302 "type" - Data type to convert the text value to. Simple types
1303 supported are "str", "float", "int", "bool" which will get
1304 converted to respective Python data types. "set" and "list"
1305 may also be specified which will split the value by
1306 whitespace (but keep the elements as strings). finally,
1307 "list:<type>" and "set:<type>" may be given which will
1308 perform a type conversion after splitting the value up.
1309 "required" - If true, raise an error if not defined. If false
1310 and "default" isn't specified, a type conversion will be
1311 done on an empty string
Anas Nashifb4754ed2017-12-05 17:27:58 -05001312 @return A dictionary containing the test key-value pairs with
Andrew Boie6acbe632015-07-17 12:03:52 -07001313 type conversion and default values filled in per valid_keys
1314 """
1315
1316 d = {}
Anas Nashif255625b2017-12-05 15:08:26 -05001317 for k, v in self.common.items():
Anas Nashiffa695d22017-10-04 16:14:27 -04001318 d[k] = v
Anas Nashif75547e22018-02-24 08:32:14 -06001319
Anas Nashifb4754ed2017-12-05 17:27:58 -05001320 for k, v in self.tests[name].items():
Andrew Boie6acbe632015-07-17 12:03:52 -07001321 if k not in valid_keys:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001322 raise ConfigurationError(
1323 self.filename,
1324 "Unknown config key '%s' in definition for '%s'" %
Anas Nashifb4754ed2017-12-05 17:27:58 -05001325 (k, name))
Andrew Boie6acbe632015-07-17 12:03:52 -07001326
Anas Nashiffa695d22017-10-04 16:14:27 -04001327 if k in d:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001328 if isinstance(d[k], str):
Paul Sokolovsky8c6d2e22019-08-29 13:20:57 +03001329 # By default, we just concatenate string values of keys
1330 # which appear both in "common" and per-test sections,
1331 # but some keys are handled in adhoc way based on their
1332 # semantics.
1333 if k == "filter":
1334 d[k] = "(%s) and (%s)" % (d[k], v)
1335 else:
1336 d[k] += " " + v
Anas Nashiffa695d22017-10-04 16:14:27 -04001337 else:
1338 d[k] = v
Anas Nashif75547e22018-02-24 08:32:14 -06001339
Andrew Boie08ce5a52016-02-22 13:28:10 -08001340 for k, kinfo in valid_keys.items():
Andrew Boie6acbe632015-07-17 12:03:52 -07001341 if k not in d:
1342 if "required" in kinfo:
1343 required = kinfo["required"]
1344 else:
1345 required = False
1346
1347 if required:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001348 raise ConfigurationError(
1349 self.filename,
Anas Nashifb4754ed2017-12-05 17:27:58 -05001350 "missing required value for '%s' in test '%s'" %
1351 (k, name))
Andrew Boie6acbe632015-07-17 12:03:52 -07001352 else:
1353 if "default" in kinfo:
1354 default = kinfo["default"]
1355 else:
1356 default = self._cast_value("", kinfo["type"])
1357 d[k] = default
1358 else:
1359 try:
1360 d[k] = self._cast_value(d[k], kinfo["type"])
Anas Nashif83fc06a2019-06-22 11:04:10 -04001361 except ValueError:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001362 raise ConfigurationError(
Anas Nashifb4754ed2017-12-05 17:27:58 -05001363 self.filename, "bad %s value '%s' for key '%s' in name '%s'" %
1364 (kinfo["type"], d[k], k, name))
Andrew Boie6acbe632015-07-17 12:03:52 -07001365
1366 return d
1367
1368
1369class Platform:
1370 """Class representing metadata for a particular platform
1371
Anas Nashifc7406082015-12-13 15:00:31 -05001372 Maps directly to BOARD when building"""
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001373
Anas Nashif83fc06a2019-06-22 11:04:10 -04001374 platform_schema = scl.yaml_load(os.path.join(ZEPHYR_BASE,
1375 "scripts","sanity_chk","platform-schema.yaml"))
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001376
Anas Nashif83fc06a2019-06-22 11:04:10 -04001377 def __init__(self):
Andrew Boie6acbe632015-07-17 12:03:52 -07001378 """Constructor.
1379
Andrew Boie6acbe632015-07-17 12:03:52 -07001380 """
Anas Nashif83fc06a2019-06-22 11:04:10 -04001381
1382 self.name = ""
1383 self.sanitycheck = True
1384 # if no RAM size is specified by the board, take a default of 128K
1385 self.ram = 128
1386
1387 self.ignore_tags = []
1388 self.default = False
1389 # if no flash size is specified by the board, take a default of 512K
1390 self.flash = 512
1391 self.supported = set()
1392
1393 self.arch = ""
1394 self.type = "na"
1395 self.simulation = "na"
1396 self.supported_toolchains = []
1397 self.env = []
1398 self.env_satisfied = True
1399 self.filter_data = dict()
1400
1401 def load(self, platform_file):
1402 scp = SanityConfigParser(platform_file, self.platform_schema)
1403 scp.load()
Anas Nashif255625b2017-12-05 15:08:26 -05001404 data = scp.data
Anas Nashifa792a3d2017-04-04 18:47:49 -04001405
Anas Nashif255625b2017-12-05 15:08:26 -05001406 self.name = data['identifier']
Anas Nashiff3d48e12018-07-24 08:14:42 -05001407 self.sanitycheck = data.get("sanitycheck", True)
Anas Nashifa792a3d2017-04-04 18:47:49 -04001408 # if no RAM size is specified by the board, take a default of 128K
Anas Nashif255625b2017-12-05 15:08:26 -05001409 self.ram = data.get("ram", 128)
1410 testing = data.get("testing", {})
Anas Nashifa792a3d2017-04-04 18:47:49 -04001411 self.ignore_tags = testing.get("ignore_tags", [])
1412 self.default = testing.get("default", False)
1413 # if no flash size is specified by the board, take a default of 512K
Anas Nashif255625b2017-12-05 15:08:26 -05001414 self.flash = data.get("flash", 512)
Anas Nashif95276932017-07-31 08:47:41 -04001415 self.supported = set()
Anas Nashif255625b2017-12-05 15:08:26 -05001416 for supp_feature in data.get("supported", []):
Anas Nashif95276932017-07-31 08:47:41 -04001417 for item in supp_feature.split(":"):
1418 self.supported.add(item)
1419
Anas Nashif255625b2017-12-05 15:08:26 -05001420 self.arch = data['arch']
Anas Nashifcc164222017-12-26 11:02:46 -05001421 self.type = data.get('type', "na")
Anas Nashif8acdbd72018-01-04 14:15:22 -05001422 self.simulation = data.get('simulation', "na")
Anas Nashif255625b2017-12-05 15:08:26 -05001423 self.supported_toolchains = data.get("toolchain", [])
Anas Nashif924a4e72018-10-18 12:25:55 -04001424 self.env = data.get("env", [])
1425 self.env_satisfied = True
1426 for env in self.env:
Anas Nashif83fc06a2019-06-22 11:04:10 -04001427 if not os.environ.get(env, None):
Anas Nashif924a4e72018-10-18 12:25:55 -04001428 self.env_satisfied = False
Anas Nashif83fc06a2019-06-22 11:04:10 -04001429
Andrew Boie6acbe632015-07-17 12:03:52 -07001430
Andrew Boie6acbe632015-07-17 12:03:52 -07001431 def __repr__(self):
Anas Nashifa792a3d2017-04-04 18:47:49 -04001432 return "<%s on %s>" % (self.name, self.arch)
Andrew Boie6acbe632015-07-17 12:03:52 -07001433
Anas Nashif83fc06a2019-06-22 11:04:10 -04001434class TestCase(object):
Andrew Boie6acbe632015-07-17 12:03:52 -07001435 """Class representing a test application
1436 """
Anas Nashif3ba1d432017-12-05 15:28:44 -05001437
Anas Nashif83fc06a2019-06-22 11:04:10 -04001438
1439 def __init__(self):
Andrew Boie6acbe632015-07-17 12:03:52 -07001440 """TestCase constructor.
1441
Anas Nashif877d3ca2017-12-05 17:39:29 -05001442 This gets called by TestSuite as it finds and reads test yaml files.
Anas Nashifa792a3d2017-04-04 18:47:49 -04001443 Multiple TestCase instances may be generated from a single testcase.yaml,
Anas Nashif877d3ca2017-12-05 17:39:29 -05001444 each one corresponds to an entry within that file.
Andrew Boie6acbe632015-07-17 12:03:52 -07001445
Andrew Boie6acbe632015-07-17 12:03:52 -07001446 We need to have a unique name for every single test case. Since
Anas Nashifa792a3d2017-04-04 18:47:49 -04001447 a testcase.yaml can define multiple tests, the canonical name for
Andrew Boie6acbe632015-07-17 12:03:52 -07001448 the test case is <workdir>/<name>.
1449
Marc Herbert1c8632c2019-04-15 17:58:45 -07001450 @param testcase_root os.path.abspath() of one of the --testcase-root
1451 @param workdir Sub-directory of testcase_root where the
1452 .yaml test configuration file was found
Anas Nashif877d3ca2017-12-05 17:39:29 -05001453 @param name Name of this test case, corresponding to the entry name
Andrew Boie6acbe632015-07-17 12:03:52 -07001454 in the test case configuration file. For many test cases that just
1455 define one test, can be anything and is usually "test". This is
1456 really only used to distinguish between different cases when
Anas Nashifa792a3d2017-04-04 18:47:49 -04001457 the testcase.yaml defines multiple tests
Anas Nashif877d3ca2017-12-05 17:39:29 -05001458 @param tc_dict Dictionary with test values for this test case
Anas Nashifa792a3d2017-04-04 18:47:49 -04001459 from the testcase.yaml file
Andrew Boie6acbe632015-07-17 12:03:52 -07001460 """
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001461
Anas Nashif83fc06a2019-06-22 11:04:10 -04001462 self.id = ""
1463 self.source_dir = ""
1464 self.yamlfile = ""
Anas Nashifaae71d72018-04-21 22:26:48 -05001465 self.cases = []
Anas Nashif83fc06a2019-06-22 11:04:10 -04001466 self.name = ""
Anas Nashifbd166f42017-09-02 12:32:08 -04001467
Anas Nashif83fc06a2019-06-22 11:04:10 -04001468 self.type = None
1469 self.tags = None
1470 self.extra_args = None
1471 self.extra_configs = None
1472 self.arch_whitelist = None
1473 self.arch_exclude = None
1474 self.skip = None
1475 self.platform_exclude = None
1476 self.platform_whitelist = None
1477 self.toolchain_exclude = None
1478 self.toolchain_whitelist = None
1479 self.tc_filter = None
1480 self.timeout = 60
1481 self.harness = ""
1482 self.harness_config = {}
1483 self.build_only = True
1484 self.build_on_all = False
1485 self.slow = False
1486 self.min_ram = None
1487 self.depends_on = None
1488 self.min_flash = None
1489 self.extra_sections = None
Andrew Boie6acbe632015-07-17 12:03:52 -07001490
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001491
Anas Nashif83fc06a2019-06-22 11:04:10 -04001492 @staticmethod
1493 def get_unique(testcase_root, workdir, name):
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001494
Marc Herbert1c8632c2019-04-15 17:58:45 -07001495 canonical_testcase_root = os.path.realpath(testcase_root)
Tyler Hallbb359252019-06-19 02:11:35 -04001496 if Path(canonical_zephyr_base) in Path(canonical_testcase_root).parents:
Marc Herbert1c8632c2019-04-15 17:58:45 -07001497 # This is in ZEPHYR_BASE, so include path in name for uniqueness
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001498 # FIXME: We should not depend on path of test for unique names.
Marc Herbert1c8632c2019-04-15 17:58:45 -07001499 relative_tc_root = os.path.relpath(canonical_testcase_root,
1500 start=canonical_zephyr_base)
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001501 else:
Marc Herbert1c8632c2019-04-15 17:58:45 -07001502 relative_tc_root = ""
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001503
Marc Herbert1c8632c2019-04-15 17:58:45 -07001504 # workdir can be "."
1505 unique = os.path.normpath(os.path.join(relative_tc_root, workdir, name))
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001506 return unique
1507
Anas Nashif83fc06a2019-06-22 11:04:10 -04001508 @staticmethod
1509 def scan_file(inf_name):
Anas Nashifaae71d72018-04-21 22:26:48 -05001510 suite_regex = re.compile(
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001511 # do not match until end-of-line, otherwise we won't allow
1512 # stc_regex below to catch the ones that are declared in the same
1513 # line--as we only search starting the end of this match
1514 br"^\s*ztest_test_suite\(\s*(?P<suite_name>[a-zA-Z0-9_]+)\s*,",
Anas Nashifaae71d72018-04-21 22:26:48 -05001515 re.MULTILINE)
1516 stc_regex = re.compile(
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001517 br"^\s*" # empy space at the beginning is ok
1518 # catch the case where it is declared in the same sentence, e.g:
1519 #
1520 # ztest_test_suite(mutex_complex, ztest_user_unit_test(TESTNAME));
1521 br"(?:ztest_test_suite\([a-zA-Z0-9_]+,\s*)?"
1522 # Catch ztest[_user]_unit_test-[_setup_teardown](TESTNAME)
1523 br"ztest_(?:user_)?unit_test(?:_setup_teardown)?"
1524 # Consume the argument that becomes the extra testcse
1525 br"\(\s*"
1526 br"(?P<stc_name>[a-zA-Z0-9_]+)"
1527 # _setup_teardown() variant has two extra arguments that we ignore
1528 br"(?:\s*,\s*[a-zA-Z0-9_]+\s*,\s*[a-zA-Z0-9_]+)?"
1529 br"\s*\)",
1530 # We don't check how it finishes; we don't care
Anas Nashifaae71d72018-04-21 22:26:48 -05001531 re.MULTILINE)
1532 suite_run_regex = re.compile(
1533 br"^\s*ztest_run_test_suite\((?P<suite_name>[a-zA-Z0-9_]+)\)",
1534 re.MULTILINE)
1535 achtung_regex = re.compile(
1536 br"(#ifdef|#endif)",
1537 re.MULTILINE)
1538 warnings = None
1539
1540 with open(inf_name) as inf:
Anas Nashif19d67e42019-11-21 11:33:12 -05001541 if os.name == 'nt':
1542 mmap_args = {'fileno':inf.fileno(), 'length':0, 'access':mmap.ACCESS_READ}
1543 else:
1544 mmap_args = {'fileno':inf.fileno(), 'length':0, 'flags':mmap.MAP_PRIVATE, 'prot':mmap.PROT_READ, 'offset':0}
1545
1546 with contextlib.closing(mmap.mmap(**mmap_args)) as main_c:
Ulf Magnusson5c2e8142019-10-29 11:50:26 +01001547 # contextlib makes pylint think main_c isn't subscriptable
1548 # pylint: disable=unsubscriptable-object
1549
Anas Nashifaae71d72018-04-21 22:26:48 -05001550 suite_regex_match = suite_regex.search(main_c)
1551 if not suite_regex_match:
1552 # can't find ztest_test_suite, maybe a client, because
1553 # it includes ztest.h
1554 return None, None
1555
1556 suite_run_match = suite_run_regex.search(main_c)
1557 if not suite_run_match:
1558 raise ValueError("can't find ztest_run_test_suite")
1559
1560 achtung_matches = re.findall(
1561 achtung_regex,
1562 main_c[suite_regex_match.end():suite_run_match.start()])
1563 if achtung_matches:
1564 warnings = "found invalid %s in ztest_test_suite()" \
Anas Nashif83fc06a2019-06-22 11:04:10 -04001565 % ", ".join({match.decode() for match in achtung_matches})
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001566 _matches = re.findall(
Anas Nashifaae71d72018-04-21 22:26:48 -05001567 stc_regex,
1568 main_c[suite_regex_match.end():suite_run_match.start()])
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001569 matches = [ match.decode().replace("test_", "") for match in _matches ]
Anas Nashifaae71d72018-04-21 22:26:48 -05001570 return matches, warnings
1571
1572 def scan_path(self, path):
1573 subcases = []
1574 for filename in glob.glob(os.path.join(path, "src", "*.c")):
1575 try:
1576 _subcases, warnings = self.scan_file(filename)
1577 if warnings:
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001578 error("%s: %s" % (filename, warnings))
Anas Nashifaae71d72018-04-21 22:26:48 -05001579 if _subcases:
1580 subcases += _subcases
1581 except ValueError as e:
Flavio Ceolinbf878ce2019-04-19 22:24:09 -07001582 error("%s: can't find: %s" % (filename, e))
Anas Nashifaae71d72018-04-21 22:26:48 -05001583 return subcases
1584
Anas Nashif83fc06a2019-06-22 11:04:10 -04001585 def parse_subcases(self, test_path):
1586 results = self.scan_path(os.path.dirname(test_path))
Anas Nashifaae71d72018-04-21 22:26:48 -05001587 for sub in results:
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001588 name = "{}.{}".format(self.id, sub)
Anas Nashifaae71d72018-04-21 22:26:48 -05001589 self.cases.append(name)
1590
Anas Nashiff16e92c2019-03-31 16:58:12 -04001591 if not results:
1592 self.cases.append(self.id)
1593
Anas Nashifaae71d72018-04-21 22:26:48 -05001594
Anas Nashif75547e22018-02-24 08:32:14 -06001595 def __str__(self):
Andrew Boie6acbe632015-07-17 12:03:52 -07001596 return self.name
1597
1598
Andrew Boie6acbe632015-07-17 12:03:52 -07001599class TestInstance:
1600 """Class representing the execution of a particular TestCase on a platform
1601
1602 @param test The TestCase object we want to build/execute
1603 @param platform Platform object that we want to build and run against
1604 @param base_outdir Base directory for all test results. The actual
1605 out directory used is <outdir>/<platform>/<test case name>
1606 """
Anas Nashif3ba1d432017-12-05 15:28:44 -05001607
Anas Nashif83fc06a2019-06-22 11:04:10 -04001608 def __init__(self, testcase, platform, base_outdir):
Anas Nashif7dd19ea2018-11-14 08:46:49 -05001609
Anas Nashif83fc06a2019-06-22 11:04:10 -04001610 self.testcase = testcase
1611 self.platform = platform
1612
1613 self.status = None
1614 self.reason = "N/A"
1615 self.metrics = dict()
1616 self.handler = None
1617
1618
1619 self.name = os.path.join(platform.name, testcase.name)
1620 self.build_dir = os.path.join(base_outdir, platform.name, testcase.name)
1621
1622 self.build_only = self.check_build_or_run()
1623 self.run = not self.build_only
1624
Anas Nashife0a6a0b2018-02-15 20:07:24 -06001625 self.results = {}
Andrew Boie6acbe632015-07-17 12:03:52 -07001626
Marc Herbert0f7255c2019-04-05 14:14:21 -07001627 def __lt__(self, other):
1628 return self.name < other.name
1629
Anas Nashif83fc06a2019-06-22 11:04:10 -04001630 def check_build_or_run(self):
Anas Nashif19d67e42019-11-21 11:33:12 -05001631 # right now we only support building on windows. running is still work
1632 # in progress.
1633
1634 if os.name == 'nt':
1635 return True
Anas Nashif7dd19ea2018-11-14 08:46:49 -05001636
Anas Nashif83fc06a2019-06-22 11:04:10 -04001637 build_only = True
1638
1639 # we asked for build-only on the command line
1640 if options.build_only:
1641 return True
1642
1643 # The testcase is designed to be build only.
1644 if self.testcase.build_only:
1645 return True
1646
1647 # Do not run slow tests:
1648 skip_slow = self.testcase.slow and not options.enable_slow
1649 if skip_slow:
1650 return True
1651
1652 runnable =bool(self.testcase.type == "unit" or \
1653 self.platform.type == "native" or \
1654 self.platform.simulation in ["nsim", "renode", "qemu"] or \
1655 options.device_testing)
1656
1657 if self.platform.simulation == "nsim":
1658 if not find_executable("nsimdrv"):
1659 runnable = False
1660
1661 if self.platform.simulation == "renode":
1662 if not find_executable("renode"):
1663 runnable = False
1664
1665 # console harness allows us to run the test and capture data.
1666 if self.testcase.harness == 'console':
1667
1668 # if we have a fixture that is also being supplied on the
1669 # command-line, then we need to run the test, not just build it.
1670 if "fixture" in self.testcase.harness_config:
1671 fixture = self.testcase.harness_config['fixture']
1672 if fixture in options.fixture:
1673 build_only = False
1674 else:
1675 build_only = True
1676 else:
1677 build_only = False
1678 elif self.testcase.harness:
1679 build_only = True
1680 else:
1681 build_only = False
1682
1683 return not (not build_only and runnable)
Anas Nashif7dd19ea2018-11-14 08:46:49 -05001684
Anas Nashifdbd76492018-11-23 20:24:19 -05001685 def create_overlay(self, platform):
Marc Herbertc7633de2019-07-06 15:52:31 -07001686 # Create this in a "sanitycheck/" subdirectory otherwise this
1687 # will pass this overlay to kconfig.py *twice* and kconfig.cmake
1688 # will silently give that second time precedence over any
1689 # --extra-args=CONFIG_*
Anas Nashif83fc06a2019-06-22 11:04:10 -04001690 subdir = os.path.join(self.build_dir, "sanitycheck")
Marc Herbertc7633de2019-07-06 15:52:31 -07001691 os.makedirs(subdir, exist_ok=True)
1692 file = os.path.join(subdir, "testcase_extra.conf")
Anas Nashiff4d8ecc2019-03-02 15:43:23 -05001693 with open(file, "w") as f:
1694 content = ""
Anas Nashif3cbffef2018-11-07 23:50:54 -05001695
Anas Nashif83fc06a2019-06-22 11:04:10 -04001696 if self.testcase.extra_configs:
1697 content = "\n".join(self.testcase.extra_configs)
Anas Nashif3cbffef2018-11-07 23:50:54 -05001698
Anas Nashiff4d8ecc2019-03-02 15:43:23 -05001699 if options.enable_coverage:
Jan Van Winkel21212f32019-09-12 00:03:35 +02001700 if platform.name in options.coverage_platform:
Anas Nashiff4d8ecc2019-03-02 15:43:23 -05001701 content = content + "\nCONFIG_COVERAGE=y"
1702
Jan Van Winkel21212f32019-09-12 00:03:35 +02001703 if options.enable_asan:
1704 if platform.type == "native":
1705 content = content + "\nCONFIG_ASAN=y"
1706
Anas Nashiff4d8ecc2019-03-02 15:43:23 -05001707 f.write(content)
Anas Nashiffa695d22017-10-04 16:14:27 -04001708
Andrew Boie6acbe632015-07-17 12:03:52 -07001709 def calculate_sizes(self):
1710 """Get the RAM/ROM sizes of a test case.
1711
1712 This can only be run after the instance has been executed by
1713 MakeGenerator, otherwise there won't be any binaries to measure.
1714
1715 @return A SizeCalculator object
1716 """
Anas Nashif83fc06a2019-06-22 11:04:10 -04001717 fns = glob.glob(os.path.join(self.build_dir, "zephyr", "*.elf"))
1718 fns.extend(glob.glob(os.path.join(self.build_dir, "zephyr", "*.exe")))
Anas Nashiff8dcad42016-10-27 18:10:08 -04001719 fns = [x for x in fns if not x.endswith('_prebuilt.elf')]
Anas Nashif83fc06a2019-06-22 11:04:10 -04001720 if len(fns) != 1:
Andrew Boie5d4eb782015-10-02 10:04:56 -07001721 raise BuildError("Missing/multiple output ELF binary")
Anas Nashif83fc06a2019-06-22 11:04:10 -04001722
1723 return SizeCalculator(fns[0], self.testcase.extra_sections)
Andrew Boie6acbe632015-07-17 12:03:52 -07001724
1725 def __repr__(self):
Anas Nashif83fc06a2019-06-22 11:04:10 -04001726 return "<TestCase %s on %s>" % (self.testcase.name, self.platform.name)
Andrew Boie6acbe632015-07-17 12:03:52 -07001727
1728
Anas Nashif83fc06a2019-06-22 11:04:10 -04001729class CMake():
Andrew Boie4ef16c52015-08-28 12:36:03 -07001730
Anas Nashif83fc06a2019-06-22 11:04:10 -04001731 config_re = re.compile('(CONFIG_[A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$')
1732 dt_re = re.compile('([A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$')
1733
1734 def __init__(self, testcase, platform, source_dir, build_dir):
1735
1736 self.cwd = None
1737 self.capture_output = True
1738
1739 self.defconfig = {}
1740 self.cmake_cache = {}
Anas Nashif83fc06a2019-06-22 11:04:10 -04001741
1742 self.instance = None
1743 self.testcase = testcase
1744 self.platform = platform
1745 self.source_dir = source_dir
1746 self.build_dir = build_dir
1747 self.log = "build.log"
1748
1749 def parse_generated(self):
1750 self.defconfig = {}
1751 return {}
1752
1753 def run_build(self, args=[]):
1754
1755 verbose("Building %s for %s" % (self.source_dir, self.platform.name))
1756
1757 cmake_args = []
1758 cmake_args.extend(args)
1759 cmake = shutil.which('cmake')
1760 cmd = [cmake] + cmake_args
1761 kwargs = dict()
1762
1763 if self.capture_output:
1764 kwargs['stdout'] = subprocess.PIPE
1765 # CMake sends the output of message() to stderr unless it's STATUS
1766 kwargs['stderr'] = subprocess.STDOUT
1767
1768 if self.cwd:
1769 kwargs['cwd'] = self.cwd
1770
1771 p = subprocess.Popen(cmd, **kwargs)
1772 out, _ = p.communicate()
1773
1774 results = {}
1775 if p.returncode == 0:
1776 msg = "Finished building %s for %s" %(self.source_dir, self.platform.name)
1777
1778 self.instance.status = "passed"
Anas Nashif83fc06a2019-06-22 11:04:10 -04001779 results = {'msg': msg, "returncode": p.returncode, "instance": self.instance}
1780
1781 if out:
1782 log_msg = out.decode(sys.getdefaultencoding())
1783 with open(os.path.join(self.build_dir, self.log), "a") as log:
1784 log.write(log_msg)
1785
1786 else:
1787 return None
1788 else:
1789 # A real error occurred, raise an exception
1790 if out:
1791 log_msg = out.decode(sys.getdefaultencoding())
1792 with open(os.path.join(self.build_dir, self.log), "a") as log:
1793 log.write(log_msg)
1794
1795 overflow_flash = "region `FLASH' overflowed by"
1796 overflow_ram = "region `RAM' overflowed by"
1797
1798 if log_msg:
1799 if log_msg.find(overflow_flash) > 0 or log_msg.find(overflow_ram) > 0:
1800 verbose("RAM/ROM Overflow")
1801 self.instance.status = "skipped"
1802 self.instance.reason = "overflow"
1803 else:
1804 self.instance.status = "failed"
1805 self.instance.reason = "Build failure"
1806
1807 results = {
1808 "returncode": p.returncode,
1809 "instance": self.instance,
1810 }
1811
1812 return results
1813
1814 def run_cmake(self, args=[]):
1815
1816 verbose("Running cmake on %s for %s" %(self.source_dir, self.platform.name))
1817
Anas Nashifa5984ab2019-10-22 07:36:24 -07001818 ldflags="-Wl,--fatal-warnings"
1819
1820 #fixme: add additional cflags based on options
1821 cmake_args = [
1822 '-B{}'.format(self.build_dir),
1823 '-S{}'.format(self.source_dir),
1824 '-DEXTRA_CFLAGS="-Werror ',
1825 '-DEXTRA_AFLAGS=-Wa,--fatal-warnings',
1826 '-DEXTRA_LDFLAGS="{}'.format(ldflags),
1827 '-G{}'.format(get_generator()[1])
1828 ]
Anas Nashif83fc06a2019-06-22 11:04:10 -04001829
1830 args = ["-D{}".format(a.replace('"', '')) for a in args]
1831 cmake_args.extend(args)
1832
1833 cmake_opts = ['-DBOARD={}'.format(self.platform.name)]
1834 cmake_args.extend(cmake_opts)
1835
1836 cmake = shutil.which('cmake')
1837 cmd = [cmake] + cmake_args
1838 kwargs = dict()
1839
1840 if self.capture_output:
1841 kwargs['stdout'] = subprocess.PIPE
1842 # CMake sends the output of message() to stderr unless it's STATUS
1843 kwargs['stderr'] = subprocess.STDOUT
1844
1845 if self.cwd:
1846 kwargs['cwd'] = self.cwd
1847
1848 p = subprocess.Popen(cmd, **kwargs)
1849 out, _ = p.communicate()
1850
1851 if p.returncode == 0:
1852 filter_results = self.parse_generated()
1853 msg = "Finished building %s for %s" %(self.source_dir, self.platform.name)
1854
1855 results = {'msg': msg, 'filter': filter_results}
1856
1857 else:
1858 self.instance.status = "failed"
1859 self.instance.reason = "Cmake build failure"
1860 results = {"returncode": p.returncode}
1861
1862
1863 if out:
1864 with open(os.path.join(self.build_dir, self.log), "a") as log:
1865 log_msg = out.decode(sys.getdefaultencoding())
1866 log.write(log_msg)
1867
1868 return results
1869
1870
1871class FilterBuilder(CMake):
1872
1873 def __init__(self, testcase, platform, source_dir, build_dir):
1874 super().__init__(testcase, platform, source_dir, build_dir)
1875
1876 self.log = "config-sanitycheck.log"
1877
1878 def parse_generated(self):
1879
1880 if self.platform.name == "unit_testing":
1881 return {}
1882
Anas Nashif83fc06a2019-06-22 11:04:10 -04001883 cmake_cache_path = os.path.join(self.build_dir, "CMakeCache.txt")
Anas Nashif83fc06a2019-06-22 11:04:10 -04001884 defconfig_path = os.path.join(self.build_dir, "zephyr", ".config")
1885
1886 with open(defconfig_path, "r") as fp:
1887 defconfig = {}
1888 for line in fp.readlines():
1889 m = self.config_re.match(line)
1890 if not m:
1891 if line.strip() and not line.startswith("#"):
1892 sys.stderr.write("Unrecognized line %s\n" % line)
1893 continue
1894 defconfig[m.group(1)] = m.group(2).strip()
1895
1896 self.defconfig = defconfig
1897
1898 cmake_conf = {}
1899 try:
1900 cache = CMakeCache.from_file(cmake_cache_path)
1901 except FileNotFoundError:
1902 cache = {}
1903
1904 for k in iter(cache):
1905 cmake_conf[k.name] = k.value
1906
1907 self.cmake_cache = cmake_conf
1908
Anas Nashif83fc06a2019-06-22 11:04:10 -04001909 filter_data = {
1910 "ARCH": self.platform.arch,
1911 "PLATFORM": self.platform.name
1912 }
1913 filter_data.update(os.environ)
1914 filter_data.update(self.defconfig)
1915 filter_data.update(self.cmake_cache)
Anas Nashif83fc06a2019-06-22 11:04:10 -04001916
Anas Nashif556f3cb2019-11-05 15:36:15 -08001917 dts_path = os.path.join(self.build_dir, "zephyr", self.platform.name + ".dts.pre.tmp")
Kumar Galaae88e442019-11-06 04:55:24 -06001918 if self.testcase and self.testcase.tc_filter:
Anas Nashif83fc06a2019-06-22 11:04:10 -04001919 try:
Kumar Galaae88e442019-11-06 04:55:24 -06001920 if os.path.exists(dts_path):
1921 edt = edtlib.EDT(dts_path, [os.path.join(ZEPHYR_BASE, "dts", "bindings")])
1922 else:
1923 edt = None
Kumar Gala7733b942019-09-12 17:08:43 -05001924 res = expr_parser.parse(self.testcase.tc_filter, filter_data, edt)
1925
Anas Nashif83fc06a2019-06-22 11:04:10 -04001926 except (ValueError, SyntaxError) as se:
1927 sys.stderr.write(
1928 "Failed processing %s\n" % self.testcase.yamlfile)
1929 raise se
1930
1931 if not res:
1932 return {os.path.join(self.platform.name, self.testcase.name): True}
1933 else:
1934 return {os.path.join(self.platform.name, self.testcase.name): False}
1935 else:
1936 self.platform.filter_data = filter_data
1937 return filter_data
1938
1939
1940class ProjectBuilder(FilterBuilder):
1941
1942 def __init__(self, suite, instance):
1943 super().__init__(instance.testcase, instance.platform, instance.testcase.source_dir, instance.build_dir)
1944
1945 self.log = "build.log"
1946 self.instance = instance
1947 self.suite = suite
1948
1949 def setup_handler(self):
1950
1951 instance = self.instance
1952 args = []
1953
1954 # FIXME: Needs simplification
1955 if instance.platform.simulation == "qemu":
1956 instance.handler = QEMUHandler(instance, "qemu")
1957 args.append("QEMU_PIPE=%s" % instance.handler.get_fifo())
1958 instance.handler.call_make_run = True
1959 elif instance.testcase.type == "unit":
1960 instance.handler = BinaryHandler(instance, "unit")
1961 instance.handler.binary = os.path.join(instance.build_dir, "testbinary")
1962 elif instance.platform.type == "native":
1963 instance.handler = BinaryHandler(instance, "native")
1964 instance.handler.binary = os.path.join(instance.build_dir, "zephyr", "zephyr.exe")
1965 elif instance.platform.simulation == "nsim":
1966 if find_executable("nsimdrv"):
1967 instance.handler = BinaryHandler(instance, "nsim")
1968 instance.handler.call_make_run = True
1969 elif instance.platform.simulation == "renode":
1970 if find_executable("renode"):
1971 instance.handler = BinaryHandler(instance, "renode")
1972 instance.handler.pid_fn = os.path.join(instance.build_dir, "renode.pid")
1973 instance.handler.call_make_run = True
1974 elif options.device_testing:
1975 instance.handler = DeviceHandler(instance, "device")
1976
1977 if instance.handler:
1978 instance.handler.args = args
1979
1980 def process(self, message):
1981 op = message.get('op')
1982
1983 if not self.instance.handler:
1984 self.setup_handler()
1985
1986 # The build process, call cmake and build with configured generator
1987 if op == "cmake":
1988 results = self.cmake()
1989 if self.instance.status == "failed":
1990 pipeline.put({"op": "report", "test": self.instance})
1991 elif options.cmake_only:
1992 pipeline.put({"op": "report", "test": self.instance})
1993 else:
1994 if self.instance.name in results['filter'] and results['filter'][self.instance.name]:
1995 verbose("filtering %s" % self.instance.name)
1996 self.instance.status = "skipped"
1997 self.instance.reason = "filter"
1998 pipeline.put({"op": "report", "test": self.instance})
1999 else:
2000 pipeline.put({"op": "build", "test": self.instance})
2001
2002
2003 elif op == "build":
2004 verbose("build test: %s" %self.instance.name)
2005 results = self.build()
2006
2007 if results.get('returncode', 1) > 0:
2008 pipeline.put({"op": "report", "test": self.instance})
2009 else:
2010 if self.instance.run:
2011 pipeline.put({"op": "run", "test": self.instance})
2012 else:
2013 pipeline.put({"op": "report", "test": self.instance})
2014 # Run the generated binary using one of the supported handlers
2015 elif op == "run":
2016 verbose("run test: %s" %self.instance.name)
2017 self.run()
2018 self.instance.status, _ = self.instance.handler.get_state()
Anas Nashif83fc06a2019-06-22 11:04:10 -04002019 pipeline.put({
2020 "op": "report",
2021 "test": self.instance,
2022 "state": "executed",
2023 "status": self.instance.status,
Stephanos Ioannidis93889002019-11-11 20:31:03 +09002024 "reason": self.instance.reason}
Anas Nashif83fc06a2019-06-22 11:04:10 -04002025 )
2026
2027 # Report results and output progress to screen
2028 elif op == "report":
Anas Nashifc1ea4522019-10-11 07:32:45 -07002029 with report_lock:
2030 self.report_out()
2031
Anas Nashif83fc06a2019-06-22 11:04:10 -04002032 def report_out(self):
2033 total_tests_width = len(str(self.suite.total_tests))
2034 self.suite.total_done += 1
2035 instance = self.instance
2036
2037 if instance.status in ["failed", "timeout"]:
2038 self.suite.total_failed += 1
2039 if VERBOSE or not TERMINAL:
2040 status = COLOR_RED + "FAILED " + COLOR_NORMAL + instance.reason
2041 else:
2042 info(
Anas Nashifc1ea4522019-10-11 07:32:45 -07002043 "\n{:<25} {:<50} {}FAILED{}: {}".format(
Anas Nashif83fc06a2019-06-22 11:04:10 -04002044 instance.platform.name,
2045 instance.testcase.name,
2046 COLOR_RED,
2047 COLOR_NORMAL,
2048 instance.reason), False)
Anas Nashifc1ea4522019-10-11 07:32:45 -07002049 if not VERBOSE:
Ulf Magnussone73d2862019-10-29 11:54:02 +01002050 log_info_file(instance)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002051 elif instance.status == "skipped":
2052 self.suite.total_skipped += 1
2053 status = COLOR_YELLOW + "SKIPPED" + COLOR_NORMAL
Anas Nashif83fc06a2019-06-22 11:04:10 -04002054 else:
2055 status = COLOR_GREEN + "PASSED" + COLOR_NORMAL
2056
2057 if VERBOSE or not TERMINAL:
2058 if options.cmake_only:
2059 more_info = "cmake"
2060 elif instance.status == "skipped":
2061 more_info = instance.reason
2062 else:
2063 if instance.handler and instance.run:
2064 more_info = instance.handler.type_str
2065 htime = instance.handler.duration
2066 if htime:
2067 more_info += " {:.3f}s".format(htime)
2068 else:
2069 more_info = "build"
2070
2071 info("{:>{}}/{} {:<25} {:<50} {} ({})".format(
2072 self.suite.total_done, total_tests_width, self.suite.total_tests, instance.platform.name,
2073 instance.testcase.name, status, more_info))
2074
2075 if instance.status in ["failed", "timeout"]:
Ulf Magnussone73d2862019-10-29 11:54:02 +01002076 log_info_file(instance)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002077 else:
2078 sys.stdout.write("\rtotal complete: %s%4d/%4d%s %2d%% skipped: %s%4d%s, failed: %s%4d%s" % (
2079 COLOR_GREEN,
2080 self.suite.total_done,
2081 self.suite.total_tests,
2082 COLOR_NORMAL,
2083 int((float(self.suite.total_done) / self.suite.total_tests) * 100),
2084 COLOR_YELLOW if self.suite.total_skipped > 0 else COLOR_NORMAL,
2085 self.suite.total_skipped,
2086 COLOR_NORMAL,
2087 COLOR_RED if self.suite.total_failed > 0 else COLOR_NORMAL,
2088 self.suite.total_failed,
2089 COLOR_NORMAL
2090 )
2091 )
2092 sys.stdout.flush()
2093
2094 def cmake(self):
2095
2096 instance = self.instance
2097 args = self.testcase.extra_args[:]
2098
2099 if options.extra_args:
2100 args += options.extra_args
2101
2102 if instance.handler:
2103 args += instance.handler.args
2104
2105 # merge overlay files into one variable
2106 overlays = ""
2107 idx = 0
2108 for arg in args:
2109 match = re.search('OVERLAY_CONFIG="(.*)"', arg)
2110 if match:
2111 overlays += match.group(1)
2112 del args[idx]
2113 idx += 1
2114
Jan Van Winkel21212f32019-09-12 00:03:35 +02002115 if (self.testcase.extra_configs or options.coverage or
2116 options.enable_asan):
Anas Nashif83fc06a2019-06-22 11:04:10 -04002117 args.append("OVERLAY_CONFIG=\"%s %s\"" %(overlays,
2118 os.path.join(instance.build_dir,
2119 "sanitycheck", "testcase_extra.conf")))
2120
2121 results = self.run_cmake(args)
2122 return results
2123
2124 def build(self):
2125 results = self.run_build(['--build', self.build_dir])
2126 return results
2127
2128 def run(self):
2129
2130 instance = self.instance
2131
2132 if instance.handler.type_str == "device":
2133 instance.handler.suite = self.suite
2134
2135 instance.handler.handle()
2136
Anas Nashif83fc06a2019-06-22 11:04:10 -04002137 sys.stdout.flush()
2138
2139
2140pipeline = queue.LifoQueue()
2141
2142class BoundedExecutor(concurrent.futures.ThreadPoolExecutor):
2143 """BoundedExecutor behaves as a ThreadPoolExecutor which will block on
2144 calls to submit() once the limit given as "bound" work items are queued for
2145 execution.
2146 :param bound: Integer - the maximum number of items in the work queue
2147 :param max_workers: Integer - the size of the thread pool
2148 """
2149 def __init__(self, bound, max_workers, **kwargs):
2150 super().__init__(max_workers)
2151 #self.executor = ThreadPoolExecutor(max_workers=max_workers)
2152 self.semaphore = BoundedSemaphore(bound + max_workers)
2153
2154 def submit(self, fn, *args, **kwargs):
2155 self.semaphore.acquire()
2156 try:
2157 future = super().submit(fn, *args, **kwargs)
2158 except:
2159 self.semaphore.release()
2160 raise
2161 else:
2162 future.add_done_callback(lambda x: self.semaphore.release())
2163 return future
2164
Andrew Boie6acbe632015-07-17 12:03:52 -07002165
2166class TestSuite:
Andrew Boie60823c22017-02-10 09:43:07 -08002167 config_re = re.compile('(CONFIG_[A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$')
Anas Nashif1c65b6b2018-12-02 19:12:21 -05002168 dt_re = re.compile('([A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$')
Andrew Boie6acbe632015-07-17 12:03:52 -07002169
Anas Nashif83fc06a2019-06-22 11:04:10 -04002170 tc_schema = scl.yaml_load(
Oleg Zhurakivskyy42822082018-08-17 14:54:41 +03002171 os.path.join(ZEPHYR_BASE,
Anas Nashif83fc06a2019-06-22 11:04:10 -04002172 "scripts", "sanity_chk", "testcase-schema.yaml"))
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07002173
Anas Nashif37f9dc52018-02-23 08:53:46 -06002174 def __init__(self, board_root_list, testcase_roots, outdir):
Anas Nashif83fc06a2019-06-22 11:04:10 -04002175
2176 self.roots = testcase_roots
2177 if not isinstance(board_root_list, list):
2178 self.board_roots= [board_root_list]
2179 else:
2180 self.board_roots = board_root_list
2181
Andrew Boie6acbe632015-07-17 12:03:52 -07002182 # Keep track of which test cases we've filtered out and why
Andrew Boie6acbe632015-07-17 12:03:52 -07002183 self.testcases = {}
2184 self.platforms = []
Anas Nashif83fc06a2019-06-22 11:04:10 -04002185 self.default_platforms = []
Andrew Boieb391e662015-08-31 15:25:45 -07002186 self.outdir = os.path.abspath(outdir)
Andrew Boie6acbe632015-07-17 12:03:52 -07002187 self.discards = None
Kumar Galac84235e2018-04-10 13:32:51 -05002188 self.load_errors = 0
Anas Nashif83fc06a2019-06-22 11:04:10 -04002189 self.instances = dict()
Andrew Boie6acbe632015-07-17 12:03:52 -07002190
Anas Nashif83fc06a2019-06-22 11:04:10 -04002191 self.total_tests = 0 # number of test instances
2192 self.total_cases = 0 # number of test cases
2193 self.total_done = 0 # tests completed
2194 self.total_failed = 0
2195 self.total_skipped = 0
Andrew Boie6acbe632015-07-17 12:03:52 -07002196
Anas Nashif83fc06a2019-06-22 11:04:10 -04002197 self.total_platforms = 0
2198 self.start_time = 0
2199 self.duration = 0
2200 self.warnings = 0
2201 self.cv = threading.Condition()
Anas Nashif61e21632018-04-08 13:30:16 -05002202
Anas Nashif83fc06a2019-06-22 11:04:10 -04002203 # hardcoded for now
2204 self.connected_hardware = []
Andrew Boie3d348712016-04-08 11:52:13 -07002205
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07002206
Anas Nashif83fc06a2019-06-22 11:04:10 -04002207 if options.jobs:
2208 self.jobs = options.jobs
2209 elif options.build_only:
2210 self.jobs = multiprocessing.cpu_count() * 2
Andy Ross9c9162d2019-01-03 10:50:53 -08002211 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002212 self.jobs = multiprocessing.cpu_count()
Daniel Leung6b170072016-04-07 12:10:25 -07002213
Anas Nashif83fc06a2019-06-22 11:04:10 -04002214 info("JOBS: %d" % self.jobs)
Andrew Boie6acbe632015-07-17 12:03:52 -07002215
Anas Nashif83fc06a2019-06-22 11:04:10 -04002216 def update(self):
2217 self.total_tests = len(self.instances)
2218 self.total_cases = len(self.testcases)
Anas Nashifbd166f42017-09-02 12:32:08 -04002219
Andrew Boie6acbe632015-07-17 12:03:52 -07002220
2221 def compare_metrics(self, filename):
2222 # name, datatype, lower results better
2223 interesting_metrics = [("ram_size", int, True),
2224 ("rom_size", int, True)]
2225
Andrew Boie6acbe632015-07-17 12:03:52 -07002226
2227 if not os.path.exists(filename):
2228 info("Cannot compare metrics, %s not found" % filename)
2229 return []
2230
2231 results = []
2232 saved_metrics = {}
2233 with open(filename) as fp:
2234 cr = csv.DictReader(fp)
2235 for row in cr:
2236 d = {}
2237 for m, _, _ in interesting_metrics:
2238 d[m] = row[m]
2239 saved_metrics[(row["test"], row["platform"])] = d
2240
Anas Nashif83fc06a2019-06-22 11:04:10 -04002241 for instance in self.instances.values():
2242 mkey = (instance.testcase.name, instance.platform.name)
Andrew Boie6acbe632015-07-17 12:03:52 -07002243 if mkey not in saved_metrics:
2244 continue
2245 sm = saved_metrics[mkey]
2246 for metric, mtype, lower_better in interesting_metrics:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002247 if metric not in instance.metrics:
Andrew Boie6acbe632015-07-17 12:03:52 -07002248 continue
2249 if sm[metric] == "":
2250 continue
Anas Nashif83fc06a2019-06-22 11:04:10 -04002251 delta = instance.metrics.get(metric, 0) - mtype(sm[metric])
Andrew Boieea7928f2015-08-14 14:27:38 -07002252 if delta == 0:
2253 continue
Anas Nashif83fc06a2019-06-22 11:04:10 -04002254 results.append((instance, metric, instance.metrics.get(metric, 0 ), delta,
Andrew Boieea7928f2015-08-14 14:27:38 -07002255 lower_better))
Andrew Boie6acbe632015-07-17 12:03:52 -07002256 return results
2257
Anas Nashif83fc06a2019-06-22 11:04:10 -04002258 def misc_reports(self, report, show_footprint, all_deltas,
2259 footprint_threshold, last_metrics):
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002260
Anas Nashif83fc06a2019-06-22 11:04:10 -04002261 if not report:
2262 return
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002263
Anas Nashif83fc06a2019-06-22 11:04:10 -04002264 deltas = self.compare_metrics(report)
2265 warnings = 0
2266 if deltas and show_footprint:
2267 for i, metric, value, delta, lower_better in deltas:
2268 if not all_deltas and ((delta < 0 and lower_better) or
2269 (delta > 0 and not lower_better)):
2270 continue
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002271
Anas Nashif83fc06a2019-06-22 11:04:10 -04002272 percentage = (float(delta) / float(value - delta))
2273 if not all_deltas and (percentage <
2274 (footprint_threshold / 100.0)):
2275 continue
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002276
Anas Nashif83fc06a2019-06-22 11:04:10 -04002277 info("{:<25} {:<60} {}{}{}: {} {:<+4}, is now {:6} {:+.2%}".format(
2278 i.platform.name, i.testcase.name, COLOR_YELLOW,
2279 "INFO" if all_deltas else "WARNING", COLOR_NORMAL,
2280 metric, delta, value, percentage))
2281 warnings += 1
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002282
Anas Nashif83fc06a2019-06-22 11:04:10 -04002283 if warnings:
2284 info("Deltas based on metrics from last %s" %
2285 ("release" if not last_metrics else "run"))
Anas Nashif61e21632018-04-08 13:30:16 -05002286
Anas Nashif83fc06a2019-06-22 11:04:10 -04002287 def summary(self, unrecognized_sections):
2288 failed = 0
2289 for instance in self.instances.values():
2290 if instance.status == "failed":
2291 failed += 1
2292 elif instance.metrics.get("unrecognized") and not unrecognized_sections:
2293 info("%sFAILED%s: %s has unrecognized binary sections: %s" %
2294 (COLOR_RED, COLOR_NORMAL, instance.name,
2295 str(instance.metrics.get("unrecognized", []))))
2296 failed += 1
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002297
Anas Nashif83fc06a2019-06-22 11:04:10 -04002298 if self.total_tests and self.total_tests != self.total_skipped:
2299 pass_rate = (float(self.total_tests - self.total_failed - self.total_skipped)/ float(self.total_tests - self.total_skipped))
2300 else:
2301 pass_rate = 0
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002302
Anas Nashif83fc06a2019-06-22 11:04:10 -04002303 info("{}{} of {}{} tests passed ({:.2%}), {}{}{} failed, {} skipped with {}{}{} warnings in {:.2f} seconds".format(
2304 COLOR_RED if failed else COLOR_GREEN,
2305 self.total_tests - self.total_failed - self.total_skipped,
2306 self.total_tests,
2307 COLOR_NORMAL,
2308 pass_rate,
2309 COLOR_RED if self.total_failed else COLOR_NORMAL,
2310 self.total_failed,
2311 COLOR_NORMAL,
2312 self.total_skipped,
2313 COLOR_YELLOW if self.warnings else COLOR_NORMAL,
2314 self.warnings,
2315 COLOR_NORMAL,
2316 self.duration))
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002317
Anas Nashif83fc06a2019-06-22 11:04:10 -04002318 platforms = set(p.platform for p in self.instances.values())
2319 self.total_platforms = len(self.platforms)
2320 if self.platforms:
2321 info("In total {} test cases were executed on {} out of total {} platforms ({:02.2f}%)".format(
2322 self.total_cases,
2323 len(platforms),
2324 self.total_platforms,
2325 (100 * len(platforms) / len(self.platforms))
2326 ))
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002327
Anas Nashif83fc06a2019-06-22 11:04:10 -04002328 def save_reports(self):
2329 if not self.instances:
2330 return
Anas Nashif61e21632018-04-08 13:30:16 -05002331
Anas Nashif83fc06a2019-06-22 11:04:10 -04002332 report_name = "sanitycheck"
2333 if options.report_name:
2334 report_name = options.report_name
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002335
Anas Nashif83fc06a2019-06-22 11:04:10 -04002336 if options.report_dir:
Paul Sokolovsky3ea18692019-10-15 17:18:39 +03002337 os.makedirs(options.report_dir, exist_ok=True)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002338 filename = os.path.join(options.report_dir, report_name)
2339 outdir = options.report_dir
2340 else:
2341 filename = os.path.join(options.outdir, report_name)
2342 outdir = options.outdir
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002343
Anas Nashif83fc06a2019-06-22 11:04:10 -04002344 if not options.no_update:
2345 self.xunit_report(filename + ".xml")
2346 self.csv_report(filename + ".csv")
2347 self.target_report(outdir)
2348 if self.discards:
2349 self.discard_report(filename + "_discard.csv")
2350
2351 if options.release:
2352 self.csv_report(RELEASE_DATA)
2353
2354 if log_file:
2355 log_file.close()
2356
2357 def load_hardware_map_from_cmdline(self, serial, platform):
2358 device = {
2359 "serial": serial,
2360 "platform": platform,
2361 "counter": 0,
2362 "available": True,
2363 "connected": True
2364 }
2365 self.connected_hardware = [device]
2366
2367 def load_hardware_map(self, map_file):
2368 with open(map_file, 'r') as stream:
2369 try:
2370 self.connected_hardware = yaml.safe_load(stream)
2371 except yaml.YAMLError as exc:
2372 print(exc)
2373 for i in self.connected_hardware:
2374 i['counter'] = 0
2375
2376 def add_configurations(self):
2377
2378 for board_root in self.board_roots:
2379 board_root = os.path.abspath(board_root)
2380
2381 debug("Reading platform configuration files under %s..." %
2382 board_root)
2383
2384 for file in glob.glob(os.path.join(board_root, "*", "*", "*.yaml")):
2385 verbose("Found plaform configuration " + file)
2386 try:
2387 platform = Platform()
2388 platform.load(file)
2389 if platform.sanitycheck:
2390 self.platforms.append(platform)
2391 if platform.default:
2392 self.default_platforms.append(platform.name)
2393
2394 except RuntimeError as e:
2395 error("E: %s: can't load: %s" % (file, e))
2396 self.load_errors += 1
2397
Anas Nashifeabaa7f2019-11-18 07:49:17 -08002398
2399 def get_all_tests(self):
2400 tests = []
2401 for _, tc in self.testcases.items():
2402 for case in tc.cases:
2403 tests.append(case)
2404
2405 return tests
2406
Anas Nashif83fc06a2019-06-22 11:04:10 -04002407 @staticmethod
2408 def get_toolchain():
2409 toolchain = os.environ.get("ZEPHYR_TOOLCHAIN_VARIANT", None) or \
2410 os.environ.get("ZEPHYR_GCC_VARIANT", None)
2411
2412 if toolchain == "gccarmemb":
2413 # Remove this translation when gccarmemb is no longer supported.
2414 toolchain = "gnuarmemb"
2415
Anas Nashifb4bdd662018-08-15 17:12:28 -05002416 try:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002417 if not toolchain:
2418 raise SanityRuntimeError("E: Variable ZEPHYR_TOOLCHAIN_VARIANT is not defined")
Anas Nashifb4bdd662018-08-15 17:12:28 -05002419 except Exception as e:
2420 print(str(e))
2421 sys.exit(2)
Anas Nashifb3311ed2017-04-13 14:44:48 -04002422
Anas Nashif83fc06a2019-06-22 11:04:10 -04002423 return toolchain
2424
2425
2426 def add_testcases(self):
2427 for root in self.roots:
2428 root = os.path.abspath(root)
2429
2430 debug("Reading test case configuration files under %s..." %root)
2431
2432 for dirpath, dirnames, filenames in os.walk(root, topdown=True):
2433 verbose("scanning %s" % dirpath)
2434 if 'sample.yaml' in filenames:
2435 filename = 'sample.yaml'
2436 elif 'testcase.yaml' in filenames:
2437 filename = 'testcase.yaml'
2438 else:
2439 continue
2440
2441 verbose("Found possible test case in " + dirpath)
2442
2443 dirnames[:] = []
2444 tc_path = os.path.join(dirpath, filename)
2445 self.add_testcase(tc_path, root)
2446
2447 def add_testcase(self, tc_data_file, root):
2448 try:
2449 parsed_data = SanityConfigParser(tc_data_file, self.tc_schema)
2450 parsed_data.load()
2451
2452 tc_path = os.path.dirname(tc_data_file)
2453 workdir = os.path.relpath(tc_path, root)
2454
2455 for name in parsed_data.tests.keys():
2456 tc = TestCase()
2457 tc.name = tc.get_unique(root, workdir, name)
2458
2459 tc_dict = parsed_data.get_test(name, testcase_valid_keys)
2460
2461 tc.source_dir = tc_path
2462 tc.yamlfile = tc_data_file
2463
2464 tc.id = name
2465 tc.type = tc_dict["type"]
2466 tc.tags = tc_dict["tags"]
2467 tc.extra_args = tc_dict["extra_args"]
2468 tc.extra_configs = tc_dict["extra_configs"]
2469 tc.arch_whitelist = tc_dict["arch_whitelist"]
2470 tc.arch_exclude = tc_dict["arch_exclude"]
2471 tc.skip = tc_dict["skip"]
2472 tc.platform_exclude = tc_dict["platform_exclude"]
2473 tc.platform_whitelist = tc_dict["platform_whitelist"]
2474 tc.toolchain_exclude = tc_dict["toolchain_exclude"]
2475 tc.toolchain_whitelist = tc_dict["toolchain_whitelist"]
2476 tc.tc_filter = tc_dict["filter"]
2477 tc.timeout = tc_dict["timeout"]
2478 tc.harness = tc_dict["harness"]
2479 tc.harness_config = tc_dict["harness_config"]
2480 tc.build_only = tc_dict["build_only"]
2481 tc.build_on_all = tc_dict["build_on_all"]
2482 tc.slow = tc_dict["slow"]
2483 tc.min_ram = tc_dict["min_ram"]
2484 tc.depends_on = tc_dict["depends_on"]
2485 tc.min_flash = tc_dict["min_flash"]
2486 tc.extra_sections = tc_dict["extra_sections"]
2487
2488 tc.parse_subcases(tc_path)
2489
2490 if tc.name:
2491 self.testcases[tc.name] = tc
2492
2493 except Exception as e:
2494 error("E: %s: can't load (skipping): %s" % (tc_data_file, e))
2495 self.load_errors += 1
2496 return False
2497
2498 return True
2499
2500
2501 def get_platform(self, name):
2502 selected_platform = None
2503 for platform in self.platforms:
2504 if platform.name == name:
2505 selected_platform = platform
2506 break
2507 return selected_platform
2508
2509 def get_last_failed(self):
2510 last_run = os.path.join(options.outdir, "sanitycheck.csv")
2511 try:
2512 if not os.path.exists(last_run):
2513 raise SanityRuntimeError("Couldn't find last sanitycheck run.: %s" %last_run)
2514 except Exception as e:
2515 print(str(e))
2516 sys.exit(2)
2517
2518 total_tests = 0
2519 with open(last_run, "r") as fp:
2520 cr = csv.DictReader(fp)
2521 instance_list = []
2522 for row in cr:
2523 total_tests += 1
2524 if row["passed"] == "True":
2525 continue
2526 test = row["test"]
2527 platform = self.get_platform(row["platform"])
2528 instance = TestInstance(self.testcases[test], platform, self.outdir)
Jan Van Winkel21212f32019-09-12 00:03:35 +02002529 instance.create_overlay(platform)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002530 instance_list.append(instance)
2531 self.add_instances(instance_list)
2532
2533 tests_to_run = len(self.instances)
2534 info("%d tests passed already, retyring %d tests" %(total_tests - tests_to_run, tests_to_run))
2535
2536 def load_from_file(self, file):
2537 try:
2538 if not os.path.exists(file):
2539 raise SanityRuntimeError(
2540 "Couldn't find input file with list of tests.")
2541 except Exception as e:
2542 print(str(e))
2543 sys.exit(2)
2544
2545 with open(file, "r") as fp:
2546 cr = csv.DictReader(fp)
2547 instance_list = []
2548 for row in cr:
2549 if row["arch"] == "arch":
2550 continue
2551 test = row["test"]
2552 platform = self.get_platform(row["platform"])
2553 instance = TestInstance(self.testcases[test], platform, self.outdir)
Jan Van Winkel21212f32019-09-12 00:03:35 +02002554 instance.create_overlay(platform)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002555 instance_list.append(instance)
2556 self.add_instances(instance_list)
2557
2558
2559 def apply_filters(self):
2560
2561 toolchain = self.get_toolchain()
2562
2563 discards = {}
2564 platform_filter = options.platform
2565 testcase_filter = run_individual_tests
2566 arch_filter = options.arch
2567 tag_filter = options.tag
2568 exclude_tag = options.exclude_tag
2569
2570 verbose("platform filter: " + str(platform_filter))
2571 verbose(" arch_filter: " + str(arch_filter))
2572 verbose(" tag_filter: " + str(tag_filter))
2573 verbose(" exclude_tag: " + str(exclude_tag))
2574
2575 default_platforms = False
2576
2577 if platform_filter:
2578 platforms = list(filter(lambda p: p.name in platform_filter, self.platforms))
2579 else:
2580 platforms = self.platforms
2581
2582 if options.all:
2583 info("Selecting all possible platforms per test case")
2584 # When --all used, any --platform arguments ignored
2585 platform_filter = []
2586 elif not platform_filter:
2587 info("Selecting default platforms per test case")
2588 default_platforms = True
2589
2590 info("Building initial testcase list...")
2591
2592 for tc_name, tc in self.testcases.items():
2593 # list of instances per testcase, aka configurations.
2594 instance_list = []
2595 for plat in platforms:
2596 instance = TestInstance(tc, plat, self.outdir)
2597
2598 if (plat.arch == "unit") != (tc.type == "unit"):
2599 # Discard silently
2600 continue
2601
2602 if options.device_testing and instance.build_only:
2603 discards[instance] = "Not runnable on device"
2604 continue
2605
2606 if tc.skip:
2607 discards[instance] = "Skip filter"
2608 continue
2609
2610 if tc.build_on_all and not platform_filter:
2611 platform_filter = []
2612
2613 if tag_filter and not tc.tags.intersection(tag_filter):
2614 discards[instance] = "Command line testcase tag filter"
2615 continue
2616
2617 if exclude_tag and tc.tags.intersection(exclude_tag):
2618 discards[instance] = "Command line testcase exclude filter"
2619 continue
2620
2621 if testcase_filter and tc_name not in testcase_filter:
2622 discards[instance] = "Testcase name filter"
2623 continue
2624
2625 if arch_filter and plat.arch not in arch_filter:
2626 discards[instance] = "Command line testcase arch filter"
2627 continue
2628
2629 if tc.arch_whitelist and plat.arch not in tc.arch_whitelist:
2630 discards[instance] = "Not in test case arch whitelist"
2631 continue
2632
2633 if tc.arch_exclude and plat.arch in tc.arch_exclude:
2634 discards[instance] = "In test case arch exclude"
2635 continue
2636
2637 if tc.platform_exclude and plat.name in tc.platform_exclude:
2638 discards[instance] = "In test case platform exclude"
2639 continue
2640
2641 if tc.toolchain_exclude and toolchain in tc.toolchain_exclude:
2642 discards[instance] = "In test case toolchain exclude"
2643 continue
2644
2645 if platform_filter and plat.name not in platform_filter:
2646 discards[instance] = "Command line platform filter"
2647 continue
2648
2649 if tc.platform_whitelist and plat.name not in tc.platform_whitelist:
2650 discards[instance] = "Not in testcase platform whitelist"
2651 continue
2652
2653 if tc.toolchain_whitelist and toolchain not in tc.toolchain_whitelist:
2654 discards[instance] = "Not in testcase toolchain whitelist"
2655 continue
2656
2657 if not plat.env_satisfied:
2658 discards[instance] = "Environment ({}) not satisfied".format(", ".join(plat.env))
2659 continue
2660
2661 if not options.force_toolchain \
2662 and toolchain and (toolchain not in plat.supported_toolchains) \
2663 and tc.type != 'unit':
2664 discards[instance] = "Not supported by the toolchain"
2665 continue
2666
2667 if plat.ram < tc.min_ram:
2668 discards[instance] = "Not enough RAM"
2669 continue
2670
2671 if tc.depends_on:
2672 dep_intersection = tc.depends_on.intersection(set(plat.supported))
2673 if dep_intersection != set(tc.depends_on):
2674 discards[instance] = "No hardware support"
2675 continue
2676
2677 if plat.flash < tc.min_flash:
2678 discards[instance] = "Not enough FLASH"
2679 continue
2680
2681 if set(plat.ignore_tags) & tc.tags:
2682 discards[instance] = "Excluded tags per platform"
2683 continue
2684
2685 # if nothing stopped us until now, it means this configuration
2686 # needs to be added.
2687 instance_list.append(instance)
2688
2689 # no configurations, so jump to next testcase
2690 if not instance_list:
2691 continue
2692
2693 # if sanitycheck was launched with no platform options at all, we
2694 # take all default platforms
2695 if default_platforms and not tc.build_on_all:
2696 if tc.platform_whitelist:
2697 a = set(self.default_platforms)
2698 b = set(tc.platform_whitelist)
2699 c = a.intersection(b)
2700 if c:
2701 aa = list( filter( lambda tc: tc.platform.name in c, instance_list))
2702 self.add_instances(aa)
2703 else:
2704 self.add_instances(instance_list[:1])
2705 else:
2706 instances = list( filter( lambda tc: tc.platform.default, instance_list))
2707 self.add_instances(instances)
2708
2709 for instance in list(filter(lambda tc: not tc.platform.default, instance_list)):
2710 discards[instance] = "Not a default test platform"
2711
2712 else:
2713 self.add_instances(instance_list)
2714
2715 for _, case in self.instances.items():
Jan Van Winkel21212f32019-09-12 00:03:35 +02002716 case.create_overlay(case.platform)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002717
2718 self.discards = discards
2719
2720 return discards
2721
2722 def add_instances(self, instance_list):
2723 for instance in instance_list:
2724 self.instances[instance.name] = instance
2725
2726 def add_tasks_to_queue(self):
2727 for instance in self.instances.values():
2728 if options.test_only:
2729 if instance.run:
2730 pipeline.put({"op": "run", "test": instance, "status": "built"})
2731 else:
2732 if instance.status not in ['passed', 'skipped']:
2733 instance.status = None
2734 pipeline.put({"op": "cmake", "test": instance})
2735
2736 return "DONE FEEDING"
2737
2738 def execute(self):
2739 def calc_one_elf_size(instance):
2740 if instance.status not in ["failed", "skipped"]:
2741 if instance.platform.type != "native":
2742 size_calc = instance.calculate_sizes()
2743 instance.metrics["ram_size"] = size_calc.get_ram_size()
2744 instance.metrics["rom_size"] = size_calc.get_rom_size()
2745 instance.metrics["unrecognized"] = size_calc.unrecognized_sections()
2746 else:
2747 instance.metrics["ram_size"] = 0
2748 instance.metrics["rom_size"] = 0
2749 instance.metrics["unrecognized"] = []
2750
2751 instance.metrics["handler_time"] = instance.handler.duration if instance.handler else 0
2752
2753 info("Adding tasks to the queue...")
2754 # We can use a with statement to ensure threads are cleaned up promptly
2755 with BoundedExecutor(bound=self.jobs, max_workers=self.jobs) as executor:
2756
2757 # start a future for a thread which sends work in through the queue
2758 future_to_test = {
2759 executor.submit(self.add_tasks_to_queue): 'FEEDER DONE'}
2760
2761 while future_to_test:
2762 # check for status of the futures which are currently working
2763 done, _ = concurrent.futures.wait(
2764 future_to_test, timeout=0.25,
2765 return_when=concurrent.futures.FIRST_COMPLETED)
2766
2767 # if there is incoming work, start a new future
2768 while not pipeline.empty():
2769 # fetch a url from the queue
2770 message = pipeline.get()
2771 test = message['test']
2772
2773 # Start the load operation and mark the future with its URL
2774 pb = ProjectBuilder(self, test)
2775 future_to_test[executor.submit(pb.process, message)] = test.name
2776
2777 # process any completed futures
2778 for future in done:
2779 test = future_to_test[future]
2780 try:
2781 data = future.result()
2782 except Exception as exc:
Anas Nashif4f043862019-11-05 06:01:49 -08002783 sys.exit('%r generated an exception: %s' % (test, exc))
2784
Anas Nashif83fc06a2019-06-22 11:04:10 -04002785 else:
2786 if data:
2787 verbose(data)
2788
2789 # remove the now completed future
2790 del future_to_test[future]
2791
2792 if options.enable_size_report and not options.cmake_only:
2793 # Parallelize size calculation
2794 executor = concurrent.futures.ThreadPoolExecutor(self.jobs)
2795 futures = [executor.submit(calc_one_elf_size, instance)
2796 for instance in self.instances.values()]
2797 concurrent.futures.wait(futures)
2798 else:
2799 for instance in self.instances.values():
2800 instance.metrics["ram_size"] = 0
2801 instance.metrics["rom_size"] = 0
2802 instance.metrics["handler_time"] = instance.handler.duration if instance.handler else 0
2803 instance.metrics["unrecognized"] = []
2804
2805
2806 def discard_report(self, filename):
2807
2808 try:
2809 if self.discards is None:
2810 raise SanityRuntimeError("apply_filters() hasn't been run!")
2811 except Exception as e:
2812 error(str(e))
2813 sys.exit(2)
2814
2815 with open(filename, "wt") as csvfile:
2816 fieldnames = ["test", "arch", "platform", "reason"]
2817 cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
2818 cw.writeheader()
2819 for instance, reason in sorted(self.discards.items()):
2820 rowdict = {"test": instance.testcase.name,
2821 "arch": instance.platform.arch,
2822 "platform": instance.platform.name,
2823 "reason": reason}
2824 cw.writerow(rowdict)
2825
2826
2827 def target_report(self, outdir):
2828 run = "Sanitycheck"
2829 eleTestsuite = None
2830
2831 platforms = {inst.platform.name for _,inst in self.instances.items()}
2832 for platform in platforms:
2833 errors = 0
2834 passes = 0
2835 fails = 0
2836 duration = 0
2837 skips = 0
2838 for _, instance in self.instances.items():
2839 if instance.platform.name != platform:
2840 continue
2841
2842 handler_time = instance.metrics.get('handler_time', 0)
2843 duration += handler_time
2844 for k in instance.results.keys():
2845 if instance.results[k] == 'PASS':
2846 passes += 1
2847 elif instance.results[k] == 'BLOCK':
2848 errors += 1
2849 elif instance.results[k] == 'SKIP':
2850 skips += 1
2851 else:
2852 fails += 1
2853
2854 eleTestsuites = ET.Element('testsuites')
2855 eleTestsuite = ET.SubElement(eleTestsuites, 'testsuite',
2856 name=run, time="%f" % duration,
2857 tests="%d" % (errors + passes + fails),
2858 failures="%d" % fails,
2859 errors="%d" % errors, skipped="%d" %skips)
2860
2861 handler_time = 0
2862
2863 # print out test results
2864 for _, instance in self.instances.items():
2865 if instance.platform.name != platform:
2866 continue
2867 handler_time = instance.metrics.get('handler_time', 0)
2868 for k in instance.results.keys():
2869 eleTestcase = ET.SubElement(
2870 eleTestsuite, 'testcase', classname="%s:%s" %(instance.platform.name, os.path.basename(instance.testcase.name)),
2871 name="%s" % (k), time="%f" %handler_time)
2872 if instance.results[k] in ['FAIL', 'BLOCK']:
2873 el = None
2874
2875 if instance.results[k] == 'FAIL':
2876 el = ET.SubElement(
2877 eleTestcase,
2878 'failure',
2879 type="failure",
2880 message="failed")
2881 elif instance.results[k] == 'BLOCK':
2882 el = ET.SubElement(
2883 eleTestcase,
2884 'error',
2885 type="failure",
2886 message="failed")
2887 p = os.path.join(options.outdir, instance.platform.name, instance.testcase.name)
2888 log_file = os.path.join(p, "handler.log")
2889
2890 if os.path.exists(log_file):
2891 with open(log_file, "rb") as f:
2892 log = f.read().decode("utf-8")
2893 filtered_string = ''.join(filter(lambda x: x in string.printable, log))
2894 el.text = filtered_string
2895
2896 elif instance.results[k] == 'SKIP':
2897 el = ET.SubElement(
2898 eleTestcase,
2899 'skipped',
2900 type="skipped",
2901 message="Skipped")
2902
2903
2904 result = ET.tostring(eleTestsuites)
2905 with open(os.path.join(outdir, platform + ".xml"), 'wb') as f:
2906 f.write(result)
2907
2908
2909 def xunit_report(self, filename):
Anas Nashifb3311ed2017-04-13 14:44:48 -04002910 fails = 0
2911 passes = 0
2912 errors = 0
Anas Nashif83fc06a2019-06-22 11:04:10 -04002913 skips = 0
2914 duration = 0
Anas Nashifb3311ed2017-04-13 14:44:48 -04002915
Anas Nashif83fc06a2019-06-22 11:04:10 -04002916 for instance in self.instances.values():
2917 handler_time = instance.metrics.get('handler_time', 0)
2918 duration += handler_time
2919 if instance.status == "failed":
2920 if instance.reason in ['build_error', 'handler_crash']:
Anas Nashifb3311ed2017-04-13 14:44:48 -04002921 errors += 1
2922 else:
2923 fails += 1
Anas Nashif83fc06a2019-06-22 11:04:10 -04002924 elif instance.status == 'skipped':
2925 skips += 1
Anas Nashifb3311ed2017-04-13 14:44:48 -04002926 else:
2927 passes += 1
2928
2929 run = "Sanitycheck"
2930 eleTestsuite = None
Anas Nashif4f028882017-12-30 11:48:43 -05002931 append = options.only_failed
Anas Nashifb3311ed2017-04-13 14:44:48 -04002932
Anas Nashif83fc06a2019-06-22 11:04:10 -04002933 # When we re-run the tests, we re-use the results and update only with
2934 # the newly run tests.
Anas Nashif0605fa32017-05-07 08:51:02 -04002935 if os.path.exists(filename) and append:
Anas Nashifb3311ed2017-04-13 14:44:48 -04002936 tree = ET.parse(filename)
2937 eleTestsuites = tree.getroot()
Anas Nashif3ba1d432017-12-05 15:28:44 -05002938 eleTestsuite = tree.findall('testsuite')[0]
Anas Nashifb3311ed2017-04-13 14:44:48 -04002939 else:
2940 eleTestsuites = ET.Element('testsuites')
Anas Nashif3ba1d432017-12-05 15:28:44 -05002941 eleTestsuite = ET.SubElement(eleTestsuites, 'testsuite',
Anas Nashif83fc06a2019-06-22 11:04:10 -04002942 name=run, time="%f" % duration,
2943 tests="%d" % (errors + passes + fails + skips),
Anas Nashif3ba1d432017-12-05 15:28:44 -05002944 failures="%d" % fails,
Anas Nashif83fc06a2019-06-22 11:04:10 -04002945 errors="%d" %(errors), skip="%s" %(skips))
Anas Nashifb3311ed2017-04-13 14:44:48 -04002946
Anas Nashif83fc06a2019-06-22 11:04:10 -04002947 for instance in self.instances.values():
Anas Nashifb3311ed2017-04-13 14:44:48 -04002948
Anas Nashif83fc06a2019-06-22 11:04:10 -04002949 # remove testcases that are a re-run
Anas Nashifb3311ed2017-04-13 14:44:48 -04002950 if append:
2951 for tc in eleTestsuite.findall('testcase'):
Anas Nashif3ba1d432017-12-05 15:28:44 -05002952 if tc.get('classname') == "%s:%s" % (
Anas Nashif83fc06a2019-06-22 11:04:10 -04002953 instance.platform.name, instance.testcase.name):
Anas Nashifb3311ed2017-04-13 14:44:48 -04002954 eleTestsuite.remove(tc)
2955
Anas Nashif83fc06a2019-06-22 11:04:10 -04002956 handler_time = 0
2957 if instance.status != "failed" and instance.handler:
2958 handler_time = instance.metrics.get("handler_time", 0)
Anas Nashifb3311ed2017-04-13 14:44:48 -04002959
Anas Nashif3ba1d432017-12-05 15:28:44 -05002960 eleTestcase = ET.SubElement(
2961 eleTestsuite, 'testcase', classname="%s:%s" %
Anas Nashif83fc06a2019-06-22 11:04:10 -04002962 (instance.platform.name, instance.testcase.name), name="%s" %
2963 (instance.testcase.name), time="%f" %handler_time)
2964
2965 if instance.status == "failed":
Anas Nashif3ba1d432017-12-05 15:28:44 -05002966 failure = ET.SubElement(
2967 eleTestcase,
2968 'failure',
2969 type="failure",
Anas Nashif83fc06a2019-06-22 11:04:10 -04002970 message=instance.reason)
2971 p = ("%s/%s/%s" % (options.outdir, instance.platform.name, instance.testcase.name))
Anas Nashifb3311ed2017-04-13 14:44:48 -04002972 bl = os.path.join(p, "build.log")
Anas Nashifc96c90a2019-02-05 07:38:32 -05002973 hl = os.path.join(p, "handler.log")
2974 log_file = bl
Anas Nashif83fc06a2019-06-22 11:04:10 -04002975 if instance.reason != 'Build error':
Anas Nashifc96c90a2019-02-05 07:38:32 -05002976 if os.path.exists(hl):
2977 log_file = hl
2978 else:
2979 log_file = bl
Anas Nashifaccc8eb2017-05-01 16:33:43 -04002980
Anas Nashifc96c90a2019-02-05 07:38:32 -05002981 if os.path.exists(log_file):
2982 with open(log_file, "rb") as f:
Anas Nashif712d3452017-12-29 22:09:03 -05002983 log = f.read().decode("utf-8")
Anas Nashifa4c368e2018-10-15 09:45:59 -04002984 filtered_string = ''.join(filter(lambda x: x in string.printable, log))
2985 failure.text = filtered_string
Anas Nashifba4643b2018-09-23 09:41:59 -05002986 f.close()
Anas Nashif83fc06a2019-06-22 11:04:10 -04002987 elif instance.status == "skipped":
2988 ET.SubElement( eleTestcase, 'skipped', type="skipped", message="Skipped")
Anas Nashifb3311ed2017-04-13 14:44:48 -04002989
2990 result = ET.tostring(eleTestsuites)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002991 with open(filename, 'wb') as report:
2992 report.write(result)
Anas Nashifb3311ed2017-04-13 14:44:48 -04002993
Andrew Boie6acbe632015-07-17 12:03:52 -07002994
Anas Nashif83fc06a2019-06-22 11:04:10 -04002995 def csv_report(self, filename):
Andrew Boie08ce5a52016-02-22 13:28:10 -08002996 with open(filename, "wt") as csvfile:
Andrew Boie6acbe632015-07-17 12:03:52 -07002997 fieldnames = ["test", "arch", "platform", "passed", "status",
Anas Nashif83fc06a2019-06-22 11:04:10 -04002998 "extra_args", "handler", "handler_time", "ram_size",
Andrew Boie6acbe632015-07-17 12:03:52 -07002999 "rom_size"]
3000 cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
3001 cw.writeheader()
Anas Nashif83fc06a2019-06-22 11:04:10 -04003002 for instance in sorted(self.instances.values()):
3003 rowdict = {"test": instance.testcase.name,
3004 "arch": instance.platform.arch,
3005 "platform": instance.platform.name,
3006 "extra_args": " ".join(instance.testcase.extra_args),
3007 "handler": instance.platform.simulation}
3008
3009 if instance.status in ["failed", "timeout"]:
Andrew Boie6acbe632015-07-17 12:03:52 -07003010 rowdict["passed"] = False
Anas Nashif83fc06a2019-06-22 11:04:10 -04003011 rowdict["status"] = instance.reason
Andrew Boie6acbe632015-07-17 12:03:52 -07003012 else:
3013 rowdict["passed"] = True
Anas Nashif83fc06a2019-06-22 11:04:10 -04003014 if instance.handler:
3015 rowdict["handler_time"] = instance.metrics.get("handler_time", 0)
3016 ram_size = instance.metrics.get("ram_size", 0)
3017 rom_size = instance.metrics.get("rom_size", 0)
3018 rowdict["ram_size"] = ram_size
3019 rowdict["rom_size"] = rom_size
Andrew Boie6acbe632015-07-17 12:03:52 -07003020 cw.writerow(rowdict)
3021
3022
Anas Nashif19ca7832019-11-18 08:16:21 -08003023 def get_testcase(self, identifier):
3024 results = []
3025 for _, tc in self.testcases.items():
3026 for case in tc.cases:
3027 if case == identifier:
3028 results.append(tc)
3029 return results
3030
3031
Andrew Boie6acbe632015-07-17 12:03:52 -07003032def parse_arguments():
3033
Anas Nashif3ba1d432017-12-05 15:28:44 -05003034 parser = argparse.ArgumentParser(
3035 description=__doc__,
3036 formatter_class=argparse.RawDescriptionHelpFormatter)
Genaro Saucedo Tejada28bba922016-10-24 18:00:58 -05003037 parser.fromfile_prefix_chars = "+"
Andrew Boie6acbe632015-07-17 12:03:52 -07003038
Marc Herbert932a33a2019-03-12 11:37:53 -07003039 case_select = parser.add_argument_group("Test case selection",
3040 """
3041Artificially long but functional example:
3042 $ ./scripts/sanitycheck -v \\
Marc Herberte5cedca2019-04-08 14:02:34 -07003043 --testcase-root tests/ztest/base \\
3044 --testcase-root tests/kernel \\
Marc Herbert932a33a2019-03-12 11:37:53 -07003045 --test tests/ztest/base/testing.ztest.verbose_0 \\
3046 --test tests/kernel/fifo/fifo_api/kernel.fifo.poll
3047
3048 "kernel.fifo.poll" is one of the test section names in
3049 __/fifo_api/testcase.yaml
3050 """)
Marc Herbertedf17592019-03-08 12:39:11 -08003051
Anas Nashif07d54c02018-07-21 19:29:08 -05003052 parser.add_argument("--force-toolchain", action="store_true",
3053 help="Do not filter based on toolchain, use the set "
3054 " toolchain unconditionally")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003055 parser.add_argument(
3056 "-p", "--platform", action="append",
3057 help="Platform filter for testing. This option may be used multiple "
3058 "times. Testcases will only be built/run on the platforms "
3059 "specified. If this option is not used, then platforms marked "
3060 "as default in the platform metadata file will be chosen "
3061 "to build and test. ")
3062 parser.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003063 "-a", "--arch", action="append",
3064 help="Arch filter for testing. Takes precedence over --platform. "
3065 "If unspecified, test all arches. Multiple invocations "
3066 "are treated as a logical 'or' relationship")
3067 parser.add_argument(
3068 "-t", "--tag", action="append",
3069 help="Specify tags to restrict which tests to run by tag value. "
3070 "Default is to not do any tag filtering. Multiple invocations "
3071 "are treated as a logical 'or' relationship")
Anas Nashifdfa86e22016-10-24 17:08:56 -04003072 parser.add_argument("-e", "--exclude-tag", action="append",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003073 help="Specify tags of tests that should not run. "
3074 "Default is to run all tests with all tags.")
Marc Herbertedf17592019-03-08 12:39:11 -08003075 case_select.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003076 "-f",
3077 "--only-failed",
3078 action="store_true",
3079 help="Run only those tests that failed the previous sanity check "
3080 "invocation.")
Anas Nashif83fc06a2019-06-22 11:04:10 -04003081
Anas Nashif3ba1d432017-12-05 15:28:44 -05003082 parser.add_argument(
Anas Nashif83fc06a2019-06-22 11:04:10 -04003083 "--retry-failed", type=int, default=0,
3084 help="Retry failing tests again, up to the number of times specified.")
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003085
Marc Herbert0c465bb2019-03-11 17:28:36 -07003086 test_xor_subtest = case_select.add_mutually_exclusive_group()
3087
3088 test_xor_subtest.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003089 "-s", "--test", action="append",
3090 help="Run only the specified test cases. These are named by "
Marc Herberte5cedca2019-04-08 14:02:34 -07003091 "<path/relative/to/Zephyr/base/section.name.in.testcase.yaml>")
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003092
Marc Herbert0c465bb2019-03-11 17:28:36 -07003093 test_xor_subtest.add_argument(
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003094 "--sub-test", action="append",
Marc Herbert932a33a2019-03-12 11:37:53 -07003095 help="""Recursively find sub-test functions and run the entire
3096 test section where they were found, including all sibling test
3097 functions. Sub-tests are named by:
3098 section.name.in.testcase.yaml.function_name_without_test_prefix
3099 Example: kernel.fifo.poll.fifo_loop
3100 """)
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003101
Anas Nashif3ba1d432017-12-05 15:28:44 -05003102 parser.add_argument(
3103 "-l", "--all", action="store_true",
3104 help="Build/test on all platforms. Any --platform arguments "
3105 "ignored.")
Andrew Boie6acbe632015-07-17 12:03:52 -07003106
Anas Nashif3ba1d432017-12-05 15:28:44 -05003107 parser.add_argument(
Anas Nashif83fc06a2019-06-22 11:04:10 -04003108 "-o", "--report-dir",
3109 help="""Output reports containing results of the test run into the
3110 specified directory.
3111 The output will be both in CSV and JUNIT format
3112 (sanitycheck.csv and sanitycheck.xml).
3113 """)
3114
Anas Nashif3ba1d432017-12-05 15:28:44 -05003115 parser.add_argument(
Anas Nashif83fc06a2019-06-22 11:04:10 -04003116 "--report-name",
3117 help="""Create a report with a custom name.
3118 """)
3119
3120 parser.add_argument("--detailed-report",
3121 action="store",
3122 metavar="FILENAME",
3123 help="""Generate a junit report with detailed testcase results.
3124 Unlike the CSV file produced by --testcase-report, this XML
3125 report includes only tests which have run and none which were
3126 merely built. If an image with multiple tests crashes early then
3127 later tests are not accounted for either.""")
3128
3129 parser.add_argument("--report-excluded",
3130 action="store_true",
3131 help="""List all tests that are never run based on current scope and
3132 coverage. If you are looking for accurate results, run this with
3133 --all, but this will take a while...""")
3134
Daniel Leung7f850102016-04-08 11:07:32 -07003135 parser.add_argument("--compare-report",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003136 help="Use this report file for size comparison")
Daniel Leung7f850102016-04-08 11:07:32 -07003137
Anas Nashif3ba1d432017-12-05 15:28:44 -05003138 parser.add_argument(
3139 "-B", "--subset",
3140 help="Only run a subset of the tests, 1/4 for running the first 25%%, "
3141 "3/5 means run the 3rd fifth of the total. "
3142 "This option is useful when running a large number of tests on "
3143 "different hosts to speed up execution time.")
Anas Nashifa8a13882017-12-30 13:01:06 -05003144
3145 parser.add_argument(
3146 "-N", "--ninja", action="store_true",
Anas Nashif25f6ab62018-03-06 07:15:11 -06003147 help="Use the Ninja generator with CMake")
Anas Nashifa8a13882017-12-30 13:01:06 -05003148
Anas Nashif3ba1d432017-12-05 15:28:44 -05003149 parser.add_argument(
3150 "-y", "--dry-run", action="store_true",
Anas Nashif12d8cce2019-11-20 03:47:27 -08003151 help="""Create the filtered list of test cases, but don't actually
3152 run them. Useful if you're just interested in the discard report
3153 generated for every run and saved in the specified output
3154 directory (sanitycheck_discard.csv).
3155 """)
Andrew Boie6acbe632015-07-17 12:03:52 -07003156
Anas Nashif75547e22018-02-24 08:32:14 -06003157 parser.add_argument("--list-tags", action="store_true",
3158 help="list all tags in selected tests")
3159
Marc Herbertedf17592019-03-08 12:39:11 -08003160 case_select.add_argument("--list-tests", action="store_true",
Marc Herbert932a33a2019-03-12 11:37:53 -07003161 help="""List of all sub-test functions recursively found in
3162 all --testcase-root arguments. Note different sub-tests can share
3163 the same section name and come from different directories.
3164 The output is flattened and reports --sub-test names only,
3165 not their directories. For instance net.socket.getaddrinfo_ok
3166 and net.socket.fd_set belong to different directories.
3167 """)
Anas Nashifc0149cc2018-04-14 23:12:58 -05003168
Anas Nashif19ca7832019-11-18 08:16:21 -08003169 case_select.add_argument("--list-test-duplicates", action="store_true",
3170 help="""List tests with duplicate identifiers.
3171 """)
3172
Anas Nashif5d6e7eb2018-06-01 09:51:08 -05003173 parser.add_argument("--export-tests", action="store",
3174 metavar="FILENAME",
3175 help="Export tests case meta-data to a file in CSV format.")
3176
Anas Nashife0a6a0b2018-02-15 20:07:24 -06003177
Anas Nashif654ec5982019-04-11 08:38:21 -04003178 parser.add_argument("--timestamps",
3179 action="store_true",
3180 help="Print all messages with time stamps")
3181
Anas Nashif3ba1d432017-12-05 15:28:44 -05003182 parser.add_argument(
3183 "-r", "--release", action="store_true",
3184 help="Update the benchmark database with the results of this test "
3185 "run. Intended to be run by CI when tagging an official "
3186 "release. This database is used as a basis for comparison "
3187 "when looking for deltas in metrics such as footprint")
Andrew Boie6acbe632015-07-17 12:03:52 -07003188 parser.add_argument("-w", "--warnings-as-errors", action="store_true",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003189 help="Treat warning conditions as errors")
3190 parser.add_argument(
3191 "-v",
3192 "--verbose",
3193 action="count",
3194 default=0,
3195 help="Emit debugging information, call multiple times to increase "
3196 "verbosity")
3197 parser.add_argument(
3198 "-i", "--inline-logs", action="store_true",
3199 help="Upon test failure, print relevant log data to stdout "
3200 "instead of just a path to it")
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -08003201 parser.add_argument("--log-file", metavar="FILENAME", action="store",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003202 help="log also to file")
3203 parser.add_argument(
3204 "-m", "--last-metrics", action="store_true",
3205 help="Instead of comparing metrics from the last --release, "
3206 "compare with the results of the previous sanity check "
3207 "invocation")
3208 parser.add_argument(
3209 "-u",
3210 "--no-update",
3211 action="store_true",
3212 help="do not update the results of the last run of the sanity "
3213 "checks")
Marc Herbertedf17592019-03-08 12:39:11 -08003214 case_select.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003215 "-F",
3216 "--load-tests",
3217 metavar="FILENAME",
3218 action="store",
Marc Herbert932a33a2019-03-12 11:37:53 -07003219 help="Load list of tests and platforms to be run from file.")
Anas Nashifbd166f42017-09-02 12:32:08 -04003220
Marc Herbertedf17592019-03-08 12:39:11 -08003221 case_select.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003222 "-E",
3223 "--save-tests",
3224 metavar="FILENAME",
3225 action="store",
Marc Herbert682961a2019-03-20 16:48:49 -07003226 help="Append list of tests and platforms to be run to file.")
Anas Nashifbd166f42017-09-02 12:32:08 -04003227
Andy Doancbecadd2019-02-08 10:19:10 -06003228 test_or_build = parser.add_mutually_exclusive_group()
3229 test_or_build.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003230 "-b", "--build-only", action="store_true",
3231 help="Only build the code, do not execute any of it in QEMU")
Anas Nashif83fc06a2019-06-22 11:04:10 -04003232
Andy Doancbecadd2019-02-08 10:19:10 -06003233 test_or_build.add_argument(
3234 "--test-only", action="store_true",
3235 help="""Only run device tests with current artifacts, do not build
3236 the code""")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003237 parser.add_argument(
Kumar Gala84cf9dc2019-06-15 09:36:06 -05003238 "--cmake-only", action="store_true",
Anas Nashif83fc06a2019-06-22 11:04:10 -04003239 help="Only run cmake, do not build or run.")
Kumar Gala84cf9dc2019-06-15 09:36:06 -05003240
3241 parser.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003242 "-j", "--jobs", type=int,
Marc Herbert9e573382019-07-03 12:49:42 -07003243 help="Number of jobs for building, defaults to number of CPU threads, "
3244 "overcommited by factor 2 when --build-only")
Anas Nashif73440ea2018-02-19 10:57:03 -06003245
3246 parser.add_argument(
Anas Nashif424a3db2018-02-20 08:37:24 -06003247 "--show-footprint", action="store_true",
3248 help="Show footprint statistics and deltas since last release."
3249 )
3250 parser.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003251 "-H", "--footprint-threshold", type=float, default=5,
3252 help="When checking test case footprint sizes, warn the user if "
3253 "the new app size is greater then the specified percentage "
3254 "from the last release. Default is 5. 0 to warn on any "
3255 "increase on app size")
3256 parser.add_argument(
3257 "-D", "--all-deltas", action="store_true",
3258 help="Show all footprint deltas, positive or negative. Implies "
3259 "--footprint-threshold=0")
Håkon Øye Amundsende223cc2018-04-25 06:24:25 +00003260 parser.add_argument(
3261 "-O", "--outdir",
Anas Nashif83fc06a2019-06-22 11:04:10 -04003262 default=os.path.join(os.getcwd(),"sanity-out"),
Håkon Øye Amundsende223cc2018-04-25 06:24:25 +00003263 help="Output directory for logs and binaries. "
Anas Nashiff114a132018-11-20 11:51:34 -05003264 "Default is 'sanity-out' in the current directory. "
Håkon Øye Amundsende223cc2018-04-25 06:24:25 +00003265 "This directory will be deleted unless '--no-clean' is set.")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003266 parser.add_argument(
3267 "-n", "--no-clean", action="store_true",
3268 help="Do not delete the outdir before building. Will result in "
3269 "faster compilation since builds will be incremental")
Marc Herbertedf17592019-03-08 12:39:11 -08003270 case_select.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003271 "-T", "--testcase-root", action="append", default=[],
3272 help="Base directory to recursively search for test cases. All "
3273 "testcase.yaml files under here will be processed. May be "
Marc Herbert932a33a2019-03-12 11:37:53 -07003274 "called multiple times. Defaults to the 'samples/' and "
3275 "'tests/' directories at the base of the Zephyr tree.")
Anas Nashif7dd19ea2018-11-14 08:46:49 -05003276
Anas Nashif3ba1d432017-12-05 15:28:44 -05003277 board_root_list = ["%s/boards" % ZEPHYR_BASE,
3278 "%s/scripts/sanity_chk/boards" % ZEPHYR_BASE]
Anas Nashif7dd19ea2018-11-14 08:46:49 -05003279
Anas Nashif3ba1d432017-12-05 15:28:44 -05003280 parser.add_argument(
Thomas Stilwell53105102019-10-23 14:25:09 +02003281 "-A", "--board-root", action="append", default=board_root_list,
Anas Nashif83fc06a2019-06-22 11:04:10 -04003282 help="""Directory to search for board configuration files. All .yaml
3283files in the directory will be processed. The directory should have the same
3284structure in the main Zephyr tree: boards/<arch>/<board_name>/""")
3285
Anas Nashif3ba1d432017-12-05 15:28:44 -05003286 parser.add_argument(
3287 "-z", "--size", action="append",
3288 help="Don't run sanity checks. Instead, produce a report to "
3289 "stdout detailing RAM/ROM sizes on the specified filenames. "
3290 "All other command line arguments ignored.")
3291 parser.add_argument(
3292 "-S", "--enable-slow", action="store_true",
3293 help="Execute time-consuming test cases that have been marked "
3294 "as 'slow' in testcase.yaml. Normally these are only built.")
Sebastian Bøe03ed0952018-11-13 13:36:19 +01003295 parser.add_argument(
3296 "--disable-unrecognized-section-test", action="store_true",
3297 default=False,
3298 help="Skip the 'unrecognized section' test.")
Andrew Boie55121052016-07-20 11:52:04 -07003299 parser.add_argument("-R", "--enable-asserts", action="store_true",
Kumar Gala0d48cbe2018-01-26 10:22:20 -06003300 default=True,
Andrew Boie29599f62018-05-24 13:33:09 -07003301 help="deprecated, left for compatibility")
Kumar Gala0d48cbe2018-01-26 10:22:20 -06003302 parser.add_argument("--disable-asserts", action="store_false",
3303 dest="enable_asserts",
Andrew Boie29599f62018-05-24 13:33:09 -07003304 help="deprecated, left for compatibility")
Anas Nashife3febe92016-11-30 14:25:44 -05003305 parser.add_argument("-Q", "--error-on-deprecations", action="store_false",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003306 help="Error on deprecation warnings.")
Anas Nashif83fc06a2019-06-22 11:04:10 -04003307 parser.add_argument("--enable-size-report", action="store_true",
3308 help="Enable expensive computation of RAM/ROM segment sizes.")
Sebastian Bøec2182612017-11-09 12:25:02 +01003309
3310 parser.add_argument(
3311 "-x", "--extra-args", action="append", default=[],
Alberto Escolar Piedras25e06362018-02-10 17:40:33 +01003312 help="""Extra CMake cache entries to define when building test cases.
3313 May be called multiple times. The key-value entries will be
Sebastian Bøec2182612017-11-09 12:25:02 +01003314 prefixed with -D before being passed to CMake.
3315
3316 E.g
3317 "sanitycheck -x=USE_CCACHE=0"
3318 will translate to
3319 "cmake -DUSE_CCACHE=0"
3320
3321 which will ultimately disable ccache.
3322 """
3323 )
Michael Scott421ce462019-06-18 09:37:46 -07003324
Andy Doan79c48842019-02-08 10:09:04 -06003325 parser.add_argument(
Anas Nashif83fc06a2019-06-22 11:04:10 -04003326 "--device-testing", action="store_true",
3327 help="Test on device directly. Specify the serial device to "
3328 "use with the --device-serial option.")
3329
3330 parser.add_argument(
3331 "-X", "--fixture", action="append", default=[],
3332 help="Specify a fixture that a board might support")
3333 parser.add_argument(
3334 "--device-serial",
3335 help="Serial device for accessing the board (e.g., /dev/ttyACM0)")
3336
3337 parser.add_argument("--generate-hardware-map",
3338 help="""Probe serial devices connected to this platform
3339 and create a hardware map file to be used with
3340 --device-testing
3341 """)
3342
3343 parser.add_argument("--hardware-map",
3344 help="""Load hardware map from a file. This will be used
3345 for testing on hardware that is listed in the file.
3346 """)
3347
3348 parser.add_argument(
Andy Doan79c48842019-02-08 10:09:04 -06003349 "--west-flash", nargs='?', const=[],
3350 help="""Uses west instead of ninja or make to flash when running with
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +02003351 --device-testing. Supports comma-separated argument list.
Andy Doan79c48842019-02-08 10:09:04 -06003352
Michael Scott4ca54392019-07-09 14:21:30 -07003353 E.g "sanitycheck --device-testing --device-serial /dev/ttyACM0
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +02003354 --west-flash="--board-id=foobar,--erase"
3355 will translate to "west flash -- --board-id=foobar --erase"
Michael Scott4ca54392019-07-09 14:21:30 -07003356
3357 NOTE: device-testing must be enabled to use this option.
Andy Doan79c48842019-02-08 10:09:04 -06003358 """
3359 )
Michael Scott421ce462019-06-18 09:37:46 -07003360 parser.add_argument(
3361 "--west-runner",
3362 help="""Uses the specified west runner instead of default when running
3363 with --west-flash.
3364
3365 E.g "sanitycheck --device-testing --device-serial /dev/ttyACM0
3366 --west-flash --west-runner=pyocd"
3367 will translate to "west flash --runner pyocd"
3368
3369 NOTE: west-flash must be enabled to use this option.
3370 """
3371 )
Jan Van Winkel21212f32019-09-12 00:03:35 +02003372
3373 valgrind_asan_group = parser.add_mutually_exclusive_group()
3374
3375 valgrind_asan_group.add_argument(
Anas Nashifc1ea4522019-10-11 07:32:45 -07003376 "--enable-valgrind", action="store_true",
3377 help="""Run binary through valgrind and check for several memory access
Jan Van Winkel21212f32019-09-12 00:03:35 +02003378 errors. Valgrind needs to be installed on the host. This option only
Anas Nashifc1ea4522019-10-11 07:32:45 -07003379 works with host binaries such as those generated for the native_posix
Jan Van Winkel21212f32019-09-12 00:03:35 +02003380 configuration and is mutual exclusive with --enable-asan.
3381 """)
3382
3383 valgrind_asan_group.add_argument(
3384 "--enable-asan", action="store_true",
3385 help="""Enable address sanitizer to check for several memory access
3386 errors. Libasan needs to be installed on the host. This option only
3387 works with host binaries such as those generated for the native_posix
3388 configuration and is mutual exclusive with --enable-valgrind.
3389 """)
3390
3391 parser.add_argument(
3392 "--enable-lsan", action="store_true",
3393 help="""Enable leak sanitizer to check for heap memory leaks.
3394 Libasan needs to be installed on the host. This option only
3395 works with host binaries such as those generated for the native_posix
3396 configuration and when --enable-asan is given.
Anas Nashifc1ea4522019-10-11 07:32:45 -07003397 """)
Andrew Boie8047a6f2019-07-02 15:43:29 -07003398
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02003399 parser.add_argument("--enable-coverage", action="store_true",
Anas Nashif8d72bb92018-11-07 23:05:42 -05003400 help="Enable code coverage using gcov.")
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02003401
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003402 parser.add_argument("-C", "--coverage", action="store_true",
Andrew Boie8047a6f2019-07-02 15:43:29 -07003403 help="Generate coverage reports. Implies "
3404 "--enable_coverage and --enable-slow")
Andrew Boie6acbe632015-07-17 12:03:52 -07003405
Andrew Boie8047a6f2019-07-02 15:43:29 -07003406 parser.add_argument("--coverage-platform", action="append", default=[],
Anas Nashifdbd76492018-11-23 20:24:19 -05003407 help="Plarforms to run coverage reports on. "
Andrew Boie8047a6f2019-07-02 15:43:29 -07003408 "This option may be used multiple times. "
3409 "Default to what was selected with --platform.")
Anas Nashifdbd76492018-11-23 20:24:19 -05003410
Anas Nashif83fc06a2019-06-22 11:04:10 -04003411 parser.add_argument("--gcov-tool", default=None,
3412 help="Path to the gcov tool to use for code coverage "
3413 "reports")
3414
Andrew Boie6acbe632015-07-17 12:03:52 -07003415 return parser.parse_args()
3416
Anas Nashif3ba1d432017-12-05 15:28:44 -05003417
Andrew Boie6acbe632015-07-17 12:03:52 -07003418def log_info(filename):
Mazen NEIFER8f1dd3a2017-02-04 14:32:04 +01003419 filename = os.path.relpath(os.path.realpath(filename))
Anas Nashif83fc06a2019-06-22 11:04:10 -04003420 if options.inline_logs:
Andrew Boie08ce5a52016-02-22 13:28:10 -08003421 info("{:-^100}".format(filename))
Andrew Boied01535c2017-01-10 13:15:02 -08003422
3423 try:
3424 with open(filename) as fp:
3425 data = fp.read()
3426 except Exception as e:
3427 data = "Unable to read log data (%s)\n" % (str(e))
3428
3429 sys.stdout.write(data)
3430 if log_file:
3431 log_file.write(data)
Andrew Boie08ce5a52016-02-22 13:28:10 -08003432 info("{:-^100}".format(filename))
Andrew Boie6acbe632015-07-17 12:03:52 -07003433 else:
Anas Nashifc1ea4522019-10-11 07:32:45 -07003434 info("\n\tsee: " + COLOR_YELLOW + filename + COLOR_NORMAL)
Andrew Boie6acbe632015-07-17 12:03:52 -07003435
Ulf Magnussone73d2862019-10-29 11:54:02 +01003436
3437def log_info_file(instance):
3438 build_dir = instance.build_dir
3439 h_log = "{}/handler.log".format(build_dir)
3440 b_log = "{}/build.log".format(build_dir)
3441 v_log = "{}/valgrind.log".format(build_dir)
3442
3443 if os.path.exists(v_log) and "Valgrind" in instance.reason:
3444 log_info("{}".format(v_log))
3445 elif os.path.exists(h_log):
3446 log_info("{}".format(h_log))
3447 else:
3448 log_info("{}".format(b_log))
3449
3450
Andrew Boiebbd670c2015-08-17 13:16:11 -07003451def size_report(sc):
3452 info(sc.filename)
Andrew Boie73b4ee62015-10-07 11:33:22 -07003453 info("SECTION NAME VMA LMA SIZE HEX SZ TYPE")
Andrew Boie9882dcd2015-10-07 14:25:51 -07003454 for i in range(len(sc.sections)):
3455 v = sc.sections[i]
3456
Andrew Boie73b4ee62015-10-07 11:33:22 -07003457 info("%-17s 0x%08x 0x%08x %8d 0x%05x %-7s" %
3458 (v["name"], v["virt_addr"], v["load_addr"], v["size"], v["size"],
3459 v["type"]))
Andrew Boie9882dcd2015-10-07 14:25:51 -07003460
Andrew Boie73b4ee62015-10-07 11:33:22 -07003461 info("Totals: %d bytes (ROM), %d bytes (RAM)" %
Anas Nashif3ba1d432017-12-05 15:28:44 -05003462 (sc.rom_size, sc.ram_size))
Andrew Boiebbd670c2015-08-17 13:16:11 -07003463 info("")
3464
Anas Nashiff29087e2019-01-25 09:37:38 -05003465def retrieve_gcov_data(intput_file):
Anas Nashifdb9592a2018-10-08 10:19:41 -04003466 if VERBOSE:
3467 print("Working on %s" %intput_file)
3468 extracted_coverage_info = {}
3469 capture_data = False
Anas Nashiff29087e2019-01-25 09:37:38 -05003470 capture_complete = False
Anas Nashifdb9592a2018-10-08 10:19:41 -04003471 with open(intput_file, 'r') as fp:
3472 for line in fp.readlines():
3473 if re.search("GCOV_COVERAGE_DUMP_START", line):
3474 capture_data = True
3475 continue
3476 if re.search("GCOV_COVERAGE_DUMP_END", line):
Anas Nashiff29087e2019-01-25 09:37:38 -05003477 capture_complete = True
Anas Nashifdb9592a2018-10-08 10:19:41 -04003478 break
3479 # Loop until the coverage data is found.
3480 if not capture_data:
3481 continue
3482 if line.startswith("*"):
3483 sp = line.split("<")
3484 if len(sp) > 1:
3485 # Remove the leading delimiter "*"
3486 file_name = sp[0][1:]
3487 # Remove the trailing new line char
3488 hex_dump = sp[1][:-1]
3489 else:
3490 continue
3491 else:
3492 continue
3493 extracted_coverage_info.update({file_name:hex_dump})
Anas Nashiff29087e2019-01-25 09:37:38 -05003494 if not capture_data:
3495 capture_complete = True
3496 return {'complete': capture_complete, 'data': extracted_coverage_info}
Anas Nashifdb9592a2018-10-08 10:19:41 -04003497
3498def create_gcda_files(extracted_coverage_info):
3499 if VERBOSE:
3500 print("Generating gcda files")
3501 for filename, hexdump_val in extracted_coverage_info.items():
3502 # if kobject_hash is given for coverage gcovr fails
3503 # hence skipping it problem only in gcovr v4.1
3504 if "kobject_hash" in filename:
3505 filename = (filename[:-4]) +"gcno"
3506 try:
3507 os.remove(filename)
Anas Nashif83fc06a2019-06-22 11:04:10 -04003508 except Exception:
Anas Nashifdb9592a2018-10-08 10:19:41 -04003509 pass
3510 continue
3511
3512 with open(filename, 'wb') as fp:
3513 fp.write(bytes.fromhex(hexdump_val))
Anas Nashif3ba1d432017-12-05 15:28:44 -05003514
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003515def generate_coverage(outdir, ignores):
Anas Nashifdb9592a2018-10-08 10:19:41 -04003516
3517 for filename in glob.glob("%s/**/handler.log" %outdir, recursive=True):
Anas Nashiff29087e2019-01-25 09:37:38 -05003518 gcov_data = retrieve_gcov_data(filename)
3519 capture_complete = gcov_data['complete']
3520 extracted_coverage_info = gcov_data['data']
3521 if capture_complete:
3522 create_gcda_files(extracted_coverage_info)
3523 verbose("Gcov data captured: {}".format(filename))
3524 else:
3525 error("Gcov data capture incomplete: {}".format(filename))
Anas Nashifdb9592a2018-10-08 10:19:41 -04003526
Anas Nashif47cf4bf2019-01-24 21:50:59 -05003527 gcov_tool = options.gcov_tool
3528
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003529 with open(os.path.join(outdir, "coverage.log"), "a") as coveragelog:
3530 coveragefile = os.path.join(outdir, "coverage.info")
3531 ztestfile = os.path.join(outdir, "ztest.info")
Anas Nashif47cf4bf2019-01-24 21:50:59 -05003532 subprocess.call(["lcov", "--gcov-tool", gcov_tool,
3533 "--capture", "--directory", outdir,
3534 "--rc", "lcov_branch_coverage=1",
3535 "--output-file", coveragefile], stdout=coveragelog)
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003536 # We want to remove tests/* and tests/ztest/test/* but save tests/ztest
Anas Nashif47cf4bf2019-01-24 21:50:59 -05003537 subprocess.call(["lcov", "--gcov-tool", gcov_tool, "--extract", coveragefile,
Anas Nashif3ba1d432017-12-05 15:28:44 -05003538 os.path.join(ZEPHYR_BASE, "tests", "ztest", "*"),
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003539 "--output-file", ztestfile,
3540 "--rc", "lcov_branch_coverage=1"], stdout=coveragelog)
3541
Anas Nashif3cbffef2018-11-07 23:50:54 -05003542 if os.path.exists(ztestfile) and os.path.getsize(ztestfile) > 0:
Anas Nashif47cf4bf2019-01-24 21:50:59 -05003543 subprocess.call(["lcov", "--gcov-tool", gcov_tool, "--remove", ztestfile,
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003544 os.path.join(ZEPHYR_BASE, "tests/ztest/test/*"),
3545 "--output-file", ztestfile,
3546 "--rc", "lcov_branch_coverage=1"],
3547 stdout=coveragelog)
Anas Nashif83fc06a2019-06-22 11:04:10 -04003548 files = [coveragefile, ztestfile]
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003549 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -04003550 files = [coveragefile]
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003551
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003552 for i in ignores:
Anas Nashif3ba1d432017-12-05 15:28:44 -05003553 subprocess.call(
Anas Nashif47cf4bf2019-01-24 21:50:59 -05003554 ["lcov", "--gcov-tool", gcov_tool, "--remove",
3555 coveragefile, i, "--output-file",
3556 coveragefile, "--rc", "lcov_branch_coverage=1"],
Anas Nashif3ba1d432017-12-05 15:28:44 -05003557 stdout=coveragelog)
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003558
Alberto Escolar Piedras834b86f2019-02-03 15:48:07 +01003559 #The --ignore-errors source option is added to avoid it exiting due to
3560 #samples/application_development/external_lib/
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003561 ret = subprocess.call(["genhtml", "--legend", "--branch-coverage",
Alberto Escolar Piedras834b86f2019-02-03 15:48:07 +01003562 "--ignore-errors", "source",
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003563 "-output-directory",
3564 os.path.join(outdir, "coverage")] + files,
3565 stdout=coveragelog)
3566 if ret==0:
3567 info("HTML report generated: %s"%
Anas Nashif83fc06a2019-06-22 11:04:10 -04003568 os.path.join(outdir, "coverage","index.html"))
Anas Nashif3ba1d432017-12-05 15:28:44 -05003569
Anas Nashif83fc06a2019-06-22 11:04:10 -04003570def get_generator():
3571 if options.ninja:
3572 generator_cmd = "ninja"
3573 generator = "Ninja"
3574 else:
3575 generator_cmd = "make"
3576 generator = "Unix Makefiles"
3577 return generator_cmd, generator
3578
3579
3580def export_tests(filename, tests):
3581 with open(filename, "wt") as csvfile:
3582 fieldnames = ['section', 'subsection', 'title', 'reference']
3583 cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
3584 for test in tests:
3585 data = test.split(".")
Anas Nashif19ca7832019-11-18 08:16:21 -08003586 if len(data) > 1:
3587 subsec = " ".join(data[1].split("_")).title()
3588 rowdict = {
3589 "section": data[0].capitalize(),
3590 "subsection": subsec,
3591 "title": test,
3592 "reference": test
3593 }
3594 cw.writerow(rowdict)
3595 else:
3596 info("{} can't be exported".format(test))
Anas Nashif83fc06a2019-06-22 11:04:10 -04003597
Anas Nashif83fc06a2019-06-22 11:04:10 -04003598
3599def native_and_unit_first(a, b):
3600 if a[0].startswith('unit_testing'):
3601 return -1
3602 if b[0].startswith('unit_testing'):
3603 return 1
3604 if a[0].startswith('native_posix'):
3605 return -1
3606 if b[0].startswith('native_posix'):
3607 return 1
3608 if a[0].split("/",1)[0].endswith("_bsim"):
3609 return -1
3610 if b[0].split("/",1)[0].endswith("_bsim"):
3611 return 1
3612
3613 return (a > b) - (a < b)
3614
3615
3616run_individual_tests = None
3617options = None
Andrew Boiebbd670c2015-08-17 13:16:11 -07003618
Andrew Boie6acbe632015-07-17 12:03:52 -07003619def main():
Andrew Boie4b182472015-07-31 12:25:22 -07003620 start_time = time.time()
Anas Nashif83fc06a2019-06-22 11:04:10 -04003621 global VERBOSE, log_file
Anas Nashife10b6512017-12-30 13:01:45 -05003622 global options
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003623 global run_individual_tests
Andrew Boie1578ef72019-07-03 10:19:29 -07003624
Anas Nashife10b6512017-12-30 13:01:45 -05003625 options = parse_arguments()
Andrew Boiebbd670c2015-08-17 13:16:11 -07003626
Anas Nashif83fc06a2019-06-22 11:04:10 -04003627
3628 if options.generate_hardware_map:
3629 from serial.tools import list_ports
3630 serial_devices = list_ports.comports()
3631 filtered = []
3632 for d in serial_devices:
Kumar Galaa4b2b5b2019-11-13 11:11:32 -06003633 if d.manufacturer in ['ARM', 'SEGGER', 'MBED', 'STMicroelectronics',
Peter Bigot52356eb2019-11-20 13:59:56 -06003634 'Atmel Corp.', 'Texas Instruments',
3635 'Silicon Labs', 'NXP Semiconductors']:
Kumar Galaa4b2b5b2019-11-13 11:11:32 -06003636 # TI XDS110 can have multiple serial devices for a single board
3637 # assume endpoint 0 is the serial, skip all others
3638 if d.manufacturer == 'Texas Instruments' and not d.location.endswith('0'):
3639 continue
Anas Nashif83fc06a2019-06-22 11:04:10 -04003640 s_dev = {}
3641 s_dev['platform'] = "unknown"
3642 s_dev['id'] = d.serial_number
3643 s_dev['serial'] = d.device
3644 s_dev['product'] = d.product
3645 if s_dev['product'] in ['DAPLink CMSIS-DAP', 'MBED CMSIS-DAP']:
3646 s_dev['runner'] = "pyocd"
Peter Bigot52356eb2019-11-20 13:59:56 -06003647 elif s_dev['product'] in ['J-Link', 'J-Link OB']:
Kumar Gala1ad35432019-11-13 07:21:12 -06003648 s_dev['runner'] = "jlink"
3649 elif s_dev['product'] in ['STM32 STLink']:
3650 s_dev['runner'] = "openocd"
Kumar Galaa4b2b5b2019-11-13 11:11:32 -06003651 elif s_dev['product'].startswith('XDS110'):
3652 s_dev['runner'] = "openocd"
Anas Nashif83fc06a2019-06-22 11:04:10 -04003653 else:
3654 s_dev['runner'] = "unknown"
3655 s_dev['available'] = True
3656 s_dev['connected'] = True
3657 filtered.append(s_dev)
3658 else:
3659 print("Unsupported device (%s): %s" %(d.manufacturer, d))
3660
3661 if os.path.exists(options.generate_hardware_map):
3662 # use existing map
3663
3664 with open(options.generate_hardware_map, 'r') as yaml_file:
3665 hwm = yaml.load(yaml_file, Loader=yaml.FullLoader)
3666 # disconnect everything
3667 for h in hwm:
3668 h['connected'] = False
Peter Bigot57c63552019-11-21 12:14:02 -06003669 h['serial'] = None
Anas Nashif83fc06a2019-06-22 11:04:10 -04003670
3671 for d in filtered:
3672 for h in hwm:
3673 if d['id'] == h['id'] and d['product'] == h['product']:
3674 print("Already in map: %s (%s)" %(d['product'], d['id']))
3675 h['connected'] = True
3676 h['serial'] = d['serial']
3677 d['match'] = True
3678
3679 new = list(filter(lambda n: not n.get('match', False), filtered))
3680 hwm = hwm + new
3681
3682 #import pprint
3683 #pprint.pprint(hwm)
3684 with open(options.generate_hardware_map, 'w') as yaml_file:
3685 yaml.dump(hwm, yaml_file, default_flow_style=False)
3686
3687
3688 else:
3689 # create new file
3690 with open(options.generate_hardware_map, 'w') as yaml_file:
3691 yaml.dump(filtered, yaml_file, default_flow_style=False)
3692
3693 return
3694
3695
Michael Scott421ce462019-06-18 09:37:46 -07003696 if options.west_runner and not options.west_flash:
3697 error("west-runner requires west-flash to be enabled")
3698 sys.exit(1)
3699
Michael Scott4ca54392019-07-09 14:21:30 -07003700 if options.west_flash and not options.device_testing:
3701 error("west-flash requires device-testing to be enabled")
3702 sys.exit(1)
3703
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02003704 if options.coverage:
3705 options.enable_coverage = True
Andrew Boie8047a6f2019-07-02 15:43:29 -07003706 options.enable_slow = True
Christian Taedckec415a4e2019-11-23 23:25:36 +01003707
3708 if not options.coverage_platform:
3709 options.coverage_platform = options.platform
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02003710
Anas Nashife10b6512017-12-30 13:01:45 -05003711 if options.size:
3712 for fn in options.size:
Andrew Boie52fef672016-11-29 12:21:59 -08003713 size_report(SizeCalculator(fn, []))
Andrew Boiebbd670c2015-08-17 13:16:11 -07003714 sys.exit(0)
3715
Anas Nashife10b6512017-12-30 13:01:45 -05003716 VERBOSE += options.verbose
Anas Nashif83fc06a2019-06-22 11:04:10 -04003717
Anas Nashife10b6512017-12-30 13:01:45 -05003718 if options.log_file:
3719 log_file = open(options.log_file, "w")
Oleg Zhurakivskyyc97054c2018-08-17 14:56:05 +03003720
Anas Nashife10b6512017-12-30 13:01:45 -05003721 if options.subset:
3722 subset, sets = options.subset.split("/")
Anas Nashif035799f2017-05-13 21:31:53 -04003723 if int(subset) > 0 and int(sets) >= int(subset):
Anas Nashif3ba1d432017-12-05 15:28:44 -05003724 info("Running only a subset: %s/%s" % (subset, sets))
Anas Nashif035799f2017-05-13 21:31:53 -04003725 else:
Anas Nashife10b6512017-12-30 13:01:45 -05003726 error("You have provided a wrong subset value: %s." % options.subset)
Anas Nashif035799f2017-05-13 21:31:53 -04003727 return
3728
Anas Nashif83fc06a2019-06-22 11:04:10 -04003729 # Cleanup
3730
3731 if options.no_clean or options.only_failed or options.test_only:
3732 if os.path.exists(options.outdir):
3733 info("Keeping artifacts untouched")
3734 elif os.path.exists(options.outdir):
3735 for i in range(1,100):
3736 new_out = options.outdir + ".{}".format(i)
3737 if not os.path.exists(new_out):
3738 info("Renaming output directory to {}".format(new_out))
3739 shutil.move(options.outdir, new_out)
3740 break
3741 #shutil.rmtree("%s.old" %options.outdir)
Andrew Boie6acbe632015-07-17 12:03:52 -07003742
Anas Nashife10b6512017-12-30 13:01:45 -05003743 if not options.testcase_root:
3744 options.testcase_root = [os.path.join(ZEPHYR_BASE, "tests"),
Andrew Boie3d348712016-04-08 11:52:13 -07003745 os.path.join(ZEPHYR_BASE, "samples")]
3746
Anas Nashif83fc06a2019-06-22 11:04:10 -04003747 suite = TestSuite(options.board_root, options.testcase_root, options.outdir)
3748 suite.add_testcases()
3749 suite.add_configurations()
Anas Nashifbd166f42017-09-02 12:32:08 -04003750
Anas Nashif83fc06a2019-06-22 11:04:10 -04003751 if options.device_testing:
3752 if options.hardware_map:
3753 suite.load_hardware_map(options.hardware_map)
3754 if not options.platform:
3755 options.platform = []
3756 for platform in suite.connected_hardware:
3757 if platform['connected']:
3758 options.platform.append(platform['platform'])
3759
3760 elif options.device_serial: #back-ward compatibility
3761 if options.platform and len(options.platform) == 1:
3762 suite.load_hardware_map_from_cmdline(options.device_serial,
3763 options.platform[0])
3764 else:
3765 error("""When --device-testing is used with --device-serial, only one
3766 platform is allowed""")
3767
3768
Anas Nashif83fc06a2019-06-22 11:04:10 -04003769 if suite.load_errors:
Kumar Galac84235e2018-04-10 13:32:51 -05003770 sys.exit(1)
3771
Anas Nashif75547e22018-02-24 08:32:14 -06003772 if options.list_tags:
3773 tags = set()
Anas Nashif83fc06a2019-06-22 11:04:10 -04003774 for _, tc in suite.testcases.items():
Anas Nashif75547e22018-02-24 08:32:14 -06003775 tags = tags.union(tc.tags)
3776
3777 for t in tags:
3778 print("- {}".format(t))
3779
3780 return
3781
Anas Nashif5d6e7eb2018-06-01 09:51:08 -05003782 if options.export_tests:
3783 cnt = 0
Anas Nashifeabaa7f2019-11-18 07:49:17 -08003784 tests = suite.get_all_tests()
Anas Nashif5d6e7eb2018-06-01 09:51:08 -05003785 export_tests(options.export_tests, tests)
3786 return
3787
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003788 run_individual_tests = []
Anas Nashif5d6e7eb2018-06-01 09:51:08 -05003789
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003790 if options.test:
3791 run_individual_tests = options.test
3792
Anas Nashif19ca7832019-11-18 08:16:21 -08003793 if options.list_tests or options.list_test_duplicates or options.sub_test:
Anas Nashif49b22d42019-06-14 13:45:34 -04003794 cnt = 0
Anas Nashifeabaa7f2019-11-18 07:49:17 -08003795 all_tests = suite.get_all_tests()
Anas Nashif49b22d42019-06-14 13:45:34 -04003796
Anas Nashif19ca7832019-11-18 08:16:21 -08003797 if options.list_test_duplicates:
3798 import collections
3799 dupes = [item for item, count in collections.Counter(all_tests).items() if count > 1]
3800 if dupes:
3801 print("Tests with duplicate identifiers:")
3802 for dupe in dupes:
3803 print("- {}".format(dupe))
3804 for dc in suite.get_testcase(dupe):
3805 print(" - {}".format(dc))
3806 else:
3807 print("No duplicates found.")
3808 return
3809
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003810 if options.sub_test:
Anas Nashifd11fd782019-11-18 10:22:56 -08003811 for st in options.sub_test:
3812 subtests = suite.get_testcase(st)
3813 for sti in subtests:
3814 run_individual_tests.append(sti.name)
3815
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003816 if run_individual_tests:
3817 info("Running the following tests:")
Anas Nashifeabaa7f2019-11-18 07:49:17 -08003818 for test in run_individual_tests:
3819 print(" - {}".format(test))
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003820 else:
3821 info("Tests not found")
3822 return
3823
3824 elif options.list_tests:
Anas Nashifeabaa7f2019-11-18 07:49:17 -08003825 for test in all_tests:
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003826 cnt = cnt + 1
Anas Nashifeabaa7f2019-11-18 07:49:17 -08003827 print(" - {}".format(test))
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003828 print("{} total.".format(cnt))
3829 return
Anas Nashifc0149cc2018-04-14 23:12:58 -05003830
Anas Nashifbd166f42017-09-02 12:32:08 -04003831 discards = []
Anas Nashif83fc06a2019-06-22 11:04:10 -04003832
3833 if options.only_failed:
3834 suite.get_last_failed()
3835 elif options.load_tests:
3836 suite.load_from_file(options.load_tests)
3837 elif options.test_only:
3838 last_run = os.path.join(options.outdir, "sanitycheck.csv")
3839 suite.load_from_file(last_run)
Anas Nashifbd166f42017-09-02 12:32:08 -04003840 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -04003841 discards = suite.apply_filters()
Andrew Boie6acbe632015-07-17 12:03:52 -07003842
Anas Nashif30551f42018-01-12 21:56:59 -05003843 if VERBOSE > 1 and discards:
Anas Nashiff18e2ab2018-01-13 07:57:42 -05003844 # if we are using command line platform filter, no need to list every
3845 # other platform as excluded, we know that already.
3846 # Show only the discards that apply to the selected platforms on the
3847 # command line
3848
Andrew Boie08ce5a52016-02-22 13:28:10 -08003849 for i, reason in discards.items():
Anas Nashiff18e2ab2018-01-13 07:57:42 -05003850 if options.platform and i.platform.name not in options.platform:
3851 continue
Anas Nashif3ba1d432017-12-05 15:28:44 -05003852 debug(
3853 "{:<25} {:<50} {}SKIPPED{}: {}".format(
3854 i.platform.name,
Anas Nashif83fc06a2019-06-22 11:04:10 -04003855 i.testcase.name,
Anas Nashif3ba1d432017-12-05 15:28:44 -05003856 COLOR_YELLOW,
3857 COLOR_NORMAL,
3858 reason))
Andrew Boie6acbe632015-07-17 12:03:52 -07003859
Anas Nashif49b22d42019-06-14 13:45:34 -04003860 if options.report_excluded:
Anas Nashifeabaa7f2019-11-18 07:49:17 -08003861 all_tests = suite.get_all_tests()
Anas Nashif49b22d42019-06-14 13:45:34 -04003862 to_be_run = set()
Anas Nashif83fc06a2019-06-22 11:04:10 -04003863 for i,p in suite.instances.items():
3864 to_be_run.update(p.testcase.cases)
Anas Nashif49b22d42019-06-14 13:45:34 -04003865
Anas Nashif83fc06a2019-06-22 11:04:10 -04003866 if all_tests - to_be_run:
Anas Nashif49b22d42019-06-14 13:45:34 -04003867 print("Tests that never build or run:")
Anas Nashif83fc06a2019-06-22 11:04:10 -04003868 for not_run in all_tests - to_be_run:
Anas Nashif49b22d42019-06-14 13:45:34 -04003869 print("- {}".format(not_run))
3870
3871 return
3872
Anas Nashife10b6512017-12-30 13:01:45 -05003873 if options.subset:
Anas Nashif83fc06a2019-06-22 11:04:10 -04003874 #suite.instances = OrderedDict(sorted(suite.instances.items(),
3875 # key=cmp_to_key(native_and_unit_first)))
Anas Nashife10b6512017-12-30 13:01:45 -05003876 subset, sets = options.subset.split("/")
Anas Nashif83fc06a2019-06-22 11:04:10 -04003877 total = len(suite.instances)
Anas Nashif035799f2017-05-13 21:31:53 -04003878 per_set = round(total / int(sets))
Anas Nashif3ba1d432017-12-05 15:28:44 -05003879 start = (int(subset) - 1) * per_set
Anas Nashif035799f2017-05-13 21:31:53 -04003880 if subset == sets:
3881 end = total
3882 else:
3883 end = start + per_set
3884
Anas Nashif83fc06a2019-06-22 11:04:10 -04003885 sliced_instances = islice(suite.instances.items(), start, end)
3886 suite.instances = OrderedDict(sliced_instances)
Anas Nashif035799f2017-05-13 21:31:53 -04003887
Andrew Boie6acbe632015-07-17 12:03:52 -07003888
Anas Nashif83fc06a2019-06-22 11:04:10 -04003889 if options.save_tests:
3890 suite.csv_report(options.save_tests)
Andrew Boie6acbe632015-07-17 12:03:52 -07003891 return
3892
Anas Nashif83fc06a2019-06-22 11:04:10 -04003893 info("%d test configurations selected, %d configurations discarded due to filters." %
3894 (len(suite.instances), len(discards)))
3895
Peter Bigot3d46ea52019-11-21 12:00:18 -06003896 if options.device_testing:
3897 print("\nDevice testing on:")
3898 for p in suite.connected_hardware:
3899 if p['connected']:
3900 print("%s (%s) on %s" %(p['platform'], p.get('id', None), p['serial']))
3901
Anas Nashif83fc06a2019-06-22 11:04:10 -04003902 if options.dry_run:
3903 duration = time.time() - start_time
3904 info("Completed in %d seconds" % (duration))
3905 return
3906
3907 retries = options.retry_failed + 1
3908 completed = 0
3909
3910 suite.update()
3911 suite.start_time = start_time
3912
3913 while True:
3914 completed += 1
3915
3916 if completed > 1:
3917 info("%d Iteration:" %(completed ))
3918 time.sleep(60) # waiting for the system to settle down
3919 suite.total_done = suite.total_tests - suite.total_failed
3920 suite.total_failed = 0
3921
3922 suite.execute()
Anas Nashif654ec5982019-04-11 08:38:21 -04003923 info("", False)
Andrew Boie6acbe632015-07-17 12:03:52 -07003924
Anas Nashif83fc06a2019-06-22 11:04:10 -04003925 retries = retries - 1
3926 if retries == 0 or suite.total_failed == 0:
3927 break
Anas Nashife0a6a0b2018-02-15 20:07:24 -06003928
Anas Nashif83fc06a2019-06-22 11:04:10 -04003929 suite.misc_reports(options.compare_report, options.show_footprint,
3930 options.all_deltas, options.footprint_threshold, options.last_metrics)
Andrew Boie6acbe632015-07-17 12:03:52 -07003931
Anas Nashif83a98e52019-11-24 07:42:06 -05003932 suite.duration = time.time() - start_time
3933 suite.summary(options.disable_unrecognized_section_test)
3934
Anas Nashife10b6512017-12-30 13:01:45 -05003935 if options.coverage:
Anas Nashif83fc06a2019-06-22 11:04:10 -04003936 if options.gcov_tool is None:
Alberto Escolar Piedrasaf30f2b2019-09-12 14:44:08 +02003937 use_system_gcov = False
Andrew Boie49cf4862019-07-08 12:02:13 -07003938
3939 for plat in options.coverage_platform:
Anas Nashif83fc06a2019-06-22 11:04:10 -04003940 ts_plat = suite.get_platform(plat)
Alberto Escolar Piedrasaf30f2b2019-09-12 14:44:08 +02003941 if ts_plat and (ts_plat.type in {"native", "unit"}):
3942 use_system_gcov = True
Andrew Boie49cf4862019-07-08 12:02:13 -07003943
Alberto Escolar Piedrasaf30f2b2019-09-12 14:44:08 +02003944 if use_system_gcov or "ZEPHYR_SDK_INSTALL_DIR" not in os.environ:
Andrew Boie49cf4862019-07-08 12:02:13 -07003945 options.gcov_tool = "gcov"
3946 else:
3947 options.gcov_tool = os.path.join(os.environ["ZEPHYR_SDK_INSTALL_DIR"],
3948 "i586-zephyr-elf/bin/i586-zephyr-elf-gcov")
3949
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003950 info("Generating coverage files...")
Anas Nashif4df6c562018-11-07 19:29:04 -05003951 generate_coverage(options.outdir, ["*generated*", "tests/*", "samples/*"])
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003952
Anas Nashif83fc06a2019-06-22 11:04:10 -04003953 if options.device_testing:
3954 print("\nHardware distribution summary:\n")
3955 for p in suite.connected_hardware:
3956 if p['connected']:
3957 print("%s (%s): %d" %(p['platform'], p.get('id', None), p['counter']))
3958
Flavio Ceolinca1feea2019-10-22 14:03:48 -07003959 suite.save_reports()
Anas Nashif83fc06a2019-06-22 11:04:10 -04003960 if suite.total_failed or (suite.warnings and options.warnings_as_errors):
Andrew Boie6acbe632015-07-17 12:03:52 -07003961 sys.exit(1)
3962
3963if __name__ == "__main__":
3964 main()